cannot enable assistive technologies in ubuntu mobile

change components category from moblin-applets to moblin-ui-framework
and subscribe developers.

Ian: have you run dogtail UI case on Ubuntu desktop?

not in PRD so change it to wish list.
No assigner for this bug too.

This is the output from a dogtail script I wrote which tests the moblin-media UI. I ran it inside a UME Xephyr window with assistive technologies enabled:

##### moblin-media_20071107-151653_debug
Detecting distribution: Ubuntu (or derived distribution)
Traceback (most recent call last):
  File "moblin-media", line 9, in <module>
    from dogtail.procedural import *
  File "/var/lib/python-support/python2.5/dogtail/procedural.py", line 18, in <module>
    import tree
  File "/var/lib/python-support/python2.5/dogtail/tree.py", line 1247, in <module>
    root = Root (atspi.registry.getDesktop ())
  File "pyspi.pyx", line 195, in atspi.Registry.getDesktop
  File "pyspi.pyx", line 152, in atspi.exception_handler
SpiException: Non-fatal SPIException: type:0 source:0 "IDL:omg.org/CORBA/INV_OBJREF:1.0"

also I received a 'warning' that that AT_SPI_REGISTRY could not be located and that moblin-media did not register itself to this registry

I can't think of an equivalent to gdm or gnome-session in UME; perhaps we need one, or perhaps we should be able to do such stuff in XSession itself.

Ian: any idea of what's actually required to be done by some program to enable a11y on login?

All that is needed is the ability to add some GTK Modules.

Here is the relevant section from gdm.conf
#AddGtkModules=false
# By default, these are the accessibility modules.
#GtkModulesList=gail:atk-bridge:/usr/lib/gtk-2.0/modules/libdwellmouselistener:/usr/lib/gtk-2.0/modules/libkeymouselistener

To enable the use of assistive technologies the "AddGtkModules" parameter from gdm.conf must be uncommented and set to "true".
Also the "GtkModulesList" parameter must be uncommented

Depending on the degree of a11y required in UME some of the above GtkModules may not need to be loaded. If end-users will need the integrated Screen Reader and Magnifier, "gail" and "atk-bridge" are needed. If end-users will be using a pointing device without buttons or switches, include "dwellmouselistener". If some users will use pointing devices with switches, alternative physical keyboards, or switch/button devices, include "keymouselistener". Enabling the a11y infrastructure does impact desktop performance so some way to selectively enable this (like gdm) is obviously preferred.

Outside of end user requirements the minimum necessary for writing UI test scripts would be "gail" and "atk-bridge". This is because
gail is dynamically loadable at runtime by a GTK application. Once loaded, those parts of an application that use standard GTK widgets will have a basic level of accessibility, without anybody needing to modify an application at all. (If GAIL is not loaded, GTK widgets will have a default accessibility implementation that essentially returns no information, though it nominally conforms to the ATK API.). Whether or not applications automatically load these accessibility support libraries depends on the value of a gconf key, "/desktop/gnome/interface/accessibility"; a boolean value of "true" enables support for assistive technologies and applications which call gnome_program_init will automatically load the appropriate accessibility libraries at runtime. "Pure GTK+ applications", e.g. those that use gtk+ but do not link to libgnome, rely on the value of the GTK_MODULES environment variable, which must be set to "gail:atk-bridge" in order to enable assistive technology support.

The above error message I believe is because the Corba IDL is trying to describe an object through the Bonobo interface which does not exists in hildon desktop. (this was originally removed by Nokia due to space constraints IIRC). A bonobo armel deb has however since been created for the Hildon Test Automation garage project (https://garage.maemo.org/projects/hildon-test-aut/)

If it's only to run dogtail automated tests, perhaps you can use .xinitrc for a "dogtail" user or simply use /etc/environment to set the env vars system-wide?

I tried putting

echo "enabling accessibility"
export GTK_MODULES=gail:atk-bridge
export GNOME_ACCESSIBILITY=1
gconftool-2 --set --type bool /desktop/gnome/interface/accessibility true
/usr/lib/at-spi/at-spi-registryd &

inside ume-xephyr-start.
Running the testing script gave the same bug report error

Setting the environment variables inside gdb
(gdb) set environment GTK_MODULES=gail:atk-bridge
(gdb) run

gave
GTK Accessibility Module Initialized
atk-bridge-warning: AT_SPI_REGISTRY was not started at session startup
atk-bridge-warning: IOR not set
atk-bridge-warning: Could not locate registry

and then the Program exited with code 0377 which means garbage was returned by the OS

I also compiled hail inside the target and substituted hail for gail like:
export GTK_MODULES=/usr/lib/gtk-2.0/modules/lib/gtk-2.0/modules/libhail:atk-bridge

putting this inside the ume-xephyr-start script meant the UI would not
start with this error:
FreeFontPath: FPE "/usr/share/fonts/X11/misc" refcount is 2, should be 1; fixing.

commenting this out in ume-xephyr-start and exporting it directly from a shell inside the UI gave this:

python moblin-media
creating logfile at /tmp/dogtail/logs/moblin-media/20071119-111356_debug
Hildon Accessibility Module Initialized
Error file accessible.c line 554 (spi_accessible_construct): assertion failed
Aborted (core dumped)

when running the testing script. This seems more encouraging than the original bug reported error message

a bit more digging uncovered this:

Hildon Accessibility Module initialized

** ERROR **: file accessible.c: line 554 (spi_accessible_construct): assertion failed: (o)
aborting...

Program received signal SIGABRT, Aborted.
[Switching to Thread -1227409744 (LWP 24587)]
0xffffe410 in __kernel_vsyscall ()
(gdb) where
#0 0xffffe410 in __kernel_vsyscall ()
#1 0xb717c875 in raise () from /lib/tls/i686/cmov/libc.so.6
#2 0xb717e201 in abort () from /lib/tls/i686/cmov/libc.so.6
#3 0xb7459f4a in g_logv () from /usr/lib/libglib-2.0.so.0
#4 0xb7459f89 in g_log () from /usr/lib/libglib-2.0.so.0
#5 0xb745a006 in g_assert_warning () from /usr/lib/libglib-2.0.so.0
#6 0xb6ad77f2 in spi_accessible_construct () from /usr/lib/libspi.so.0
#7 0xb6ad86c2 in spi_application_new () from /usr/lib/libspi.so.0
#8 0xb6c4877f in ?? () from /usr/lib/gtk-2.0/modules/libatk-bridge.so
#9 0x00000000 in ?? ()
(gdb)

here are the relevant lines from http://codebrowse.launchpad.net/~vcs-imports/at-spi/trunk/annotate/vcs-imports%40canonical.com-20071115183048-fvvqhx4ssia77dm5?file_id=libspiaccessible.c-20071008164538-4qprwj1cg939hp12-35

545 SpiAccessible *
546 spi_accessible_construct (GType type, AtkObject *o)
547 {
548 SpiAccessible *retval;
549 CORBA_Environment ev;
550
551 CORBA_exception_init (&ev);
552
553 g_assert (o);
554 g_assert (g_type_is_a (type, SPI_ACCESSIBLE_TYPE));

Ian,
Have you installed the Dogtail envrionment on MID successfully?
I am working on this, but I am afraid the AT-SPI is only use CORBA.
Are there any D-Bus supported AT-SPI pacakges we can installed?

I have some 'solution' and dogtail is now running inside ume...i am still getting errors but now i think they are related to the application under tests. This is what I did:

1. Created a new lpia image in MIC

2. In a target:
aptitude install libgail-common libgail-dev libbonobo2-0 libbonobo2-common libbonobo2-dev autoconf autotools-dev libtool gtk-doc-tools automake1.7
automake1.9 hildon-desktop-dev outo libatspi-dev libatspi1.0-0 python-pyatspi libffi4 libffi4-dev python2.5-dev

3. Compile and Install Hildon Accessibility Library to the target (https://edge.launchpad.net/hail/)
./autogen.sh --prefix=/usr
make && make install

4. Put the rocknrole.py script inside the targets /usr/lib/python2.5 directory.
5. Start 2 target terminals...in one run ume-xephyr-start and when the ume desktop appears run moblin-media

In the other terminal other do this:

(T: lpia-traget)root@lawrence:/home/ume# python
Python 2.5.1 (r251:54863, Oct 5 2007, 13:40:04)
[GCC 4.2.1 (Ubuntu 4.2.1-5ubuntu4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from rocknrole import *
Creating logfile at /tmp/dogtail/logs/log_20071217-135744_debug ...
Detecting distribution: Ubuntu (or derived distribution)
Hildon Accessibility Module initialized
Creating logfile at /tmp/dogtail/logs/log_20071217-135744_results ...

>>> the_mutts_nuts=appR('moblin-media')
searching for child of {root}: "moblin-media" application (attempt 3)
searching for child of {root}: "moblin-media" application (attempt 4)
searching for child of {root}: "moblin-media" application (attempt 5)
searching for child of {root}: "moblin-media" application (attempt 6)

....

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.5/rocknrole.py", line 414, in appR
    return RNR(tree.root.application(app))
  File "/var/lib/python-support/python2.5/dogtail/tree.py", line 1092, in application
    return Application(root.findChild(predicate.IsAnApplicationNamed(appName),recursive=False))
  File "/var/lib/python-support/python2.5/dogtail/tree.py", line 875, in findChild
    raise SearchError(describeSearch(self, pred, recursive, debugName))
SearchError: child of {root}: "moblin-media" application

this error is now related to moblin-media (not registering itself properly i think) and not dogtail which now imports fine

> this error is now related to moblin-media (not registering itself
> properly i think) and not dogtail which now imports fine

Ideas on what is needed to "register itself properly" ?
Bob

Ian Lawrence wrote:
> I have some 'solution' and dogtail is now running inside ume...i am
> still getting errors but now i think they are related to the
> application under tests. This is what I did:
>
> 1. Created a new lpia image in MIC
>
> 2. In a target:
> aptitude install libgail-common libgail-dev libbonobo2-0
> libbonobo2-common libbonobo2-dev autoconf autotools-dev libtool
> gtk-doc-tools automake1.7
> automake1.9 hildon-desktop-dev outo libatspi-dev libatspi1.0-0
> python-pyatspi libffi4 libffi4-dev python2.5-dev
>
> 3. Compile and Install Hildon Accessibility Library to the target
> (https://edge.launchpad.net/hail/) ./autogen.sh --prefix=/usr
> make && make install
>
> 4. Put the rocknrole.py script inside the targets /usr/lib/python2.5
> directory.
> 5. Start 2 target terminals...in one run ume-xephyr-start and when
> the ume desktop appears run moblin-media
>
> In the other terminal other do this:
>
> (T: lpia-traget)root@lawrence:/home/ume# python Python 2.5.1
> (r251:54863, Oct 5 2007, 13:40:04) [GCC 4.2.1 (Ubuntu
> 4.2.1-5ubuntu4)] on linux2 Type "help", "copyright", "credits" or
> "license" for more information.
>>>> from rocknrole import *
> Creating logfile at /tmp/dogtail/logs/log_20071217-135744_debug ...
> Detecting distribution: Ubuntu (or derived distribution) Hildon
> Accessibility Module initialized Creating logfile at
> /tmp/dogtail/logs/log_20071217-135744_results ...
>
>>>> the_mutts_nuts=appR('moblin-media')
> searching for child of {root}: "moblin-media" application (attempt 3)
> searching for child of {root}: "moblin-media" application (attempt 4)
> searching for child of {root}: "moblin-media" application (attempt 5)
> searching for child of {root}: "moblin-media" application (attempt 6)
>
> ....
>
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/usr/lib/python2.5/rocknrole.py", line 414, in appR
> return RNR(tree.root.application(app))
> File "/var/lib/python-support/python2.5/dogtail/tree.py", line
> 1092, in application return
>
>
>
Application(root.findChild(predicate.IsAnApplicationNamed(appName),recur
sive=False))
> File "/var/lib/python-support/python2.5/dogtail/tree.py", line 875,
> in findChild raise SearchError(describeSearch(self, pred, recursive,
> debugName))
> SearchError: child of {root}: "moblin-media" application
>
>
> this error is now related to moblin-media (not registering itself
> properly i think) and not dogtail which now imports fine
>
>
> ** Attachment added: "Work on dogtail nodes by creating objects with
> rolenames as attributes"
> http://launchpadlibrarian.net/10934968/rocknrole.tar.gz

The above statement is a slight misnomer as rather than "register itself properly" it should read "nothing to register itself too".

To explain this :

# ps aux | grep moblin-media
root 464 0.0 0.0 1732 488 ? S 11:25 0:00 /bin/sh /usr/bin/moblin-media

it seems that although moblin-media has a process ID it is not registered:

# export GTK_MODULES=gail:hail:atk-bridge
# python
Python 2.5.1 (r251:54863, Oct 5 2007, 13:40:04)
[GCC 4.2.1 (Ubuntu 4.2.1-5ubuntu4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyatspi
>>> reg = pyatspi.Registry
>>> print reg.getDesktopCount()
1
>>> desktop = reg.getDesktop(0)
>>> print len(desktop)
0

(so we have a running desktop with apparently no apps running in it)

this is confirmed by:

>>> from rocknrole import *
Creating logfile at /tmp/dogtail/logs/log_20071218-133920_debug ...
Detecting distribution: Ubuntu (or derived distribution)
Creating logfile at /tmp/dogtail/logs/log_20071218-133920_results ...
>>> isRunning('moblin-media')
False

So the ume session manager needs somehow to start the accessibility registry (at-spi-registryd).

In GNOME it has been implemented using code in gnome-session (if a11y was on, start the registry and set the GTK_MODULES variables) which is run at login. I tried a work around this by putting
/usr/lib/at-spi/at-spi-registryd into the ume-xephyr-start script and got this result (this time with the midbrowser started not moblin-media)

>>> import pyatspi
>>> reg = pyatspi.Registry
>>> desktop = reg.getDesktop(0)
>>> print len(desktop)
2
>>> for app in desktop: print app
...
[application | gnome-terminal]
[application | .]
>>> for i in range(desktop.childCount): print desktop[i]
...
[application | gnome-terminal]
[application | .]
>>> isRunning('.')
True
>>> h=appR('.')
>>> h.helper()

#################################################################################################
{"." application}
#################################################################################################

>>>

It works (well it doesn't error at least) but returns none of the co-ordinates for the midbrowser GUI components.

The next stage is obviously to see why i am getting a '. 'and not 'midbrowser' returned as the application name

Hi, Ian,
Thank you for your step-by-step descriptions. I am working on with your
descriptions.
I found something not same as you description. Did you get same symptom
before?
---------------- my command ---------------
root@mobile-7e1d:~# python
Python 2.5.1 (r251:54863, Oct 5 2007, 13:40:04)
[GCC 4.2.1 (Ubuntu 4.2.1-5ubuntu4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyatspi
>>> reg=pyatspi.Registry
>>> desktop=reg.getDesktop(0)
>>> print len(desktop)
0 [Lei Y] I think
here should print some applications running
>>> print desktop
[unknown | main]
>>> isRunning('.')
True
>>> h=appR('.')
searching for child of {root}: "." application (attempt 3)
searching for child of {root}: "." application (attempt 4)
searching for child of {root}: "." application (attempt 5)
searching for child of {root}: "." application (attempt 6)
searching for child of {root}: "." application (attempt 7)
searching for child of {root}: "." application (attempt 8)
searching for child of {root}: "." application (attempt 9)
searching for child of {root}: "." application (attempt 10)
searching for child of {root}: "." application (attempt 11)
searching for child of {root}: "." application (attempt 12)
searching for child of {root}: "." application (attempt 13)
searching for child of {root}: "." application (attempt 14)
searching for child of {root}: "." application (attempt 15)
searching for child of {root}: "." application (attempt 16)
searching for child of {root}: "." application (attempt 17)
searching for child of {root}: "." application (attempt 18)
searching for child of {root}: "." application (attempt 19)
--------------------------------------------

Thanks and Best Regards
Lei Yang

Hi Lei Y
Look at the output from a normal gnome session with accessibility enabled at gdm login

Python 2.5.1 (r251:54863, Oct 5 2007, 13:36:32)
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyatspi
>>> reg=pyatspi.Registry
>>> desktop=reg.getDesktop(0)
>>> print len(desktop)
18
>>> print desktop
[unknown | main]
>>>

so you are correct, ume applications are not registering to the at-spi register