Re: RFC: keyboard/mouse usage for accessibility?

[Samuel Thibault <samuel thibault ens-lyon org>]

Eric Johansson, le lun. 18 mars 2019 14:31:54 -0400, a ecrit:
> On 3/18/2019 2:15 PM, Samuel Thibault wrote:
> > Eric Johansson, le lun. 18 mars 2019 13:44:04 -0400, a ecrit:
> > > I'm a speech recognition user and quite frankly, almost all the
> > > accessibility features people have put in place do not work  with speech
> > > recognition. How can we get our needs injected in the conversation?
> > By talking about it :)
>
> Thanks :-) where should I be talking about it?

Here is a good start :)
Just make sure to keep gnome-accessibility-devel gnome org in Cc since
that's where at-spi is mostly discussed AFAIK.

> > I should be able to just say "restart numbering" and have it activate
> > > that "right click, hunt through the menu and click on the item"
> > > action.
> > Right, this is actually supported in at-spi, actionable widgets are
> > supposed to have a list of actions with a name for each, that can be
> > used in such a situation (and could be shown to the user so she knows
> > she can trigger it). I have also added that to the list.
>
> Is there any capabilities in at-spi that allow a speech recognition
> environment to query the application and find out enough context to be
> able to generate appropriate grammars? for example, using  a
> multipurpose editor. I may want to have different grammars for different
> types of data. I need to know what tab has focus and something about
> that tab (TBD) so that I can generate the right grammar to operate on
> data within that tab.

At-spi provides information of which widget has focus, and then with
at-spi you can inspect the list of actions.

> Each cell of the database has a type and a name. For text fields,
> saying "Change <name>" Should put me in the cell of that name. But if
> the cell is a multi-selection list,  the grammar should be all of the
> options for the multi-selection list. If if the cell is a number, I
> want to limit the grammar to just numbers.

AFAICT, the type of content is not exposed in at-spi, but that could be
added.

> There are a bunch of other types such as email address phone number URL
> people and other limited board non-English language elements that could
> be spoken. Each of which need their own grammar.

The exact allowed grammer could be passed through at-spi.

> One of the problems though with the notion database is that there are no
> row names except by convention. Therefore whenever you use a name to
> sell, somebody needs to keep track of the role you are on and no
> command  should take you off of that row.

I'm not sure to understand.

> The last issue is a method of bypassing JavaScript  backed editors. I
> cannot dictate into Google Docs, have difficulty dictating into
> LinkedIn. In the browser context, only naked text areas seem to work
> well with NaturallySpeaking.

That is the kind of example where plugging only through at-spi can fail,
when the application is not actually exposing an at-spi interface, and
thus plugging at the OS level can be a useful option.

Samuel