Re: [ui-dev] Q: How to implement XAccessibleText interface in formula editor ?

[Peter Korn <peter korn sun com>]

Hi Thomas,

I've cross-posted to the GNOME Accessibility list, the Mozilla
accessibility list, and the Java Accessibility list, as I think folks in
those communities may have some good suggestions as well.  If nothing else,
there are communities that should be thinking about these issues.  I think
our ATV friends in the community are especially important to get feedback
from on these questions (BAUM, University of Toronto, Freedom Scientific, &
Ai Squared).  I've also directly cc-ed John Gardener, a professor of
Physics at Oregon State University in the U.S.  John has spent many years
looking at ways of representing mathematical equations to the blind.

For the GNOME and Mozilla communities: the question Thomas raises is how
should special characters (that are multi-character strings as they are
stored within a document), and how should characters in formulas, be
represented in the AccessibleText interface.  Thomas' question is specific
to the OpenOffice UNO Accessibility API (and the XAccessibleText
interface), but the problem faces us in GNOME, Netscape, and Java land as
well.

I'm appending my person thoughts at the end of Thomas original e-mail
below.

Thomas Lange wrote:
> 
> Hi,
> 
> How should I implement the functions
>   virtual awt::Rectangle SAL_CALL getCharacterBounds( sal_Int32 nIndex )
>   virtual sal_Int32 SAL_CALL getIndexAtPoint( const awt::Point& aPoint )
> from the XAccessibleText interface in the formula editor ?
> 
> Especially considering that some symbols in the graphical display are
> represented as a single character (for example the greek Sigma 
> character) where the character will be represented by a 'word' (here 
> '%SIGMA') in the text.
> 
> If for example the formula text is "%ALPHA + %BETA equals %KAPPA" which
> would result in a graphical representation like "A+B=K" which has only 5
> characters.
> 
> Should I return the bounding rectangle of "A" for the indices 0-5
> (%ALPHA) when calling getCharacterBounds ?
> And when calling getIndexAtPoint for a point of "A" should I return 0 as
> index ?
> 
> Or should I expand the symbolic names to the unicode character they are
> representing and thus having the
>   rtl::OUString SAL_CALL getText()
> function return the string "A+B=K" also where the symbolic names will
> have been replaced with the corresponding unicode character.
> This approach would make the implentations of  getCharacterBounds  and
> getIndexAtPoint  quite obvious.
> However many of the unicode characters used in the formula editor come
> from within the private area of the unicode character set. Will this be 
> a problem ?
> 
> The next problem will be how should I implement these functions for
> formatted text ?
> For example when a text or character has super- and subscripts in its
> graphical representation ?
> 
>    n
>   AAA  x
>    m
> 
> Which should be the order of assigning the indices to the text ?
> Should index 0 be the 'n', 1 'AAA' and 2 'm' ?
> 
> How about some unusual attributed text like symbols with a (single)
> vector arrow above ?
>  -->
>   V
> 
> Regards,
> Thomas

I think for symbols like "%ALPHA", they should be seen as a single (Unicode
or UTF8) character wherever possible.  Failing that, I think the characters
of that "word" should share the same bounding rectangle.  But a single
character would, I think, be better.

For more complex formulas, like a summation (or integral) from m to n, I
think you'll have to pick a cannonical order (and it should be one we all
agree to so it's standard throughout all accessible representations of math
formulas).  We could also introduce a bunch of new AccessibleRelations to
try to cover formulas, making each of these characters (or collections of
characters) their own object with relations to other objects.

Unfortunately, none of these approaches is really all that satisfying.


A longer-term approach we should consider is finding a way to represent the
entire formula as it's own object, and create an AccessibleDescription for
the formula that explains what it is (so something like "Integral from m to
n of the function f(x)").  Also, we should look toward standards like
MathML to represent the formulas, and expose the MathML directly for a
screen access product to parse as it chooses.  If there is a MIME type for
MathML, we could use the AccessibleStreamable interface (in GNOME, not yet
in the Java Accessibility API) to indicate there is a MathML stream which
can be parsed for this object.


Regards,

Peter Korn
Sun Accessibility team