RFC: scrolling

[Benjamin Otte <otte gnome org>]

Hey,

Here's another one of those mails about stuff to think about for the
hackfest. This time I have no clue at all what I consider the solution
we should take and I've been thinking about this since at least the
last hackfest. Relevant links to this are:
  https://live.gnome.org/GnomeOS/Design/Whiteboards/Scrolling
  http://developer.gnome.org/gtk3/stable/GtkScrolledWindow.html
  http://developer.gnome.org/gtk3/stable/GtkScrollable.html

Also, I will be using two terms in this discussion, and I think I
should define them:
- "scrollable"
The scrollable is the widget that is scrolled. In Gtk, the important
ones are GtkIconView, GtkTreeView, GtkTextView and WebKitWebView.
There is also a bunch of other ones such as GtkViewport, GtkLayout or
GnomeCanvas (and its various forks and clones). And a bunch of other
cases where GTK apps do scrolling that don't use a common scrolling
infrastructure, such as EOG's image view or libchamplain's maps
widget.
- "viewport"
This is the widget that is actually managing the scrolling by drawing
scrollbars, reacting to scroll events and such. This is roughly
equivalent to GtkViewport from the GTK side. But other widgets do
exist, such as HildonPannableArea.
(Note: I'm using this term for its OpenGL meaning; do not confuse it
with GtkViewport.)

You will notice that there is a split between viewport and scrollable.
I can come up with a few methods to implement this split:

A) Don't implement it
Instead, just rely on scrollables to do the scrolling themselves. The
toolkit could provide some helper functionality, but if a widget
wanted to scroll, it would do it itself. An example of this behavior
is the map on http://maps.google.com
B) Let the scrollable do the scrolling
The viewport manages scrollbars and whenever it is smaller than the
scrollable, it communicates the sizes and the current position to the
scrollable. It is then the scrollable's job to render itself in a
suitable way. This is the current approach of GTK with
GtkScrolledWindow and GtkScrollable.
C) Let the viewport do the scrolling
In this approach, one treats the scrollable as just a large canvas and
the viewport provides a view onto an area of the scrollable. Scrolling
is managed completely by the viewport.
D) Implement scrolling automatically
Whenever a widget needs more space than is available, just
automatically add scrollbars. This is what the overflow property of
HTML does.

And then the question becomes: If we redo scrolling, which of these
solutions should we pick? And should we only pick one? Or should we do
it completely different?
Or should we just keep it? I can come up with a bunch of reasons why
we should not just keep it, that are specific to the current design:
- the GtkAdjustments are underspecified
Who sets what values on the adjustments? Who provides the adjustment?
What do the values on the adjustment even mean? 5 years ago having a
"page size" and a "step size" probably made sense, but today we have
smooth scrolling...
- the size request mechanism is unspecified
This was a huge oversight in the height-for-width code review. Nowhere
is it specified what the requested sizes mean, how they should be
interpreted and in what way scrolled windows should do layout of their
children. Also, up to today, nobody has even bothered porting any of
the existing scrollables to the layouting mechanism. I know of a few
cases in GTK code where the sizing code already special-cases
scrollables or viewports, and I do expect this to become more. It also
means that it is practically impossible to use scrollables without
putting them in a viewport, because their sizes will be completely
screwed up.
- the scrollables rely on GdkWindows
Scrolling is today exclusively implemented using GdkWindow. And we are
trying very hard to get rid of them. So we will need to touch all the
code anyway.
- the scrolling mechanism is not up to todays requirements
What requirements do exist? A lot. Let me enumerate some below.


The biggest problem I think for the current world of scrolling is that
requirements for scrolling have changed a lot from the days when GTK
competed with Windows 95. Touch devices in particular try a lot of
variations on scrolling for their user interfaces. So this is where I
will draw most of my examples from. I'll also try to list the
method(s) from above (enumerated with A, B, C, ...) that make
implementing that feature particularly easy or hard when there's a big
difference. And I'll try to list examples for where I've seen these
features exist or be requested.

1. Scrolling only parts of the scrollables
http://foo-projects.org/~benny/projects/thunar/images/preview-0.4.0rc1/rubberbanding.png
http://www.barisione.org/img/gtksourceview-demo.png
This is something we already support in a limited, but that is not
generally thought about much. It is worth pointing out however, as it
complicates a lot of things.
As can be seen in the screenshots, GtkTreeView does not scroll the
headers vertically and GtkTextView has the line numbers not scroll
horizontally.
Easy: A, B
Hard: C, D

2. Changing scrollables based on view
http://www.w3schools.com/cssref/tryit.asp?filename=trycss_position_fixed
This is really an extension of the point above, and is what's
basically known in CSS as "position:fixed". Parts of the HTML widget
need to be positioned relative to the viewport. This means that the
scrollable needs to know the position of the viewport, so it can
position things correctly.
Easy: A, B
Hard: C, D

3. Attaching widgets to the "edge"
http://www.youtube.com/watch?v=RxO6W06LzvM
http://seilo.geekyogre.com/uploads/2011/12/Screenshot-at-2011-12-03-024001.png
The idea is that you attach widgets to the end of the scrollable so
that you scroll them into view on demand and they don't show up in
your user interface while you interact with the main content. The
Youtube video demonstrates that nicely for phones. Depending on the
design, these widgets on the edge may or may not be included in the
scrollbar.
Easy: B, C
Hard: A, D

4. Detaching the scrollbars from the scrollables
http://dl.dropbox.com/u/7162902/shell-20111122-1.webm
Here you want the scrollbar to extend further than the scrollable
does, either to get a larger scrollbar or to indicate that certain UI
elements conceptually belong to the scrollable.
Easy: B, C
Hard: A, D

5. Inserting widgets between the scrollable and the viewport
http://meld.sourceforge.net/meld_file1.png
http://seilo.geekyogre.com/uploads/2011/12/Screenshot-at-2011-12-03-024001.png
This is somewhat of a generalization of the concepts from the 2 points
above. One creates a scrollable based on multiple scrollables (2
iconviews in the screenshot above) and potentially other widgets, but
uses only one scrollbar to navigate the contents.
Easy: C
Hard: A

6. Multiple views on the scrollable
https://maps.google.com
http://www.jiscdigitalmedia.ac.uk/images/gimp-navigation.jpg
Here two views exist on the contents: One is the regular view and
another one is a smaller, zoomed out view to ease navigation. This is
particularly useful when the contents provide useful clues for
navigation even when zoomed out a lot. Note that the GIMP shows the
whole contents of the scrollable, while Google Maps only shows an area
roughly twice as large as what you can already see.
Easy: C
Hard: A, D

7. Highlighting
http://www.techdreams.org/wp-content/uploads/2008/09/google-chrome-scrollbar-higlighting.png
http://meld.sourceforge.net/meld_file1.png
The idea here is to let the scrollable mark several areas of interest
so that it's easy to scroll to them in the viewport.
Easy: A
Hard: D

8. Feedback while scrolling
http://3.bp.blogspot.com/_jm4FARolmSo/S88VYT6sW2I/AAAAAAAAOWA/QpT9N8M88Uw/s1600/Scr002533.jpg
Another way that eases scrolling in large areas that I've seen on some
phones is feedback that pops up while scrolling or hovering over the
scrollbars. This of course requires interaction between scrollable and
viewport as context-relevant information should be displayed.
Easy: A
Hard: D

9. Different mechanisms for scrolling
No examples here, but I don't think those are necessary. Depending on
the device, there are different ways to scroll around in content.
Keyboards use certain keys to move "steps" or "pages". The mouse
generally uses scrollbars or the wheel. Touch devices - both touch
pads on laptops and touchscreens support various gestures, mostly for
kinetic scrolling. The question then becomes what kinds of information
these mechanisms need to effectively scroll the scrollables. In GTK,
adjustments provide a bunch of information, but is that enough?

10. Integration with other scrolling systems
http://webkitgtk.org
There are a few places out there, where scrolling systems exist and
interfaces we come up with should aim to integrate well with these
systems.
Easy: A

11. Different types of viewports
Depending on use case you want viewports to do different things, for example:
- support zoom in image editors or PDF readers
- show a navigation image in the GIMP
- support double click to zoom in/out on mobile
Providing these features in a generic way might be complex for a
single viewport implementation and might make a strong case for
allowing subclassing of viewports, or even providing multiple
viewports in GTK itself.
Easy: C
Hard: A, D

12. Displaying "infinite" amounts of data
http://www.novixys.com/Exult/Access_1021_01.jpg
https://mail.gnome.org/archives/gtk-devel-list/2009-June/msg00092.html
http://git.gnome.org/browse/gtk+/commit/?id=33ada9e9
https://bugzilla.gnome.org/show_bug.cgi?id=658995
try scrolling in/resizing http://popcon.debian.org/by_inst
In a lot of cases applications want to display a lot of "items". In a
lot of cases, viewports expect information from scrollables that
cannot be provided in a way that is not at least O(N). In my
experience performance problems often get noticable when N reaches
100s and can make applications easily unusable in the 10,000s. Various
hacks exist to delay things - you can observe this for example in
gedit when the scrollbar gets smaller while it loads a large document
or in the treeview when it's busy resizing columns - but it's also a
question on how you design the interaction between viewport and
scrollable.
Easy: A

13. Loading data lazily
https://bugzilla.mozilla.org/show_bug.cgi?id=574330
It is often not necessary to let the scrollable be initialized fully.
Only the parts that are actually visible need to be fully there. This
can provide very noticable performance improvements. But it requires
the scrollable to know which parts are visible - or will soon be
visible.
Easy: A

14. Loading data while scrolling
http://slashdot.org/
http://www.novixys.com/Exult/Access_1021_01.jpg
Many applications - mostly those with database backends - allow
loading more data as you scroll. In fact, they might not even know how
much data still remains to be loaded. This again however requires a
careful design of the interaction between scrollable and viewport, so
that loading all data is not necessary and newly loaded data gets
input into the system as seamless as possible.
Easy: A
Hard: C

15. zoom
http://www.ted.com/talks/blaise_aguera_y_arcas_demos_photosynth.html
http://www.youtube.com/watch?v=Ow7OHOIbTsQ
Zooming is a rather interesting part. It is not generally thought
about as related scrolling, but is often using the same infrastructure
in the code, so it makes sense to think about it. It also provides
unique ways to navigate large scrollables as can be seen by the TED
video linked above or by the way people generally use maps
applications (zoom out, scroll to where you want, zoom in a bit, move
even closer, ...).
That said, even for the cases where scrolling is used today (EOG,
Evince, GIMP, ...), the applications could benefit from shared
functionality to implement zooming. With all of this, one should keep
in mind that zoomability touches a lot of the points brought up above.
Easy: A, C

16. non-affine transformations
http://en.wikipedia.org/wiki/Cover_Flow
I'm not sure how far out of the "scrolling" topic we are by now, but
it's been requested a lot when people saw iTunes for the first time. I
don't think I need to explain it any more than the wikipedia page.
Easy: A

?. your feature here
I'm sure I forgot something, so feel free to point it out before we
design the new scrolling.
Easy: A


So, if you followed along up to here, you'll surely have an opinion.
So please go ahead. Ideally, you'd propose an API that solves all
these issues and make me stop thinking about it.

Benjamin