Follow up about X clipboard

[Jerry Haltom <wasabi larvalstage net>]

It's been about 4 days now since I sent my first mail outlining ONE idea
for persisting state with the X clipboard. A lot of responses were
received. The majority of them, of course, were negative. I will sum up
these negative reasons and offer counter arguments for them.


1. Non-deterministic shutdown of applications is confusing.

The basis for this argument is that when a user closes the last window
of an application, they expect that process to terminate... as in no
longer be running. The main problem I see with this argument is that
users today have absolutely no concept of what a "process" is anyways.
They have no need to know this concept. A user sees a window, with some
data in it, they do not understand that this window is backed by a OS
concept called a process. Thus, it is not confusing for users at all.
For technical users however, the argument has merit. We understand what
a process is, and naturally we expect that process to end when a window
is closed. This however isn't a technical argument, but more one of
expectations on our part. It's something we can get over, being
technical.


2. Non-deterministic shutdown leaves possibly broken applications
running with no option in the GUI to terminate them.

Good one! After an application closes all it's windows, and frees all
memory required for them (or whatever), it enters a sort of daemon
state, where it keeps running until it no longer owns the clipboard.
What happens however, if during this daemon state, or perhaps while
entering or exiting it, the application locks up in such a way as to
consume 100% of the CPU, etc. First off, this is a bug on the
applications part, and it needs to be fixed. However, that doesn't stop
our architecture from needing to be robust enough to handle a failure
like this gracefully.

During this state, the X server still has a handle to this application,
in the form of whatever connection the X server will use to pull the
clipboard later. Thus, we can expect that the application is still
reachable over this handle in some fashion. I want you to keep in mind I
really have no idea how the X clipboard works internally, so this might
be off base. Since the application owns the clipboard, there can be a
process which can attempt to "probe" the clipboard at some sort of
interval. For example, gnome-session could detect that an application
now owns the clipboard, and then ask it for some information about what
it's holding on a timed basis. If the application does not respond in a
reasonable amount of time, a dialog box can be shown, as is shown now
when attempting to close a frozen application. Preserving clipboard
contents in a case such as this is not important: an application
crashed.


3. Some applications are big. It makes no sense to keep them running in
the background forever.

Yes it does. The user has indicated he WANTS to copy (make available
later) a certain amount of data. This means the system is obligated to
either not offer him this feature, or preserve the data in some fashion.
For some applications such as OpenOffice, this might be pretty large!
For example, if somebody copies a range of 100x100 spreadsheet
columns/tabs, the program would need to preserve all data in memory
required to get that data into the paste format desired. This is
probably a substantial amount. However, OpenOffice will not have any
open windows, or GTK widgets hanging around. So one must weigh this
against the benefit of having a clipboard that works. There is also no
reason why the "daemonized" version of OpenOffice cannot be resurrected
if the user opens another OpenOffice document later.

Additionally, while there is data available on the clipboard, a
notification applet icon can be present. This icon can describe the type
of data available (mime->description?), the size perhaps, and present
the user the ability to discard the data. Is there a way to discard the
data? If not, the applet could just copy nothing. Some people seem to
have a lot against notification area icons, so we'll see how this flies.

Furthermore, clipboard data can and should have a timeout. There is no
reason to keep a 100MB copied bitmap image around until the user decides
to copy something else. A reasonable time that will not interfere with a
user's activities needs to be chosen. The majority of clipboard
interaction is very fast. The user copies, and then he pastes. There are
also scenarios where the user copies, to save some data from
destruction, deletes the original, and then pastes. Or the user copying
once, and then pasting 100 times in a production line fashion. This
timeout time can be variably based on the type of data being copied or
the amount of it.


4. Why don't we just inform the user the clipboard or an amount of
clipboard data will be lost when he closes a window?

This is directly involving the user in a decision involving a deficiency
in the implementation. The user does not know what "data types" are. The
user does not know why he will lose data. This is a distraction in the
user's work flow. Copy paste needs to work as expected. Copy and then
later, paste. Forcing the user to remember arcane rules about making
sure to keep the originating window open is silly. (Does a timeout fall
under arcane rules?)


5. Remove copy paste menus totally and use traditional X clipboard.
There is no problem.

I need to answer this one. This suggestion seems to undermine the points
of Gnome itself. The reasons we created a HIG. A user needs to be able
to interact with his software in the easiest way possible, without being
exposed to implementation details.

Additionally, not having Copy/Paste menus gives no point to integrate
accessibility for those unable to use a mouse to select some text the "X
way".


6. Long running applications with a non-deterministic shutdown provide
more points for fault.

Of course. Any ability we add to anything provides a fault point. I'd
like to believe, based on the other points 1-5 mentioned above, that
such fault points are not as disastrous as they are made out to be.


7. Is there a requirement that the clipboard mechanism must be
compatible with current applications?  If so, then you end up tying your
hands to the point of not being able to successfully get to where you
want to go.

There is never any such requirement. Being able to fulfill such a
requirement however, without sacrificing the design of your code is
wonderful. It means existing applications continue to work exactly as
they do now, including with "new" applications.


8. Why not implement some sort of Clipboard Daemon (clipd) which has the
logic available to it, either through plug-ins or natively to transform
the data from one type to another.

Consideration of the size of future/present data sets. Linux/Unix today
doesn't have any good examples of widespread software such as
audio/movie editing. However, the ability for this type of data to work
with the system must be addressed. This type of data is generally LARGE.

Consider a user who opens up a MP3 in an audio editor, and then opens up
a WAV in another, different, audio editor. The user wants to be able to
select 60 minutes from the MP3 and paste it into the WAV. The MP3
program supports audio/mpg and audio/wav. The WAV program however only
supports audio/wav (if it didn't, it would be an MP3 program instead!)
When the user copies the 60 minutes from the MP3 program, the MP3
program either a) copies the 60 minutes to an in memory buffer, or b)
marks the time frame in it's existing data/memory as being "copied". b
is definitely preferred obviously, as it takes no time. The user then
goes to the WAV program and attempts to paste. The WAV program sees the
two types advertises, and decides he wants the audio/wav type. He asks
the X server to give him this data. The MP3 program converts the
selected MP3 into a WAV file, and sends it. The final data set being
transfered is 600MB.

Yes, this is a massive amount of data, and no, the current X clipboard
implementation does not support this (should not?) It would take forever
to transfer. However, the two applications are able to preform this
action if they have a proper transport between the two. The X transport
could be upgraded independently of the applications themselves to make
such a data set feasible.

However, in comes a clipd. When the initial copy is made, essentially,
600MB is copied to the clipd. Additionally, what if the user wanted to
paste his audio data into a MP3 program instead of a WAV program. Either
you can a) send both the MP3 and WAV from the first audio program to the
clipd, resulting in 660MB data, and then 60 more when the user pastes,
or b) re-encode the WAV data from the clipd into a MP3 by the second MP3
program, resulting in lost quality (without the user being aware of it!
BAD!)

In the publish/pull model however, the MP3->MP3 copy/paste example would
only transfer 60MB, the original MP3.

Additionally, it is up to the copying program how it implements
copy/paste in accordance with it's dataset. In the previous example, the
act of copying consists of nothing more than marking the beginning and
end times of the sound sample, and doing no actual data copying until
the actual paste... which by the way, may never occur.

Copying a large data set like that necessitates storing it SOMEPLACE, so
it is required that 60MB (the copied MP3) be kept around in memory until
it is needed, or times out. It is up to the MP3 program how it wants to
handle it's own data sets. For such large data, a program might already
have a memory->swap space archetecture already built in, which it can
reuse so the 60MB doesn't need to be in memory all the time. The GIMP is
a fine example of this.

When it comes right down to it, the idea of a clipd for large data sets
is not feasible. As a backup for text/* however, it might be just fine.
But, in no way should we limit our future clipboard to such things.
Windows and Mac already allow you to copy such large datasets between
programs.


9. Translator framework like BeOS?

I don't know enough about this idea to really totally understand it...
but I will try.

Every application would provide modules to the system that allow its
internal data representations to be transformed to other formats. These
modules could run independently from the apps, so that a module could be
used to hold the clipboard content after an application is closed
(really closed). Such a framework additionally would allow much greater
import/export capabilities in all programs.

Back to our MP3/WAV example. The system would have plug-ins (in some
form), such as wav2mp3 and mp32wav. Most likely these plug-ins would be
installed in some central place such as /usr/share/translators or some
such. When a user copies from the MP3 program, the actual MP3 data goes
to the clipd (60MB transfer, which may never be pasted, wasted time).
When the user pastes, the pasting application asks for the data in the
audio/wav format. The clipd finds it's mp32wav plug-in, passes the data
through it, and outputs a WAV file. The WAV file is passed to the WAV
program (600MB transfered). In total, 660MB was transfered between
software. Additionally, the WAV program itself could receive audio/mp3,
and pass it to the mp32wav plug-in without even knowing what MP3 was
(120MB transfer to/from clipd) That suffers from a deficiency, it
totally breaks with certain remote X app configurations.

We have the privilege(!) to be sure our software works with remote X
configurations just as well as it does with local X configurations. Lots
of workstation installs probably use REALLY BIG servers to power lots of
workstations, doing audio/cad manipulation.

The MP3 program itself comes with the mp32wav and wav2mp3 plug-ins. Both
the MP3 program and the WAV program run on the same system along with
the clipd, then all is well. They both have access to the plug-ins.
However, if the MP3 program is on a remote system, and the WAV program
is on the local system, then only the remote system has the mp32wav
program. The WAV program cannot do the conversion. If the clipd is on
the local system, it can't either.

Every software developer would have to agree on formats and converter
implementations. If two people wanted to make a audio/mp3 encoder, which
would be used? BeOS had the benefit of a central authority/company
dictating what could be done. We don't have such luxury, and we
shouldn't need to.

All of this is blissfully ignoring how slow it would be to copy 60MB, or
600MB between remote X servers, heck, remote anything. This however goes
right back to the implementation of a better transport between
applications. And, these days, with gigabit networking showing up, such
a scenario might not be as unreasonable for much longer.


A possible implementation idea for GTK specifically was mentioned by
Soeren Sandmann: One downside is that the necessary application changes
probably can't be done automatically by changing GTK+. Just adding
gtk_main_quit_after_losing_selection() and have applications call that
instead of gtk_main_quit() might be enough, though.

I would propose something more like gtk_main_quit_interactive(), which
says basically "quit being interactive". Is it possible
gtk_main_quit_interactive() could support restarting the main loop if
the user wanted the application back after having already closed it, but
it is still hanging around. This would mean we could reuse processes
that didn't close... instead of having to create a new instance in these
cases.


Here is a little bit of soapbox about how, to me, it feels as if
applications should be implemented.

Programs, are servers, essentially. They are servers which provide
services such as text editing, audio manipulation, email reading. They
are manifested to the user as UI elements, but they are not purely so.
The UI component of a program is simply one of the services it provides
to the user. The UI component of a piece of software does not need to be
directly tied to the lifetime of the program itself. The lifetime of a
program should only be tied to how long it's "doing something useful".
This does not ignore the principle that when a piece of software is not
doing anything useful, it should terminate to save memory for the
system. It just extends the definition of "doing something useful" from
"showing a window" to "providing data to the user". I believe this
abstraction is genuinely useful.


Thanks for all your replies. Glad lots of people got involved.

-- 
Jerry Haltom <wasabi larvalstage net>

Please avoid sending me Word or PowerPoint attachments.
See http://www.fsf.org/philosophy/no-word-attachments.html