Re: Plans for gnome-vfs replacement

[Alexander Larsson <alexl redhat com>]

On Mon, 2006-09-18 at 23:06 -0400, Dan Winship wrote:
> Cool.
> 
> Alexander Larsson wrote:
> > At some point in time gnome_vfs_uri_is_local() started detecting and
> > returning TRUE for NFS mounts and other type of local network
> > mounts. This is both slow and unexpected, and has led to problems and
> > unnecessary changes in many places. 
> 
> s/TRUE/FALSE/, I think. (gnome_vfs_uri_is_local() used to say that NFS
> filesystems were local, now it says they're not.) Are you suggesting
> that this behavior is going to go away? It's useful in places as a
> heuristic, even if it's not 100% reliable. (Although obviously it's only
> useful if it's guaranteed to not hang.)

Oh, yeah FALSE. I'm not really suggesting anything in particular here.
Just describing a problem. Its true that there are cases where you want
to know whethe a file is network mounted or not, but almost all cases
that were using gnome_vfs_uri_is_local() meant it to check for a
file:/// uri. The fact that it changed behaviour meant a lot of places
suddenly broke on NFS shares, and were made a lot slower (since it
suddenly is a call that does i/o). I've personally changed a lot of code
to do a g_str_has_prefix("file://") in lots of places.

> > As a consequence of the stateful model we don't need the stateless
> > properties that URIs has as identifier. To avoid all the problems
> > comming from the use of URIs  we use a much simpler form of
> > identifier. Namely filenames
> 
> We still need to support URIs too at least in some places, because of
> '%u' in .desktop files. If GNOME apps switched to using '%f', then
> konqueror (and old versions of GNOME) wouldn't be able to pass remote
> files to them any more. Likewise, if nautilus/libgnomedesktop didn't
> support using URIs for remote files any more, then they wouldn't be able
> to pass remote files to KDE apps.

Clearly we would somehow need to mark the desktop files such that we
know they support the "new vfs", whatever system we use. But using
filenames doesn't make it impossible to pass uris to old gnome apps or
kde apps. In fact, that is one of the strengths of the idea. We just
pass them a "gvfs:///XXX" uri, and then the gvfs implementations in
gnome-vfs and kio automatically makes this work for all old apps.

The usage of %u and %f to mark what types of arguments to pass is
problematic in itself. For example a gnome-vfs app and a kio app both
specify %u, but they can't open the same set of uris. It also doesn't
allow us to mark an app with "only opens local files and http files",
something which is very common with non-gnome/kde apps like say mplayer.

We really need a better way to handle this.

> > I imagine all/most actual mounting
> > of shares will happen in the file manager and the file selector, or at
> > gnome-session startup, so applications don't really need to handle
> > this themselves.
> 
> Hm... "Recent files" may be on unmounted shares. Or in the konqueror
> case above, konq would have the remote fs mounted as a kioslave, but
> when it passed the URI to a GNOME app, that app would need to do a gvfs
> mount in order to read the file.

Recent files may be on an unmounted filesystem too. What do we do in
that case? If you open them from the fileselector maybe it can mount
first, same with the panel and nautilus.

Also, the kde URI case above is an example how using URIs automatically
lead you to a stateless model, which we're trying to get out of. I don't
have perfect answers here, but I'm pretty sure that having each i/o
operation able to result in blocking, possibly modal recursive dialogs
is just wrong.

> > For example, we could be to use "//.network/" as a prefix for the vfs
> > filename namespace.
> 
> Ew. OK, what's the idea with the fake-paths-instead-of-fake-URIs thing?
> As points against URIs, you say:
> 
>     1. Using non-standard ones is evil.
>     2. gnome-vfs uses broken pathname-handling semantics to make things
>        easier for applications.
>     3. Escaping and Unescaping is hard, but people want to do it to
>        make pretty-looking names.
>     4. It makes people think gnome-vfs is more web-browsery than it
>        really is.
> 
> But (1) also sorta applies to using things that look like file paths but
> aren't, (2) seems like it ought to be covered by GFile ("This means you
> don't have to do tedious string operations on the pathnames to navigate
> the filesystem."), and (3) seems like it's covered by the display name
> thing ("These filenames would be ... not really presentable to the user
> as is. You'd need to ask for the display name via the vfs to get a user
> readable utf8-encoded string for display.").
> 
> Another point in favor of paths over URIs might be "you can share the
> same representation between gvfs-aware and gvfs-naive apps (if you have
> FUSE)", but with the representation you've chosen, you don't even get
> that; you have to use a different path when talking to gvfs-naive apps.

I initially wanted to use $home/.network as the prefix for the vfs
"filenames". However, that is problematic, as such filename refer to the
home directory which can be different on another machine, for another
user or for another process. This means such filenames can't be
transfered, nor stored.

> For points in favor of URIs, there's the fact that KDE uses them,
> various fdo standards use them, and various existing GNOME APIs use them
> (eg, the recent files api mentioned before).
> 
> So are paths really better than URIs? (I'm totally willing to believe
> the answer is "yes", I just don't see it yet.)

There are some advantages to using paths. For instance, that model
suggests a "mount-required" stateful model, whereas using URIs tends to
suggest a stateless model. It also lets us easily do backwards
compatibility by exposing the whole new model as a single uri method in
the old systems (we can do this, because since we mount stuff you can
fully enumerate the root of such a gvfs:/// method). Another issue is
that the win32 vfs (the windows shell api) isn't uri-based, and we need
to handle that if we want to wrap it on win32.

Of course, as you mention, URIs have advantages too. And some of its
problems might be solvable. For examples we could rename them to
something like VRIs to distance ourselves from web URIs.

Thinking more closely on this it seems that neither plain paths or URIs
are really good enough. For instance, even local filenames need some
sort of escaping for us to be able to display them, since they might
contain binary data (everything but zero and slash is allowed), and we
really want to display e.g. korean smb uris with readable text instead
of crazy escape codes.

Lets take a step back and define the requirements we have:

We need to have a form of identifier for files. These come in two forms,
absolute and relative (e.g. name of a child to a folder). They must be
persistable (if we save a reference somewhere it must make sense on a
later login) and transportable (if we pass a reference via e.g. a
desktop link to another machine or another user it must make some sort
of sense). The exact contents of such identifiers have no special
requirements, in fact they could theoretically be binary blobs for all
users care, but it does seem nice if we could use something as close as
possible to filenames for local files.

We also need to have display names. The main property of display names
are that they are valid UTF-8, which means that they are strings that
can be displayed in the user interface and edited in an entry. A
secondary property is that they tend to be "more understandable" to
users.

Display names are used in a variety of ways, with different
requirements. They can be used for an absolute name in e.g. a location
entry like in nautilus. For this we need a form that must survive a
roundtrip from identifier to display name and back without modifying the
identifier. We also need to be able to represent most local filenames by
using the traditional unix pathname. 

Another form is the relative display name, that is used for instance in
each icon in a nautilus view (including when renaming files). These
doesn't have to be round-trippable per se, but when we use them to
rename a file to a new name the new name will have to be mapped back to
a filename in some way. (Although some backends, like db-backed ones or
desktop files, might choose save the new display name without actually
changing the identifier.) 

We also want display names that are not absolute, but neither are they
relative to their parent in the traditional way. For instance, the
parent of //.network/smb/server:share would be //.network/smb, and its
relative display name in that parent folder would be "server:share".
However, when displaying this in say the nautilus window title we want
to use something like "shared folder share on server" (but we want it
translated).

MSDN has some docs on how windows shell solves this issue here:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/shellcc/platform/shell/reference/enums/shgno.asp
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/shellcc/platform/shell/reference/ifaces/ishellfolder/parsedisplayname.asp
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/shellcc/platform/shell/reference/ifaces/ishellfolder/getdisplaynameof.asp
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/shellcc/platform/shell/reference/ifaces/ishellfolder/setnameof.asp

Its an interesting read.

I'll try to think some more on this. Hopefully we can come up with a
good solution.

> If gvfs deviates too far from POSIX, then it's the http-method problem
> all over again; the gnome-vfs/kio/FUSE modules will have to have hacks
> to deal with the non-POSIX semantics of the gvfs filesystem. So the
> hacks like write-whole-file-on-close still need to exist somewhere,
> although at least you'll never hit them if you use the preferred APIs.

Unfortunately, yes. Hopefully we can minimize the pain.

> > Furthermore, if FUSE is supported on the system we can write a FUSE
> > filesystem so that we can access the files as $HOME/.network/XXX.
> 
> Rather than having both a vfs daemon and a FUSE daemon, it seems like it
> ought to be possible to squish them both into the same process.
> "libgvfsbackend" or whatever would link to libfuse, receive the POSIX
> requests from the kernel, make the appropriate not-quite-POSIX gvfs
> requests of the backend (via local procedure calls, not dbus), and
> perform whatever unfortunate hacks were necessary to get POSIX-like
> semantics.

This is more or less a technical question. It depends on things like the
details of libfuse or the plattform independence of the fuse kernel api
if we can't use libfuse. It might be possible, or it might not.

> Then the gnome-vfs and kio bridges could also use this more POSIXy
> interface (presumably via dbus), so that they could get POSIX-like
> semantics too without needing to reimplement the kludges.
> 
> And actually, for the more filesystem-like backends (smb, etc), we could
> just let the backend implement the POSIX interfaces itself, in addition
> to the gvfs ones, to avoid the need for kludges when accessed via
> FUSE/gnome-vfs/kio. Then libgvfsbackend would just have default kludgey
> methods for anything the backend didn't implement natively...

I'm not sure they need the exact same workarounds. For instance, kio is
higher level than posix. But if there is some things that can be shared
that would be nice.

> > I've been doing some initial sketching of the glib API, and I've
> > started by introducing base GInputStream and GOutputStream similar to
> > the stream objects in Java and .Net. These have async i/o support and
> > will make the API for reading and writing files nicer and more
> > modern. There is also a GSeekable interface that streams can
> > optionally implement if they support seeking.
> 
> How do these relate to GIOChannels? Part of why I ask is that
> GIOChannels on Windows apparently have issues. (I don't know all the
> details, but see Tor's gtk-devel mail from Saturday for an example.)
> Someone with Windows clue should pay attention to these APIs to make
> sure those problems can be avoided here (if possible).

A GIOChannel is an implementation. Its a way to read from a unix file
descriptor with optional buffering and charset encoding handling. You
create one by calling g_io_channel_unix_new (int fd). The problems
related to windows is that an "int fd" isn't really the same thing in
windows as in unix.

A GInputStream is an abstract baseclass for a stream. You can't really
create one, and they support no features. However, you can create any
type of input stream (== class that inherits from GInputStream) and pass
it to any code that reads from a GInputStream. For instance, one could
have g_file_input_stream_new(char *filename). If you are on the
consuming side of the stream and want specific features you can chain
the streams. For instance you could use something like GInputStream
*g_buffered_input_stream (in_stream) if you need buffering. 

The nice thing about this approach is that when you call something like
GInputStream *g_file_read(GFile *file) the active vfs module can just
create a stream of the right sort and return that, and the user doesn't
have to know if its a plain GFileInputStream or something that talks to
the vfs daemon via dbus.

Java also has things like DataInputStream that takes any input streams
and makes it easy to read binary data (like read_uint32()) from it. So,
apps just create a file stream, wrap it in a buffered stream, and then
wrap that in a DataInputStream to get a nice efficient way to read
binary data.

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 Alexander Larsson                                            Red Hat, Inc 
                   alexl redhat com    alla lysator liu se 
He's an all-American hunchbacked shaman from the Mississippi delta. She's a 
plucky antique-collecting barmaid who can talk to animals. They fight crime!