Re: File monitor rewrite: Solaris (and other) help wanted

[Ryan Lortie <desrt desrt ca>]

hi Aleksander,

On Thu, Jan 15, 2015, at 06:28, Aleksander Morgado wrote:
> Currently GFileMonitor doesn't have a unique way to know whether a file
> got closed. There is the changes-done-hint event, but that covers IIRC 2
> things: files getting closed and also a "virtual" emission which happens
> after some time even if the files were not closed (think of a log file
> which never gets closed). The main issue is that if you want to get
> notified of when a file was fully updated *and* closed, you need to
> fallback to raw inotify. The rationale for wanting to get notified only
> when the file got closed is e.g. Tracker monitoring the Downloads/
> directory. We may not want to extract file info for an ongoing download,
> just for when the download is fully finished (and destination file
> closed). More background here:
> https://bugzilla.gnome.org/show_bug.cgi?id=635765

Short story: I want to add a flag to either disable or enable emission
of virtual changes-done hints on monitor backends that can reliably
handle it themselves.

Even for fully-capable backends, I think virtual emissions are
potentially important because, even if the file is not closed, someone
watching it may want to update their opinion of its contents
periodically.  The question is only about what the default should be. 
Those who favour a nice clean API would say that virtual emissions
should be off by default.  Those who favour backwards compatibility
would suggest that today's behaviour of virtual emissions should be kept
as-is unless explicitly disabled.  I'm not sure what we will do.

Unfortunately, there's a longer story: None of the backends support
reliable emission of non-virtual changes-done.

Here's why:

My plan is to make it a guarantee of the API that both CREATED and
CHANGED events will always be followed by a CHANGES_DONE hint.  That's
already enforced in the state machine logic in GFileMonitorSource in the
branch.  My reason for that is that apps like Tracker should not want to
response to CREATED events until the file content is complete.

The idea (taking your download example) is that a file is created
something like so:

  creat()
  write()
  write()
  close()

which sends us IN_CREATE, IN_MODIFY, IN_MODIFY, IN_CLOSE_WRITE.

As you mention, it doesn't make sense for the app to respond to the
empty (or maybe very slightly populated) download just because it saw a
CREATED event from GFileMonitor -- it should wait for the CHANGES_DONE. 
Consider this case:

  creat()
  close()

in that case, we'd see IN_CREATE, IN_CLOSE_WRITE, with no IN_MODIFY.  We
still want to see a CHANGES_DONE event in that case, though so that the
app knows that the empty file is the 'final result'.  This is the basis
of my opinion that CREATED should always get a CHANGES_DONE after it,
even without actual CHANGED events.

With the new support for move and rename events, a file created by the
"write to temp and mv into place" method will be reported either as
MOVED_IN or RENAMED with no CHANGES_DONE.  That's okay, because you know
that a file that was MOVED_IN or RENAMED into place is ready to be
handled immediately.

Unfortunately, there is another set of cases.  IN_CREATE is sent both
for the creat() case (in which case it will be followed by
IN_CLOSE_WRITE) but also for cases like mknod(), bind() on a unix
socket, mkdir(), etc..  In those cases, we will receive no
IN_CLOSE_WRITE.  We could detect that situation by looking at the
filesystem and seeing that the newly-created file is of a special type,
but link() also produces IN_CREATE without IN_CLOSE_WRITE.

The only thing that saves us in this second case is that we get a
virtual emission of CHANGES_DONE a couple of seconds later.  At least
link() is uncommon, although it stands to become a more common way of
creating files, via O_TMPFILE.

In short, I believe that we currently don't have any backend for which
we could safely disable emission of virtual CHANGES_DONE events. 
Ideally, in the future, we will gain a mechanism in inotify to tell the
difference between "file created via open for writing, IN_CLOSE_WRITE
coming soon" and "inode appeared in the file system in complete form". 
At that point we could expose a new event type in GFileMonitor
(_APPEARED?) that doesn't need CHANGES_DONE to be emitted.

Cheers