Re: gnome-continuous: speeding up "builddisks"

[Bastien Nocera <hadess hadess net>]

On Wed, 2016-08-31 at 09:37 -0400, Owen Taylor wrote:
> [ This mail is somewhat specific to gnome-continuous, but CC'ing the
> Â  ostree mailing list, since the question of how to efficiently
> deploy
> Â  ostree content into a VM is general ]
<snip>
> So this means that we should be accessing the image filesystem
> through
> a VM. Which also has the advantage of being able to work directly
> with
> a qcow, rather than having to do a conversion step later. If the
> guestfs approach doesn't work, what options are there?
>
> Â 1) Do the ostree operations outside the container on the host
> Â  Â  filesystem, then copy the entire results into the container.
>
> Â Â Â Â The main problem with this is one of filesystem ownership: as
> Â  Â  non-root, we can't create files with the ownership that will be
> Â  Â  needed inside the VM.
"fakeroot" can save those permissions, which you can then reapply
within the container, avoiding manual permission fixes.

>  It would be possible to enhance ostree to
> Â  Â  write ownership as xattrs, and then convert when copying into the
> Â  Â  VM, or to even have ostree directly stream a tarfile to be
> unpacked
> Â  Â  in the VM, but either would be significant work.
>
> Â Â Â Â A secondary issue is that this only applies to the initial
> Â Â Â Â creation of the VM, and doesn't allow for optimizing updates.
Do we need to optimise updates if the initial image creation is fast
enough, or would the created image not be updatable if we went that
way?

> Â 2) Do the ostree operations inside a VM with the image mounted.
>
> Â Â Â Â The issue here is that we need an operating system with ostree
> Â Â Â Â in it to do the deployment, causing a bootstrapping issue - we
> Â Â Â Â need an operating system to make an operating system. Using
> Â Â Â Â a fixed non-GNOME-continuous operating system image for this
> Â Â Â Â would be possible, though with some potential for ostree version
> Â Â Â Â skew.
>
> Â Â Â Â But the approach that I prototyped out is creating a
> Â Â Â Â "bootstrap" image by checking out the destination ostree,
> Â Â Â Â copying it into the VM at / instead of as a proper ostree
> Â Â Â Â deployment, then doing the necessary fixups to make that boot.
>     (E.g., adding a fstab that mounts a tmpfs on /var.) This is a 
> Â  Â  bit hacky, but not too bad.
>
> Â Â Â Â (Another approach would be to do this from the initramfs but
> Â Â Â Â getting ostree into the initramfs would require a gigantic
> Â Â Â Â pile of libraries.)
ostree + glib + libsoup (and glib-networking, libproxy, I guess).
That's probably not any worse than when we put Xorg in the initramfs to
do boot splashes :)

> Â Â Â Â Once you have that bootstrap image, it can pull from the
> repository
> Â Â Â Â on the host over HTTP, or by exporting the repository via 9p.
>
> Timing for just the x86_64-runtime image. (This is testing on my Core
> i5 laptop with the IO on an external SSD via USB 3.0)
>
> Â Current method:
> Â  Initial creation: 40m
> Â  Small update (modified 45 files): 7m50s
>
> Â Operations outside, copy in: (hacking file ownership issues)
> Â  Initial creation: 1m41s
> Â  Small update: N/A
>
> Â Operations inside, pull via 9p:
> Â  Initial creation: 2m1s
> Â  Small update: 37s
>
> Â Operations inside, pull via http (lighttpd, bridged networking):
> Â  Initial creation: 2m0s
> Â  Small update: 35s
>
> Creating the bootstrap image from scratch takes 24s; there's no real
> reason to recreate it except for changes to ostree or syslinux,
> though
> not recreating at all would risk other aspects of it bit-rotting.
>
> Other than the current method, everything is pretty close. Pulling
> via
> 9p is significantly easier than the HTTP version since it avoids
> complications of setting up networking - I wouldn't expect the
> timings
> to be competitive with user-space networking, though I didn't
> actually
> measure that.
>
> Does this seem like something that is worth pursuing? Do people have
> any other ideas for improvements?
Is the problem an I/O throughput problem, or a CPU throughput problem? 
If it is the former, maybe we could still use guestfs but with the
image in RAM.
If it's a CPU problem, we could possibly use a VM with a physical
partition instead of an image, so that direct I/O kicks in (libguestfs
uses a VM internally, but I'm not sure it can do direct I/O, especially
on top of an image).

HTH