Re: [atomic-devel] Proposing a new way to deliver Atomic systems: rpm-ostree jigdo ♲📦

[Petros Angelatos <petrosagg gmail com>]

Hi all,

The docker delta situation might not be as unsolvable as it seems.
I've been working on Balena[1], a moby based container engine focused
on IoT usecases. One of its main features is the ability to pull
images using binary deltas in a way that produces bit-identical copies
of layers and image metadata. The deltas are based on librsync, which
I ported to Go, but other algorithms could be added in the future like
bsdiff. It currently achieves 10x-71x reduction in payload size
compared to docker pull[2].

I completely agree that "checksumming compressed content" was a
mistake. It does give the advantage of not decompressing potentially
malicious data before verifying their authenticity, but I think it's
not worth it. One of the challenges is around publishing a digital
signature for an image and still being able to verify it on the client
after having downloaded it with deltas. The good news is that notary
image signatures are implemented completely on the client side (the
Docker CLI) and that the notary server doesn't know what the hash that
is being signed represents. This means one can ignore the compressed
checksums completely and work with the image id, which is the hash of
all uncompressed layers plus metadata. This is how we plan to verify
delta pulled images.

> Now while the "obvious" thing would be to try to align with OCI in some way, the complete lack of wire 
> deltas there is very problematic for uses outside of server clusters, and given that we already have lots 
> of extensive linkage to RPM via libdnf, it makes the most sense to move that direction.

Where I'm getting with this needs another piece of information that is
not on the balena website, but we've basically implemented a bootable
container system with it used in resinOS. It's very similar to OSTree
in that the sysroot only contains a docker-like directory and some
metadata about what is the active sysroot. Then, a special statically
linked PID 1 will load the container sysroot, do a pivot root, and
exec to /sbin/init of the new root. This means signed, atomic updates
with binary deltas using docker images.

To conclude my thoughts, an alignment with OCI could be using OSTree
as a storage driver for balena. More concretely, I see the current
balena deltas providing bandwidth efficiency and OSTree providing
storage efficiency using the hardlink farm approach.

I'd love to hear your thoughts on this. It looks like we're moving
towards the same goals and it would be great if we could work on
something together.

[1] https://github.com/resin-os/balena
[2] https://resin.io/blog/announcing-balena-a-moby-based-container-engine-for-iot/#technical_comparison
[3] https://github.com/resin-os/meta-resin/pull/805

-- 
Petros Angelatos
CTO & Founder, Resin.io
BA81 DC1C D900 9B24 2F88  6FDD 4404 DDEE 92BF 1079