GNOME.org
 

Re: Releasing binaries from BuildStream to the world

Hi Sam,


On Sep 28, 2017, at 9:42 PM, Sam Thursfield <sam thursfield codethink co uk> wrote:

Hello,

I've been doing some work on switching the Baserock project[1] to use
BuildStream as its build tool.

Existing Baserock build tools allow bootstrapping a compiler and sysroot
using tools from the host. BuildStream doesn't allow this, which means
that Baserock needs to start releasing sysroot binaries somewhere that
we can reference in our builds.


Im glad this conclusion has been reached.

Things for baserock will look all around nicer when producing the very binaries needed to get off the ground, 
full circle :)


I have 2 questions on how best to do this which are relevant to the
wider BuildStream community because deploying binaries is something
we'll pretty much all need to do.

1. How and where should we host these binaries? How do we transfer them
  there once the build completes?





2. How do we record the provenance of those binaries?

Hosting
-------

The sensible options for hosting a sysroot seem to be:

 a. tar.gz on a web server
 b. a tree in an OSTree repo
 c. Docker image in a Docker registry

OSTree is my preferred option as it supports file-level deduplication,
and we already have infrastructure in place to host OSTree repos.


Seeing as baserock itself is strictly linux, I would also choose OSTree (as it would always be available on a 
baserock system), otherwise I would go for tarballs.


But how do we get the files from BuildStream to the OSTree repository?


Using `bst checkout` somewhere and then committing the checkout to an OSTree repository.

Remember we only really mutate data (often by compiling it) but intentionally treat deployment as a separate 
problem space.


One option is to "re-use" our OSTree cache as a releases repo. In fact
I'm proposing that as our interim solution[2] but there are two issues
with it. Firstly it makes the cache no longer be a cache -- we'll need
to ensure that the release artifacts never get deleted and are backed
up and that's much more difficult if they're mixed in with 100s of GBs
of less useful artifacts. Secondly, the BuildStream artifact format is
effectively an implementation detail that may change in future. The ref
names used by BuildStream are not optimized for human readability. An
artifact may not necessarily contain all the files we need. Also,
BuildStream artifacts (in its current format) don't carry full
provenance info -- you can't see what the source repo and source commit
was just by looking at its embedded metadata.

If we don't re-use BuildStream's artifact sharing functionality then
we need some other way of getting the binaries from BuildStream's local
cache to our public releases repo.


I am skeptical of using an artifact cache share as a means of distribution/hosting in this way.

We may end up addressing your concerns of deletion, however what you propose seems to heavily rely on 
buildstream's internal artifact cache structure - I wont even try to guarantee that this structure will never 
change, only that buildstream will continue to be able to interface with its caches and properly 
push/pull/checkout etc.



We could resurrect "ostree-push"[3] for this perhaps? Our release
workflow would then become:

   bst build bootstrap/stage3-sysroot.bst
   bst checkout bootstrap/stage3-sysroot.bst ./sysroot
   ostree init --repo=./repo
   ostree commit --repo=./repo --branch=stage3-sysroot/x86_64 ./sysroot
   ostree remote-add releases ssh://ostree.baserock.org/releases
   ostree-push --repo=./repo --update releases stage3-sysroot/x86_64

Does that seem sensible? Or is this something we could usefully build
into BuildStream itself? Could we have an `ostree_deploy` element for
example?


Internally we are using an ad-hoc fork of the ostree-push code, which is more optimized and fault tollerent, 
a similar script could be fashioned out of buildstream's push/pull code.

Or, one could run a process on the machine hosting the ostree every now and again, to run buildstream to `bst 
pull` the artifacts you want and then checkout and create commits to your ostree repo locally.




Provenance
----------

Baserock takes the stance that you should always know where your
binaries come from, i.e. what was the source code, what arguments
were passed to `configure`, etc.

The way existing Baserock build tools did this was by dumping the cache
key's inputs into the filesystem of each artifact. So you would end up
with a /baserock/*.json directory, and each file would contain stuff
like this:

   {
       "repo-alias": "upstream:make-tarball",
       "repo": "git://git.baserock.org/delta/make-tarball",
       "original_ref": "4978d7129e42340ab9efeb0cb9cae4ad0fa052d4",
       "sha1": "4978d7129e42340ab9efeb0cb9cae4ad0fa052d4",
       "configure-commands": [
           "./configure --prefix=\"$PREFIX\" --disable-nls"
       ],
       ...
   }

BuildStream has all this info available too, but it lacks the ability to
dump it as JSON. I propose that we add a `bst show --mode=json` command
that allows us to dump the cache key inputs into the sysroot before we
push it to the releases repo. So our release process would just gain a
line like:

   bst show --mode=json bootstrap/stage3-sysroot.bst \
       > stage3-sysroot/manifest.json

Does this seem sensible?


Not so much, why would we do this in JSON when we use YAML everywhere else ? I dont think there is a lack of 
YAML parsers out there.

There are some things you can already extract with bst show in YAML format, and i'd be happy to look at 
expanding that to cover what you consider is needed.

Other than that, we are explore a possibility (not sure yet of how/when) of GPG signing the artifacts, 
allowing real assertion of provenance beyond bits of encoded information which you assume comes from a 
trusted source.

Cheers,
    -Tristan



I'm interested if anyone else is looking at similar processes, and
how much it makes sense / is possible to automate this stuff with
BuildStream. Perhaps we could/should wrap the release process up in
a `bootstrap/stage3-sysroot-release.bst` script element? Perhaps a
kind of "deployment" base element would be useful in BuildStream??

Thanks
Sam

-- 
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575
_______________________________________________
Buildstream-list mailing list
Buildstream-list gnome org
https://mail.gnome.org/mailman/listinfo/buildstream-list
[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]