Re: [BuildStream] bst-artifact-server plans

[Jürg Billeter <j bitron ch>]

Hi Chandan,

On Fri, 2019-09-06 at 12:46 +0100, Chandan Singh wrote:
> > > # Query support
> > >
> > > To manage a server with a large number of artifacts, supporting queries
> > > and deletions will be important. The details are not defined yet,
> > > however, I expect us to create a CLI tool in the same repository for
> > > this.
>
> You probably already had this in mind, but I'd like to call out that
> it will be nice to be able to delete the objects from the actual CAS,
> and not just the database. In fact, keeping the metadata in the
> database might be preferred in some cases. It should be much smaller
> comparatively and might be useful for audit trails. The deletions can
> always be done externally, but will be nice if the CLI/API supported
> this already.

That may not be as simple as it sounds.

The standard REAPI CAS protocol doesn't support deletion.  And as bst-
artifact-server will rely on an external CAS server (or buildbox-casd
but that's not scalable), it can't use an internal interface.

More importantly, deleting an object would only be reasonable if we
could first ensure that no other artifacts or directories still
reference that particular object.  This check wouldn't be cheap
(especially at scale) and it would have to disallow deletion of very
recent objects as they might be needed by a still in-progress
directory/artifact upload.

Also, a single CAS server may be used by different bst-artifact-server
instances (you could have project-specific artifact server instances
sharing a single CAS instance) and by other CAS indexing services such
as REAPI Action Cache.  I.e., it would not just be expensive but
actually impossible for bst-artifact-server to determine whether an
object could be deleted without breaking clients. That's presumably
also the reason why the CAS protocol doesn't support it.

As I see it the idea is to rely on some form of timestamp-based object
expiry in the CAS server, as is already in place in buildbox-casd and
planned for BuildGrid.  I.e., if an artifact proto is deleted (or
marked as retired), the clients are expected to cease accessing the
corresponding artifact objects (if they aren't referenced anywhere
else) and over time these unused blobs will be removed as part of the
expiry operation.

While buildbox-casd's relatively simple expiry implementation should
work well for its use case (local cache), other CAS servers could
implement more elaborate schemes such as having per-object expiry
policies, depending on what client uploaded it (e.g., production CI vs.
development CI vs. random developer using RE).  This could allow
relatively fast expiration of certain objects without breaking
important artifacts (an alternative would be to use separate CAS
instances).

Or did you have something else in mind that I'm missing?

Cheers,
Jürg