On Thu, 2018-11-01 at 22:25 +0000, Sander Striker via BuildStream-list
wrote:
> Hi,
>
> I expect CAS and ArtifactCache (and ActionCache) to contain partial
> artifacts at some point due to different retention policies for
> different parts. For example, build trees might expire much faster
> than the core artifact (the part that is staged when depended upon).
> Similarly logs may expire as well.
>
> This is all policy based, and very much use case specific. I do not
> wish for us to design or implement how partial expiry of artifacts
> might work, I merely want us to acknowledge that artifacts may become
> partial in the future. And with that acknowledgement consider future
> proofing buildstream to gracefully deal with partial artifacts.
Since we have agreed to implement configurability of uploads of the
build trees[0], yes.
However for the rest, log files, etc; I think not - this is not a
generic thing and I consider it very unwise to go down a path where
arbitrary parts of an artifact might be missing.
It wouldn't be arbitrary. We would need to agree on what these pieces
are. The policy on what to retain and what not will be different in different
contexts. I don't see a need for BuildStream to provide this functionality in
its bundled cache, but external cache providers should be allowed to do this
to properly manage available resources.
BuildStream needs to know that an artifact exists or not, and rely on
this to make the assumption that all of the components of the artifact
exist.
Why? I hope you're not saying that I need to have the logs of an artifact I
depend on to be able to build? At build time, I would argue that we don't
want to pull down _any_ information over the network that we don't actually
need. Nor want to store in our local caches (which people are already
aggressively wanting to clean).
And this being the case, why couldn't BuildStream then fail friendly when
I run `bst artifact log mine.bst` ?
Implementing configurability of the build trees on it's own is going to
already impose considerable complexity, we should not allow this to
grow any further.
I am probably missing something, can you elaborate on the considerable complexity? Because I see the complexity coming in regardless for different reasons (as outlined above).
Cheers,
-Tristan
[0]: https://gitlab.com/BuildStream/buildstream/issues/566
Cheers,
Sander
