> While it is orthogonal, we are not talking about this in the
> abstract. On the contrary, proposed solutions imply separate
> elements for testing. I am also not sure this applies the same to
> packaging. I can see packaging be more limited to leaf nodes, for
> example final applications, whereas testing would apply to every
> element, including for example every library dependency.
> In short, let's treat this as orthogonal but let's also not forget
> the importance of this problem.
Indeed it is an important problem.
>From my perspective; it has occurred several times now that proposals
involve adding complexity to the base Element class, which has a high
cost and also tends to impede on the flexibility of what one can
express in pipelines of many elements.
I see these proposals rooted in an aversion to having to maintain many
.bst files in a given project - which is of course very rational.
Without getting into specifics of a separate proposal, clearly we need
a more convenient way to express elements in YAML, such that more
complex pipeline/project configurations can be expressed more easily
and conveniently.
+1.
[...]
> That implies that app-pkg.bst will not actually proceed with building
> until all tests of the dependent elements complete?
Yes it does.
Note that if parallelism and optimization of build planning is an issue
here, there is nothing stopping one from declaring an
`app-pkg-test.bst` stack element which has this requirement instead.
However, setting things up like this implies that you dont get the
desired behavior that:
"If the dependencies dont all pass their tests, this package will
never exist."
Given you that you can do both that's completely fair.
[...]
> > * While the requirement of the "tested" condition imposed by
> > app-pkg.bst is recursive in nature, inasmuch as it requires
> > the "tested" condition recursively across it's explicitly
> > declared dependencies, it is not recursive across implicit
> > condition based boundaries.
> >
> > This is to say that:
> > - app-pkg.bst requires the tested condition of app.bst AND lib.bst
> > - app-pkg.bst does NOT require the tested condition of lib-test.bst
> >
> > This is because lib-test.bst was implicitly pulled into the graph
> > due to the condition requirement.
> >
> > However, if lib-test.bst imposes other requirements on it's own
> > dependencies, those will be treated in the same way as app-pkg.bst.
>
> This could get rather complicated.
I wanted to add this to the specification here in order to avoid
complications, actually.
:)
* As soon as an element can impose that it wants conditions satisfied
by dependencies, this can happen at any level of the pipeline, not
only in leaf nodes.
Makes sense, I think. It runs the risk of blocking the reverse dependency
builds to block on tests of dependencies succeeding first.
* By explicitly limiting these conditions to the regular dependency
chain, we avoid blowing this up and ending up with surprising
conclusions, or possibly unexpected circular dependency chains.
[...]
> I do think there is another consideration missing. Instead of
> declaring in app-pkg.bst that I want all of my (transient)
> dependencies to satisfy the 'tested' condition I want to be able to
> run "bst build app.bst" declaring that I want all elements that are
> built during my run to have the tested condition met at the end of my
> run. And otherwise consider my build a failure. I don't think it's
> satisfactory If the only way for me to do this is to "bst build app-
> pkg.bst".
This is an interesting insight which we did not discuss at the
gathering.
I think that this could safely be implemented as an extension to this
plan (possibly as added user configuration and command line option, or
even the project.conf could recommend a preference here).
Great. I do believe this is a must-have.
[...]
> While I don't see why it couldn't work, I do think there is a balance
> we need to strike between keeping things generic (and simpler from an
> implementation/maintainablity PoV in BuildStream), and making things
> more specialized and convenient for users.
>
> If we have project maintainers provide standardized build elements
> (kinds), can those also take care of testing in one go, or do I now
> need to create a -test.bst element that has a related kind?
You absolutely must create a separate element, however this separate
element should not have to be defined in a separate file, and it should
ultimately be convenient to express this other element (this ties back
into what Chandan is referring to as an orthogonal problem).
I know, it's hard to ignore the issue :).
There are several aspects to this question I think:
* Does a specific "kind" (plugin) exist for the purpose of running
tests ?
This is not covered in the proposal, I personally think just a
"manual" element would be fine for this purpose.
I think this boils down to preference. Manual elements sound great
because they are generic and very flexible. However, more specific
elements that drive tests in a certain way may be more appropriate.
For instance if you have a large number of elements that are all
following the same convention, you may end up with a catA kind with
a matching catA-test kind.
Previous iterations of the proposal discussed a possibility
of having a specific test element which reads command lists
off of the public data of it's dependencies and runs them.
While that could be implemented orthogonally to this proposal,
I think that if we allow multiple elements to be declared in the
same `.bst` file more conveniently, we obviate the need for this.
* Again in the context of discussing a separate proposal to allow
pipelines to be expressed in YAML more conveniently, Jürg has been
suggesting a new type of plugin which could be responsible for
defining how multiple elements defined in the same file relate
to eachother.
From this angle, a plugin could be created which says that a
.bst file containing multiple element declarations have an implicit
relationship for the "tested" condition.
Here is a vague idea of what this might look like in YAML for an
element like lib.bst:
=====================================
kind: tested-condition
--
kind: autotools
depends:
- foo.bst
- bar.bst
sources:
- kind: git
url: https://foo.bar/baz.git
...
--
kind: manual
build-commands:
- make check
=====================================
In the above, we would have a .bst file which defines a "stream" of
objects (YAML supports this with the `--` delimiter as illustrated
above)
Then the "tested-condition" plugin, would be responsible for
defining the relationship of the objects declared in the given .bst
file.
In this example, the "tested-condition" plugin would simply say
that second object in the stream depends on the first object in
the stream, and that it satisfies the "tested" condition of the
first object.
There could of course be other ways to make this convenient to
express... but I think we need to shift the attention towards making
project _expression_ more convenient,
Up to here we fully agree.
and away from piling on more capability into individual elements.
And on this part I simply don't know. To be clear, that isn't meant
as a negative nor a positive.
Cheers,
-Tristan
Cheers,
Sander
