Re: Project options and other format enhancements (and dropping "variants")

[Sam Thursfield <sam thursfield codethink co uk>]

On 16/09/17 07:22, Tristan Van Berkom wrote:
> On Fri, 2017-09-15 at 15:53 +0100, Sam Thursfield wrote:
> > Hi Tristan,
> >
> > In general I agree with your premises, and I think the proposal is
> > workable. I don't have anything better to propose.
> > >      Option Declaration
> > >      ------------------
> > >      A project declares which options are valid for the
> > >      project in the project.conf.
> > >
> > >      These options should have some metadata which can be used
> > >      to declare the defaults, assert valid values of the options,
> > >      and also a description string which the CLI can use to communicate
> > >      the meaning of project options to buildstream users (not all users
> > >      building a project wrote the project.conf).
> >
> > Are you expecting to support only enum values, or freeform strings and
> > integers too?
>
> I certainly dont want to support only enums, although I was undecided
> on whether the data would simply be in string form and have conditional
> statements deal with typing; or to have typing encoded into the option
> metadata.

I would much prefer typed values. Otherwise you end up in the land of 
"do I turn off this feature with 'no', 'false' or 'off'" ?

I would also like to avoid or at least strongly discourage having 
freeform strings. From an analysis point of view, an enum option with 15 
possible values represents 15 possible variants of your build. A 
freeform string option represents effectively infinite variants of your 
build. (In small projects you may be able to drill down and see how many 
string values are actually used, but in huge projects that becomes 
impractical).

> One idea I am liking more and more is the (contains list_opt "string")
> kind of conditional, where one could check for the presence of a word
> in a whitespace separated wordlist, which one could use to whitelist
> elements for a feature or to test for a single feature in a list (like
> the compiler tuning example I made in my reply to Sander).

You probably already made the connection between this idea and Gentoo's 
USE flags: https://gentoo.org/support/use-flags/

It's a good system provided the possibilities are limited. So the option 
definition should have to declare all the possible flags that can be 
used in the list, rather than it just being a freeform string.

> <slight deviation from topic>
>
> As you've been looking into bootstrapping compilers with BuildStream
> maybe you can shed some light on what we could do for this, because I
> feel my approach doesnt solve it perfectly.
>
>  From what I understand, currently we can only single case symbolic
> machine names and make a huge list of full tunings flags depending on
> that symbolic name. This is an area I think yocto excels at and I would
> like to have a solution that allows enough flexibility for this (of
> course without being shell scripts which execute and source eachother).
>
> At the basic level, maybe this could be done by allowing a project to:
>
>    A.) Define symbolic names, maybe they are "macros" or "presets"
>    B.) The symbolic preset defines values for options
>    C.) Write conditionals based on the options
>
> It's just the first approach that comes to mind, but would allow us to
> define feature lists associated to symbolic machine names, and then
> write conditional YAML fragments based on the resulting feature sets
> instead of having to special case every machine name individually.
>
> Any other ideas ?
>
> Implementing a solution to this should not block our landing of a
> project options feature; however, our approach should probably be
> informed by if/how we intend to address this kind of complex case.

My goal with architecture names in Baserock has always been to limit the 
possibilities to a fixed set that we can exhaustively test. So in a way 
it's a feature rather than a bug that the way to enable extra 
project-wide compile flags is to modify gnu-toolchain/gcc.bst or 
project.conf directly.

I'm fine with people setting random GCC tuning flags if they want, but 
it should be be clear that if they do this then they have diverged from 
the upstream reference builds and any breakages are for them to fix, not us.

So I don't think we need any complex solution here. However there are 
certain variations where we probably do need to support both upstream 
and for those encoding info in the architecture name sucks. The only 
current example I can think of is ARM hard-float versus soft-float.

Having a "CPU architecture" flag set which can contain specific flags 
based on the overall symbolic architecture name seems like the nicest 
approach there. So if your arch name is "armv8l64" you get 
arch_flags=hard-float and you can override that to arch_flags=soft-float 
if you wish. If your arch name is "x86_64" then neither option is valid 
(or perhaps more accurately 'hard-float' is the only valid option).

> </slight deviation from topic>



> As I've said else where I've migrated from:
>
>    '??':
>      condition:
>      ...
>
> to:
>
>    (?):
>      condition:
>      ...
>
> I feel like it will stand out more and I dont like the quotes. That
> said I'm open to changing the conditionals to something more
> conforming, if we really expect that the result is going to be more
> legible.

I don't like the quotes either, (?) is probably better.

> Ok so frankly I'm not attached to parenthesis nested lists esthetically
>   speaking, I am however quite attracted by the simplicity of it, we
> probably can achieve similar simplicity with something else.
>
> I'm not convinced that a:
>
>     condition: |
>
>       %{foo} == "bar"
>
> kind of notation is simple though; it tries to be very human friendly
> and programing languagey, and then leaves us a bit blind if we want to
> later extend the operators, what would we use for the kind of word in
> list 'contains' conditional ? (maybe we do like python sets and use C
> bitwise field test operators on them ?).
>
> This operator comparison expression approach is also more rigid and
> demanding, it would have to be done perfectly the first time.
>
> We dont have a nice relaxed namespace in which we can deprecate the
> 'ifeq' symbol for a new 'equals' symbol on the day that we figure out
> that comparisons should have been case insensitive, we might instead be
> in a corner left with yucky workaround alternatives, advising users
> that they should use the '===' operator in new projects, instead of the
> existing but botched and unrecommended '==' operator.

I think deprecating "ifeq" for "equals" is pretty horrible too. We 
should just get it right first time, whichever approach we choose :-)

I agree the simplicity of s-expressions and nested lists is a strength, 
but I still think it comes at a usability cost. Also, just because it's 
simple now doesn't mean that it won't have grown to be monster in 2 
years time because everyone who uses it gets excited and sends in a 
patch to add their favourite language construct.

What I'm suggesting is that we avoid creating a new DSL for expressions 
in the first place, we try and reuse an existing one instead. Ansible's 
use of Jinja proves that this is possible. I'm not entirely sure how it 
works though -- if I get time I will try and make a proof of concept 
expression parser and we can see if it can beat your 218 lines of code :-)

Sam

--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575