GNOME.org
 

Re: Jinja2 syntax for conditionals


Sorry but this is going to be a bit... long.


On Wed, 2017-09-20 at 14:00 +0000, Sander Striker wrote:
[...]

On the other hand, when not taken too far, it's comforting to know that
with a python like syntax we can still add helpers (which can help us
wiggle out of corners if we get stuck).


That is definitely appealing.


Or, what might be even more appealing still is to *only* use callables
and disallow conditional operators entirely ?

So instead of using Jinja2 for:

  (?):
    condition: foo_var == "foo literal"
    ...
  
We could use it strictly for function calls and compound AND/OR
expressions, it might look like:

  (?):
    condition: equals(foo_var, "foo literal")
    ...


It doesnt look like what people want it to look like of course, but at
least we define the behavior of everything we use, not Jinja2.






That said the jury is still out on the general direction of this; you
seem to be after the exact opposite of what I'm looking for and I'm
trying to understand / balance the reasoning behind this:

   Do we want something fully specified and very cute, with a rigid
   non-extensible syntax that other programmers may recognize because
   they might have used that syntax before ?

Or

   Do we only want a data serialization format that is a bit more
   practical than YAML is for the purpose of serializing optionally
   quoted strings and numbers ?

   The exact same thing can be done with JSON or YAML, it just happens


   to be practical to use S expressions for these one liners.


I don't see this as the question at all. Either way we're adding logical 
expressions to what was previously a serialization format, and making 
something that is (a) a serialization format with syntax sugar for 
variants, or (b) something else.

Sure you can represent the syntax tree of the S-Expressions as a YAML 
data structure, the same is true of the syntax tree of the Jinja2 
expressions.

For me the questions are:

1) do we want to use S-Expressions, or a more Python-like syntax?


I'm hesitant because on the one hand it is like a bike shed... except that it is standing right in front of 
the house.  I'm leaning more towards Python-like.


Mhm, and I'm leaning the opposite way...

On Tuesday, Sam and I had a short chat about this in #buildstream:

https://irclogs.baserock.org/buildstream/%23buildstream.2017-09-19.log.html#t2017-09-19T14:03:41


I'll detail here what I took home from the couple of points we
discussed, and just to make sure we're all still on the same page; the
discussion is about:

  S-Expressions - or something similarly data-format-ish, that we have
  full control of, just like our YAML.

VS:

  Jinja2 - or something similarly programming-language-ish, which will
  instead dictate the rules of the game to us and have it's own
  predefined characteristics.

BUT NOT:

  Let's exclude the case where we write our own full blown custom
  programming-language-ish parser for the sake of retaining a
  measure of control, that's too expensive.



1) It's possible; that our own S-Expressions will evolve to a level of
   complexity to rival that of the parts we use from Jinja2.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is honestly of little to no concern to me - if we take the
S-Expression route and there is some real justification for a lot of
comparison types; this does not pose any danger.

On the other hand, choosing a programming-language-ish thing, poses an
implied risk, because the implementation (which need not be very
complex anyway), is out of our hands.


2) What could possibly go wrong if we were to choose something
   pre-defined ?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is the wrong question to be asking, something will always go
wrong, we just can't know what it is yet.

But I've been toying with this case sensitivity example as a model of
what we would have to do when there is something in the conditionals
that must change.

So lets say; that one day we decide that string comparisons for
buildstream conditionals should be case insensitive; or that string
comparisons for a *certain type* of option (remember, options are to be
typed in the project.conf at declaration time) should be case
insensitive.

To keep it interesting, lets assert that this case insensitivity is a
new policy of buildstream itself; and in no way should the user really
be burdened to care about that detail when writing out a conditional
which compares strings.


S-Expression approach:

  Two things can happen, it is possible that changing a behavior has no
  negative impact on existing projects, but only enables something
  which was previously impossible; in this case we would just change
  the behavior of `ifeq` and call it a day.

  Alternatively, we deprecate `ifeq` and introduce `equals`, and
  recommend using the new one.


Jinja2 approach:

  In the case that we want to change the behavior of '==', we're up
  that well known creek without a paddle...

  In the case that we want to introduce something new to replace it, we
  have a few roads:

    o Implement a function 'ifeq(a, b)', and recommend users to use
      that instead. This is rather undesirable because we shift from
      operators to function calls.

    o Implement a function 'tolower(str)', and recommend that people
      write out their conditionals as:

         tolower(foo) == tolower(bar)

      This is even far less desirable, because then the user is
      explicitly knowing about case sensitivity, in a situation where
      it should simply be a matter of policy.


One more point I'll add to this which I think I didn't bring up before:

  o Do we want users to perceive the conditional statements like code ?

    Honestly this part scares me a bit; the intention here was to do
    something more convenient than YAML just for the conditionals,
    saving users some typing, and using S-Expressions instead of
    rolling a completely ad-hoc parser.

    If we shove something that looks like python there, we risk it
    starting to look like a programming language; anticipate feature
    requests for inline variable and function declarations encoded
    into YAML strings.


So reality check:

Are these concerns not real ? I feel like what we're about to land is a
pretty HUGE API surface, including a project YAML format, a Python API
surface for custom plugins; a configuration surface for users and
admins, a CLI to be consumed by automated scripts... ALL of this API is
going to have to be stable; the project YAML format probably taking the
highest priority in terms of stability (where perhaps configuration is
the least concern but still a concern).


A rough summary of the negative points for either camp that I'm
currently seeing is:

   Jinja2 (or other code like thing):
     o Dependency is a liability, replacing it or even forking is
       going to be non trivial
     o Is already defined, we cannot easily bend it to our will
     o Possible negative point: Looks too much like a programming
       language, instead of a natural extension of the bst format

   S-Expressions (or other data like thing):
     o Uses parenthesis,
makes people think of lisp


From this perspective, it looks to me like it doesn't make any sense at

all to paint the bike shed green; if waterproof paint is not available
in green.


2) do we want to write and maintain the execution engine ourselves, or 
reuse something that exists?


My understanding is that with S expressions, your answer to (2) is that
we basically use something that exists, and that the few functions for
crawling the loaded data and resolving the expression is negligible.

For (1), I would really like to find something more attractive as a
project to depend on than jinja2 for this.


Would you be comfortable using it and replacing at a later point given we would only use a subset of 
functionality?


I'm not a parser guy, but I assume it would take me up to a week to
drop the dependency (if I had that week to spend); assuming we take the
route of delicately forking out only the code we want and ensuring it
still works (not writing a replacement from scratch).

I've worked on well funded projects and unfunded projects; either way,
a real week of work is hard to come by when it comes to things like
paying off technical debt, or refactoring bits for the sake of project
longevity.

It's not really an answer to your question; but I think you know as
well as I do that removing Jinja2 will quite likely not happen until
the day it actually poses a problem - IF there happens to still be
active developers around either paid or willing to actually do the
work.



Also, when you say:

   "What I'm suggesting is that we avoid creating a new DSL for
    expressions in the first place"

I'm not sure that this qualifies; i.e. if parsing some serialized data
to be interpreted as nested conditional expressions qualifies as
creating a new DSL, certainly the BuildStream format itself, even
though expressed through YAML, is also it's own DSL.


There's no universally agreed place where we draw the line between 
"code" and "data". In my view, once we have self-modifying conditional 
expressions we're crossing line into "code", regardless of whether the 
syntax used is S-Expressions or Jinja2 expressions, or YAML. Where the 
line is doesn't really matter, my point is that (if we ignore 
implementation details) our proposals amount to the same thing.


+1.


Eeeeh I don't think they are the same no.

Just because it's hard to draw a line in the sand between where data
stops and code starts; does not mean that opposing sides of the
spectrum resemble eachother and have the same properties. This decision
is going to have real life imacts.


After rereading the preceding paragraphs 5 or 10 times in a row... The
real point here; is that Sam is suggesting to reuse something that
formally exists.

Personally I could care less if it looks like lisp or python, I'm more
afraid of not having control over it's behavior than anything else
really (in other words, I think it definitely *should* be it's own DSL,
that seems to be the safe way forward). 

Cheers,
    -Tristan
[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]