Re: [BuildStream] YAML rework - Another go

[Daniel Silverstone <daniel silverstone codethink co uk>]

On Tue, Mar 05, 2019 at 15:28:03 +0000, Daniel Silverstone via buildstream-list wrote:
[snip]

After some discussion and work with James, we're altering the plan for the
mapping structure slightly to:

> Appendix 2 - Raw data representation in memory

* All values are represented as a tuple of the value, file number, line number,
  and column number.  This applies to strings, mappings, and lists.
* Mapping keys are plain python strings
* All lists, empty or otherwise are represented as above, providing provenance not
  only for the list object, but also for the entries therein.
* Mappings do not have any special provenance keys inside them.

[snip]

Everything else about the raw representation remains, though the above pretty
much sums it all up anyway.

Proposal for reworking tracking to work with this new format:

To track:

1. Create cleaned copy of node in question (i.e. bare dict/list/str)
2. Clone that
3. Hand the clone to the plugin to mutate
4. For each key/value pair coiterating through the copies and node together:
      a. If the same, continue
      b. If different, acquire the provenance from the node
      c. If the file referenced is invalid for tracking, abort
      d. Reload the file from the top level, walk it to find the requisite
         entry, retaining the walked path
      e. Acquire a ruamel round-trippable load of the file (from store if
         we've already edited it once)
      f. Make the same change to that ruamel variant
5. For any edited ruamel file, dump it out

Things to consider if they're permitted during tracking:

1. Editing a list (should we treat that as a full value replacement?)
2. Removing a key (needs detecting)
3. Changing the type of a value (e.g. list to string, or somesuch)
4. Adding a brand new key -- tracking via provenance of containing dict?

NOTE: Part 4c encapsulates the check for whether or not the file comes via
a junction, or was synthetic in some sense.  The exact mechanisms for those
checks have yet to be decided upon.

I appreciate this is all a bit wooley still, but James and I are working on
things, and we're making progress.

D.

-- 
Daniel Silverstone                          https://www.codethink.co.uk/
Solutions Architect               GPG 4096/R Key Id: 3CCE BABE 206C 3B69