[BuildStream] Workspace related DX features and design, call for proposal

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi all,

There has been a lot of interest in the last cycle to enhance how the
user can interact with workspaces, and I think it's time someone pay
closer attention to this feature set as a whole and make a clear
proposal to the list about how we intend to support these desired
features.

These features cannot be thought of as independent improvements, as the
result will undoubtedly be confusing and undesirable.

As an example:

   We recently landed a branch[0] to support project relative workspace
   paths, except it doesn't actually do that. In fact, it only supports
   relative recording of workspace paths when the workspace is stored
   as a subdirectory of the project; which is a configuration we cannot
   recommend.

   Project data is controlled by the project, workspace location is
   decided by the end user; we should not be encouraging a setup where
   the user shoves their own data into the project, and risk
   accidentally adding unrelated stuff to the project itself, or having
   `git pull` fail because the user happened to place an unrelated file
   into the project directory, blocking the way for upstream changes to
   merge into a local project checkout.


A lot of thought has been put into this by Sander, Chandan and myself
in discussion on related issues, but I don't think this has amounted to
anything conclusive, as such; further development on these individual
issues lack the guidance of a master plan that we can agree on.

As such, I am asking that someone shoulder the responsibility of:

  * Understanding the underlying motivations for each of the related
    issues.

  * Coming up with a proposal to this mailing list which outlines the
    new CLI semantics, and describes how these proposed changes
    satisfies the requirements of each of the mentioned issues.

  * After discussing this on the mailing list; the related issues
    should be updated to specify more accurately what should be
    done as a result of the proposal thread, so that developers can
    more easily understand the master plan and implement the related
    features.

    Possibly, some of the existing issues need to be closed if it
    is assessed that our master plan cannot realistically cater to
    all of the desired functionalities (or if those requirements can
    be satisfied in different ways), and possibly new issues need to
    be opened to reflect the result of the thread.


The list of related issues as far as I can see (I may have missed some
of them ?), are these:

    Allow configuring default location for workspaces
    https://gitlab.com/BuildStream/buildstream/issues/229

    Allow running bst commands from project sub-directories
    (implemented, but the discussion can be helpful)
    https://gitlab.com/BuildStream/buildstream/issues/368

    Allow running bst commands from workspace directory
    https://gitlab.com/BuildStream/buildstream/issues/222

    Allow project relative workspace paths (closed but as
    mentioned above, does not really allow project relative
    workspaces, and encourages a workflow we cannot really
    recommend)
    https://gitlab.com/BuildStream/buildstream/issues/191

    Allow workspace commands to work on multiple elements
    https://gitlab.com/BuildStream/buildstream/issues/362

    Opening a workspace with a cached build
    https://gitlab.com/BuildStream/buildstream/issues/311


I think it makes sense to freeze any development on these workspace
productivity / DX features until we actually have a sensible plan, and
I am asking those who have vested interest in seeing those features
materialize, to step up and outline a proposal for how we should
semantically support as much of these features as possible.

Best Regards,
    -Tristan


[0]: https://gitlab.com/BuildStream/buildstream/merge_requests/504
[1]: https://gitlab.com/BuildStream/buildstream/issues/191