|
Hi all,
As recently
raised by Tristan [1], there are numerous related issues &
hanging
merge
requests revolving around improvements & new functionality
for BuildStream
workspaces.
As such it make sense to address them in a common proposal
which
hopefully
addresses & highlights any overlapping or common changes
needed to
meet the
respective goals. Tackling & grouping related requests
together also
means that
if we have to conclude that a CLI change is needed [2], then it
can
be done once
to minimise & hopefully mitigate further changes. It's also
important to
understand how any changes to the CLI could be
consumed, in the
sense of a
user (porcelain) or in automation (machine readable) so any
feedback
from those
perspectives is welcome.
Overview
========
We want to
enhance the usability of workspaces in BuildStream. This
includes
increasing
the flexibility of the existing implementation (such as
supporting
new defaults
for workspace location & names [3]), QoL changes (such as
the
ability to
use relevant bst commands inside workspaces, and bst commands
targeting
multiple elements, [4][5]) and new additions for aiding
development
&
debugging (such as cached workspaces & multiple workspaces
in shell, [6][7]).
Some of the
issues raised in the call for proposal are months old and as
such
some of the
included comments have now been rectified or made redundant; as
such
consolidation is needed.
When
reviewing the state of the tickets that extend the support for
debugging
&
development within the shell, it has been highlighted that
cached build trees
may be the
more appropriate option instead of extending workspaces. As such
it
may be more
appropriate to create a new bst command for this workflow &
consider
it outside
of this scope. Remote execution should also take advantage of
build
trees (for
failed build artifact debugging [8]) and not need any new API
changes
to
workspaces, however if there is disagreement we should capture
those changes
in this
proposal. Once a change has been developed in a workspace it
would be
desirable to
have an easy way to get this information into the respective
upstream/vcs
(or even just locally for the projects consuming it); we think
this
may need to
expand the workspace API but not change any existing features
e.g.
`bst
workspace patch create` (or a documented patch/quilt workflow)
or similar.
Proposal
========
These are
the workspace related features to be considered together:
* run bst
commands in workspace directories, including defaulting command
to
said
workspace element.
* Make
workspace creation easier.
-
Automatic names.
- Default
location.
- From
cached build.
- Make
multiple workspaces.
-
Relative paths outside of project.
* Debug,
workspace/cached build may be used to preserve the required
artifacts
so
including this here.
Implementation
==============
Comands in
workspaces
~~~~~~~~~~~~~~~~~~~~
This was
started by chandan/cs-shadow in [9] however it leaves some
questions.
[9] uses a
file in ~/.local/share to achieve this but then it cannot
easily
implement
relative workspaces. We propose using a `fragment` file that we
will
create in a
.bst/project.yml of a workspace's root directory. This is
similar
to how git
submodules identify which directories in the main project are
subdirectories.
Support for
multiple projects pulling a workspace is not allowed by default
but
it can be
forced with known side-effects. Default behaviour is to say a
workspace
already exists at a given location with a given name. If forced
to be
a workspace
for multiple projects then the original parent project takes
precedence
(i.e. is the default resolution unless overridden) (bst
commands
again need
to be forced & warn the user that it may affect other
projects
consuming
the shared workspace).
We will add
a `fragment` to workspaces that points back to the project which
created it.
This will allow for commands run in workspaces to find the
project
that they
should run from. Also it allows for projects to check for two
projects
using the
same workspace.
The fragment
will also be relative if the workspace was created as relative
to
allow for
moving workspaces.
Workspace
creation commands
~~~~~~~~~~~~~~~~~~~~~~~~~
Note:
Defaults already exist at a project and system level so we will
just store
the default
workspace location information alongside it.
We will
change from:
bst
workspace open -element.bst workspacename
to:
bst
workspace open --directory /path/to/workspaces element.bst
[workspacename]
When opening
a single element then you can:
bst
workspace open --directory /path/to/workspaces --multi
elementA.bst elementB.bst
The
directory option specifies the
directory, in which the folder
containing/(the
root of) the workspace will be created. The folder
containing/(the
root of) the workspace is either the
"workspacename" from the
command line
or if not given then it is the element name without the trailing
".bst".
eg. `bst
workspace open --directory /path/to/workspaces elementA.bst`
would
create:
path/to/workspaces/elementA/
as the root
or the workspace
and `bst
workspace open --directory /path/to/workspaces element.bst workspacename`
would create
path/to/workspaces/workspacename/
If the
directory is relative, e.g. not
starting with a "/" then we treat it as
relative to
the project and the workspaces fragment will be in relative
terms,
so you can
move both or, more likely, the folder that contains both.
In
anticipation of a feature to allow a workspace to be created
from a build
tree we
would also add a pair of exclusive options (--no-cache
--use-cache) that
would
override the buildstream/system/project
default option for, if a cached
build should
be used. The open command will only accept a single one of
these
options and
so if you want some of the elements to be cached and some not,
you
will need to call the
command twice - once with the elements to be
cached and
once for those to be created without. As we propose to land all
the
CLI changes
to `workspace open` command in one MR, we may need to land the
option
before the feature is ready. In this case
we will have bst issue a "not
implemented" error if
the --use-cache option was used.
Moving
projects
~~~~~~~~~~~~~~~
This will
only be supported if the workspaces are defined with a relative
path.
Note: you
could fudge by manually changing the fragment or we could add a
utility to
automate this.
This is
because otherwise the fragment will get lost and you cannot run
bst
commands
from the workspace. If run in the project then the fragment
check will
fail.
/projects/Stuff/ProjectA/
buildstreamfolder/project.conf
workspaceA/source/etc
workspaceB/source/etc
could be
moved to:
/projects/better/ProjectA/
buildstreamfolder/project.conf
workspaceA/source/etc
workspaceB/source/etc
Debug
~~~~~
Note: while
the scope of this task is workspaces, they are clearly an
important
factor in
debugging especially when debugging something you are using as
a
workspace,
so we need to catch any potental API changes to workspaces.
This
therefore
requires some detail of how to debug to ensure that no API
changes
are needed.
We will add
an example of using GDB from a
binary package to show how a generic
debugger
could be used from within the bst shell. However it may well be
that
BuildStream
could build the debugger, and it could be added to the shell as
a
runtime
dependency. There has been relevant talk surrounding the
perceived
specifics of
how to utilise 'SwankyDebug' inside a bst produced shell here
[10].
This will
then form the reference implementation that we can then talk
around
to implement
other debuggers that may have other requirements.
This example
will include debugging a library from an app (getting break
points/messages)
from the library when running `gdb app` from the app's shell.
Mounting
workspaces vs cached build trees was brought up in [7] for this
perceived
usecase and as such it needs to be clarified whether the
commitment
to implement
this feature with the current workspaces is 'worth it', or if
the
work to use
cached build trees is ultimately the best long-term path. One
other
point of
consideration is do we want to differentiate a flag for runtime
dependency
debugging vs build debugging (which is currently -b), or would
runtime
debugging be the default functionality of 'shell'?
The
following must be implemented without affecting the project
files (we dont
want debug
options to be accidentally added to the project's vcs) but
parts will
affect the
cache key:
- Get list
of runtime/build time dependencies
- Reference
against open workspaces for the project
- Mount them
into the respective buildroots, provide feedback if there are
clashes
-
Environment is set up for target element, user has to be aware
of this if
they want
to tinker with dependencies within shell.
- Mount in
debugger (local element type if not to be built by BuildStream,
either as
dependency or debug flag)
- Debug,
this can include making source changes & recompiling within
shell
- Drop out
of shell & carry on with bst workflow as appropriate to the
debugging
outcome.
Outstanding
issues
==================
Issues to
close
~~~~~~~~~~~~~~~
The
description has been implemented so this can be closed:
Creating
workspaces should be one ticket setting out a concise API, so we
will
close the
related ones and create a new one with a clear description and
link
to this
list:
Issues to
Update
~~~~~~~~~~~~~~~~
If we accept
the fragment idea, then we will add the details and submit a MR
Issues to
create
~~~~~~~~~~~~~~~~
- A new
'epic' issue which will contain the outcomes of this proposal as
well
as a link
to the original mailing list post, to act as a related ticket
for
all new
work and to serve as a point of reference/origin.
- A new
ticket for relative workspaces that covers "out of project"
workspaces.
- An issue
to cover all the changes to the command line CLI for the
workspace
open
command.
- An issue
to track the proposed PoC for debugging an app & related
libraries
from
within bst shell with GDB as a generic tool.
Many Thanks Tom & Will # Links
|