Re: Feature proposal: incremental build support

[Sander Striker <s striker striker nl>]

Related https://gitlab.com/BuildStream/buildstream/issues/21.

We should also revisit the dual cache key calculation and build planning modes thread.

Cheers,

Sander

On Tue, Oct 24, 2017, 14:49 Chandan Singh <chandan chandansingh net> wrote:

Hi,

When building large projects, it is very desirable to have support for doing
incremental builds so that when one makes changes to just a small subset of
files in the project, not everything has to be rebuilt. So, I would like to
propose adding a feature that would allow users to do incremental builds (or
dirty builds).

While the actual mechanics of how it would work are still open for discussion,
I envision that it would allow users to indicate that they are doing a dirty
build, by providing an option like `--dirty` to `bst build`. When this option
is provided, BuildStream would preserve the contents of the build directory,
including modification times. It is quite important that we preserve
modification time as a lot of build systems (make, ninja etc) use it to
determine if a file needs to be rebuilt.

Current implementation
----------------------

Currently, BuildStream does not support incremental builds at all. Compared to
using the underlying build tools (make, ninja etc) directly, this results
in an inferior workflow. When building outside the sandbox using those build
tools directly, it is very natural to do incremental builds. But the same is
not true when building in BuildStream as it starts fresh for each build,
resulting in wasted time and resources while re-compiling the same files over
and over again.

While the option of doing a clean build is very useful when one is doing a
production build, it slows down developers when one is iterating over a project
locally.

Proposed implementation
-----------------------

To support this functionality, there would be minimal user-facing changes:

  - add `--dirty` option to `bst build` command
  - provide a way to clear dirty build areas - this can be done as a separate
    `bst` sub-command.

To support dirty builds, we will need to change the way BuildStream stages
sources and manages the build area. I am going to describe one way in which
this can be implemented. When `--dirty` is not specified, the behavior of
BuildStream will stay unchanged from what it is currently. When `--dirty` is
supplied, I propose following changes:

  - Don't override modification times of source files. This can be achieved by
    omitting the call to `utils._set_deterministic_mtime()` here:
    https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/element.py#L1389

  - Name the build directories in a deterministic way so that it is possible to
    use the same build directory in a future build. Currently, it is created as
    a temporary directory but if this changed to use the name of the source,
    we should be set.

  - Preserve the contents of the build directory and do not clean it up after
    the build finishes, regardless of whether or not the build succeeds.

  - Instead of trying to create a new build directory for each build, try to
    re-use the build directory with the same name if one already exists.

If we add an option to allow dirty builds, it would also be reasonable to
provide users a way to clean their dirty build areas. To support this, we
can add something like `bst clean path-to-bst-file.bst`. Alternatively, a
`--clean` option can be added to `bst build` as well.

Please let us know your comments.

I'm looking forward to hearing your ideas for implementation. My current idea
is based on my quick attempt to create a proof-of-concept.

Cheers,
Chandan

--
Chandan Singh
https://chandansingh.net
_______________________________________________
Buildstream-list mailing list
Buildstream-list gnome org
https://mail.gnome.org/mailman/listinfo/buildstream-list