Hi Sander, On Oct 19, 2019, at 10:02 PM, Sander Striker <s striker striker nl> wrote:
Hi, I replied a bit soon as I see now others have already raised the issue of lost parallelism.
:). If tracking is really a big issue for small projects with large junctions, let's investigate why that is the case and address that. In the same way we addressed the performance optimizations: by gathering data, analyzing and then making balanced decisions on how to improve.
How prolific is this use case? In common use how does bst build stack up to bst build --track in terms of frequency of use?
I still do think it is an important feature, and feel that it might be a bit embarrassing to not have this feature, considering everything else can be well parallelized, so why not tracking also.
Because this optimization comes at a cost. I'm not sure that cost is worth it. The feeling of embarrassment is subjective, I find that hard to assess ;)
Yeah I understand that :)
So, the main reason I added this long ago was indeed because I planned to use it in a very specific CI situation (besides the fact that I think it’s just a very cool feature, which is indeed subjective).
Consider a CI scenario where you have many people committing to the master branch of many git repositories simultaneously and you just don’t have the horse power to run builds of all reverse dependencies for each commit (a scenario I happen to be facing right now actually), and you want to report build failures as fast as possible.
Build + track parallelism should buy us some valuable minutes in turnaround, where the CI is using that in combination with non-strict build plans.
Beyond this, I would like to revisit client side event triggers (user configurable scripts/triggers) so we can send these notifications of failures immediately while the server is building with `—on-error=continue`.
Cheers, -Tristan
Cheers,
-Tristan
Cheers,
Sander
> On Oct 19, 2019, at 11:56 AM, Tristan Van Berkom <tristan vanberkom codethink co uk> wrote:
>
> Hi,
>
>>> On Oct 18, 2019, at 10:56 PM, Chandan Singh <chandan chandansingh net> wrote:
>> Hi all,
>> tl;dr: Remove `--track-*` options from `bst build` and simplify codebase.
>> ---
>> I wanted to propose a clearer separation between tracking sources and building
>> them in BuildStream. Currently `bst build` offers a few `--track-*` options,
>> all related to whether or not BuildStream should attempt to track sources
>> before starting the build.
>> In my opinion, these are two different kinds of operations, and don't really
>> need to be part of the same pipeline.
>
> I disagree with this.
>
> The reason why we can track and build in the same session is an optimization, we should not have to wait until all the elements finish tracking before being able to build the first elements which reach a consistent state. I understand the code would be easier to maintain if we didn’t have this feature but I think it really provides good value.
>
> I do however agree that the UX for this is undesirable and could be improved, one suggestion somebody made on IRC some months ago was to make things a bit scriptable for the more complex invocations, and for running multiple commands in a single session (so we could also have build + checkout for instance, which has been requested as a convenience by some users).
>
> This might mean removing some of the more complex CLI options (or not) and adding a new option to read commands from a file (or a quoted string on the command line, or stdin or such).
>
> Not sure how popular this idea might be, just throwing it out there.
>
> Cheers,
> -Tristan
>
>
>> My concerns about this can be broken down
>> into two broad areas:
>> 1. Simplifying scheduling/update_state() logic
>> 2. UI/UX
>> First, let's talk about scheduler. At present, when we start a build, we expect
>> the elements being tracked to be resolved only after they've been tracked. This
>> means that we can't be sure that all elements in the pipeline are fully resolved
>> and ready to be built.
>> Decoupling tracking and build operations should simplify the `update_state()`
>> logic as it would eliminate one of the needs for doing so.
>> Moving on to UI/UX issues, `bst build --track-*` essentially duplicates
>> functionality from `bst source track`. Hence we end up with two ways of doing
>> the same thing and have to support both.
>> Since tracking sources and building are rather different kinds of operations,
>> I am proposing to have them clearly separated. More concretely, I am proposing
>> to remove the following options from `bst build`:
>> * `--track`
>> * `--track-all`
>> * `--track-except`
>> * `--track-cross-junctions`
>> * There's also `--track-save` which is deprecated and ignored
>> `bst source track` already supports all functionality offered by the above
>> options, so that won't require any additional features.
>> I can understand the desire to update all source references in a CI build, or
>> for testing purposes. But, I think the same can be achieved by running
>> `bst source track` followed by `bst build`. Can people think of any other use
>> cases that would be negatively affected by this change, other than having to
>> type two commands instead of one?
>> Let me know what you think.
>> Thanks!
>> Chandan
>> _______________________________________________
>> buildstream-list mailing list
>> buildstream-list gnome org
>> https://mail.gnome.org/mailman/listinfo/buildstream-list
_______________________________________________
buildstream-list mailing list
buildstream-list gnome org
https://mail.gnome.org/mailman/listinfo/buildstream-list
|
