Re: Large base environments and delays during staging deps

[Angelos Evripiotis <angelos evripiotis gmail com>]

Heya Tristan,

Thanks for the prompt response! I especially appreciate it now as I think there
are a lot of requests for your time at the moment.

> Are you at *least* using SSD ? Does anyone try to build huge
> things on HDD anymore ?

Hehe, yup it's SSD at the bottom, on a 2015 MacBook Pro. On top of that it's
running in a Docker container. The sources usually live on a bind mount, which
is expected to be slow for this. ~/.cache/buildstream is in /tmp for my
experiment so I don't expect the bind mount to affect staging.

> I am staging ~1GB runtimes in *much* less time

It seems that size is less important than the number of files in my
experiement:

  o Roughly 30 seconds to stage 100,000 files, 431M total.
  o Less than 1 second to stage 1 file, 2GB total.

Would it help if I do some more thorough testing (e.g. more machines / setups),
and provide something easily reproducible? I've appended details of my current
simple experiment as an example. Perhaps I can try to build the same thing as
you and see how that works for me.

> One thing to keep in mind, is that there will always be a point where
> the workload is too much for the tooling, even if the tooling is
> perfect - when you get into needing a ridiculously large amount of
> dependencies staged at the same time, I have the feeling that making
> BuildStream more complex to cater to that need, is working around a bug
> that is in fact in the design of your dependency layout (which must be
> a very, very complex dependency layout).

I can see that unbounded growth will always reach a limit. In the case of lots
of files, I don't expect BuildStream to distinguish itself by specialising in
this. I share your concern of not want BuildStream to become overly complex,
especially not so that it can provide a temporary crutch for an unscalable
dependency layout.

That being said, I do think it is reasonable for BuildStream to be very quick
at staging the base operating system. My expectation there has probably been
set by using Docker.

Perhaps we won't agree about how many files should be in the base :)

Base operating system choice seems like a fundamental problem that most users
will face. I think that minimalism will work well for green fields projects. I
suspect many other projects not starting from scratch will have difficulty
achieving it.

If Gnome is a good example of BuildStream usage, I should probably also test
with that and see how my experience compares. Is this repo the best place to
try out building Gnome? https://gitlab.com/BuildStream/gnome-test-data

Cheers!
Angelos

P.S. here are my tests, in both cases I'm interested in the time taken to stage
when building 'compose.bst'.

--- Testing with lots of files to stage ---

echo 'name: lotsafiles' > project.conf

mkdir lotsafiles
cd lotsafiles
mkdir -p {1..100}/{1..100}
time for FILE in {1..100}/{1..100}/{1..10}; do echo $FILE > $FILE; done
# real    0m4.861s
# user    0m1.280s
# sys    0m2.730s

du -sh
# 431M .

find . | wc -l
# 110101

cd ..

cat <<EOF > lotsafiles.bst
kind: import

sources:
- kind: local
  path: lotsafiles
EOF

cat <<EOF > compose.bst
kind: compose

depends:
- filename: lotsafiles.bst
  type: build
EOF

bst build lotsafiles.bst

# (run 1)
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# [--:--:--][17abf0b2][ main:lotsafiles.bst  ] START   Starting build
# [--:--:--][17abf0b2][build:lotsafiles.bst  ] START
dpkg-ic/lotsafiles/17abf0b2-build.26048.log
# [--:--:--][17abf0b2][build:lotsafiles.bst  ] START   Staging sources
# [00:00:35][17abf0b2][build:lotsafiles.bst  ] SUCCESS Staging sources
# [--:--:--][17abf0b2][build:lotsafiles.bst  ] START   Caching Artifact
# [00:01:10][17abf0b2][build:lotsafiles.bst  ] SUCCESS Caching Artifact
# [00:02:23][17abf0b2][build:lotsafiles.bst  ] SUCCESS
dpkg-ic/lotsafiles/17abf0b2-build.26048.log
# [00:02:23][17abf0b2][ main:lotsafiles.bst  ] SUCCESS Build Complete

# (run 2)
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# [--:--:--][dccc1820][ main:lotsafiles.bst  ] START   Starting build
# [--:--:--][dccc1820][build:lotsafiles.bst  ] START
lotsafiles/lotsafiles/dccc1820-build.3511.log
# [--:--:--][dccc1820][build:lotsafiles.bst  ] START   Staging sources
# [00:00:35][dccc1820][build:lotsafiles.bst  ] SUCCESS Staging sources
# [--:--:--][dccc1820][build:lotsafiles.bst  ] START   Caching Artifact
# [00:01:01][dccc1820][build:lotsafiles.bst  ] SUCCESS Caching Artifact
# [00:02:22][dccc1820][build:lotsafiles.bst  ] SUCCESS
lotsafiles/lotsafiles/dccc1820-build.3511.log
# [00:02:22][dccc1820][ main:lotsafiles.bst  ] SUCCESS Build Complete

bst build compose.bst

# (run 1)
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# [--:--:--][813eacd9][ main:compose.bst     ] START   Starting build
# [--:--:--][813eacd9][build:compose.bst     ] START
dpkg-ic/compose/813eacd9-build.26055.log
# [--:--:--][813eacd9][build:compose.bst     ] START   Staging dependencies
# [00:00:26][813eacd9][build:compose.bst     ] SUCCESS Staging dependencies
# [--:--:--][813eacd9][build:compose.bst     ] START   Integrating sandbox
# [--:--:--][813eacd9][build:compose.bst     ] INFO    Integration
effected 0 files
# [00:00:03][813eacd9][build:compose.bst     ] SUCCESS Integrating sandbox
# [--:--:--][813eacd9][build:compose.bst     ] START   Caching Artifact
# [00:00:59][813eacd9][build:compose.bst     ] SUCCESS Caching Artifact
# [00:01:56][813eacd9][build:compose.bst     ] SUCCESS
dpkg-ic/compose/813eacd9-build.26055.log
# [00:01:56][813eacd9][ main:compose.bst     ] SUCCESS Build Complete

# (run 2)
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# [--:--:--][f0c61f16][ main:compose.bst     ] START   Starting build
# [--:--:--][f0c61f16][build:compose.bst     ] START
lotsafiles/compose/f0c61f16-build.3518.log
# [--:--:--][f0c61f16][build:compose.bst     ] START   Staging dependencies
# [00:00:24][f0c61f16][build:compose.bst     ] SUCCESS Staging dependencies
# [--:--:--][f0c61f16][build:compose.bst     ] START   Integrating sandbox
# [--:--:--][f0c61f16][build:compose.bst     ] INFO    Integration
effected 0 files
# [00:00:02][f0c61f16][build:compose.bst     ] SUCCESS Integrating sandbox
# [--:--:--][f0c61f16][build:compose.bst     ] START   Caching Artifact
# [00:00:58][f0c61f16][build:compose.bst     ] SUCCESS Caching Artifact
# [00:01:59][f0c61f16][build:compose.bst     ] SUCCESS
lotsafiles/compose/f0c61f16-build.3518.log
# [00:01:59][f0c61f16][ main:compose.bst     ] SUCCESS Build Complete

--- Testing with a single large file to stage ---

echo 'name: bigfile' > project.conf

mkdir bigfile
cd bigfile
time dd if=/dev/urandom bs=1M count=2000 of=bigfile
# 2000+0 records in
# 2000+0 records out
# 2097152000 bytes (2.1 GB, 2.0 GiB) copied, 12.3964 s, 169 MB/s
# real 0m12.400s
# user 0m0.000s
# sys 0m10.890s
cd ..

cat <<EOF > bigfile.bst
kind: import

sources:
- kind: local
  path: bigfile
EOF

cat <<EOF > compose.bst
kind: compose

depends:
- filename: bigfile.bst
  type: build
EOF

bst build bigfile.bst

# (run 1)
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# [--:--:--][132c00a6][ main:bigfiles.bst  ] START   Starting build
# [--:--:--][132c00a6][build:bigfiles.bst  ] START
dpkg-ic/bigfiles/132c00a6-build.26091.log
# [--:--:--][132c00a6][build:bigfiles.bst  ] START   Staging sources
# [00:00:08][132c00a6][build:bigfiles.bst  ] SUCCESS Staging sources
# [--:--:--][132c00a6][build:bigfiles.bst  ] START   Caching Artifact
# [00:00:23][132c00a6][build:bigfiles.bst  ] SUCCESS Caching Artifact
# [00:00:40][132c00a6][build:bigfiles.bst  ] SUCCESS
dpkg-ic/bigfiles/132c00a6-build.26091.log
# [00:00:40][132c00a6][ main:bigfiles.bst  ] SUCCESS Build Complete

# (run 2)
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# [--:--:--][dce64cad][ main:bigfile.bst  ] START   Starting build
# [--:--:--][dce64cad][build:bigfile.bst  ] START
bigfile/bigfile/dce64cad-build.3535.log
# [--:--:--][dce64cad][build:bigfile.bst  ] START   Staging sources
# [00:00:07][dce64cad][build:bigfile.bst  ] SUCCESS Staging sources
# [--:--:--][dce64cad][build:bigfile.bst  ] START   Caching Artifact
# [00:00:21][dce64cad][build:bigfile.bst  ] SUCCESS Caching Artifact
# [00:00:37][dce64cad][build:bigfile.bst  ] SUCCESS
bigfile/bigfile/dce64cad-build.3535.log
# [00:00:37][dce64cad][ main:bigfile.bst  ] SUCCESS Build Complete

bst build compose.bst

# (run 1)
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# [--:--:--][170c448f][ main:composebig.bst] START   Starting build
# [--:--:--][170c448f][build:composebig.bst] START
dpkg-ic/composebig/170c448f-build.26125.log
# [--:--:--][170c448f][build:composebig.bst] START   Staging dependencies
# [00:00:00][170c448f][build:composebig.bst] SUCCESS Staging dependencies
# [--:--:--][170c448f][build:composebig.bst] START   Integrating sandbox
# [--:--:--][170c448f][build:composebig.bst] INFO    Integration
effected 0 files
# [00:00:00][170c448f][build:composebig.bst] SUCCESS Integrating sandbox
# [--:--:--][170c448f][build:composebig.bst] START   Caching Artifact
# [00:00:16][170c448f][build:composebig.bst] SUCCESS Caching Artifact
# [00:00:16][170c448f][build:composebig.bst] SUCCESS
dpkg-ic/composebig/170c448f-build.26125.log
# [00:00:16][170c448f][ main:composebig.bst] SUCCESS Build Complete

# (run 2)
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# [--:--:--][af7de14f][ main:compose.bst  ] START   Starting build
# [--:--:--][af7de14f][build:compose.bst  ] START
bigfile/compose/af7de14f-build.3542.log
# [--:--:--][af7de14f][build:compose.bst  ] START   Staging dependencies
# [00:00:00][af7de14f][build:compose.bst  ] SUCCESS Staging dependencies
# [--:--:--][af7de14f][build:compose.bst  ] START   Integrating sandbox
# [--:--:--][af7de14f][build:compose.bst  ] INFO    Integration effected 0 files
# [00:00:00][af7de14f][build:compose.bst  ] SUCCESS Integrating sandbox
# [--:--:--][af7de14f][build:compose.bst  ] START   Caching Artifact
# [00:00:18][af7de14f][build:compose.bst  ] SUCCESS Caching Artifact
# [00:00:18][af7de14f][build:compose.bst  ] SUCCESS
bigfile/compose/af7de14f-build.3542.log
# [00:00:18][af7de14f][ main:compose.bst  ] SUCCESS Build Complete