[Notes] [Git][BuildStream/buildstream][jjardon/ci_show

Javier Jardón pushed to branch jjardon/ci_show_timings at BuildStream / buildstream

Commits:

e1ed09aa

by Jim MacArthur at 2018-08-24T09:51:27Z

element.py: chmod 777 directories if delete fails.

Staging may produce directories with less than 'rwx' permissions which
will cause tempfile to throw an exception on deleting the temporary
directory if not changed.

01fe504b

by James Ennis at 2018-08-24T09:51:27Z

tar tests: Add tarball which contains a read-only dir

a3755c9a

by James Ennis at 2018-08-24T09:51:27Z

tar.py: Add test for a tarball containing a read-only dir

1bfcca1a

by James Ennis at 2018-08-24T10:31:41Z

Merge branch 'jmac/tempfile-extraction-bug' into 'master'

Correct crash after staging tars with read-only directories

See merge request BuildStream/buildstream!713

744925f3

by James Ennis at 2018-08-24T10:34:57Z

HACKING.rst: Remove -k flag for individual tests as this is not required

dfee6d75

by James Ennis at 2018-08-24T11:15:45Z

Merge branch 'jennis/correct_HACKING' into 'master'

'-k' flag not needed to run individual tests

See merge request BuildStream/buildstream!714

923bfaec

by Chandan Singh at 2018-08-24T11:22:33Z

doc: Add instructions to install BuildStream via PyPI

Add instructions to install and update BuildStream python package via
PyPI, and also make it the recommended method.

Part of https://gitlab.com/BuildStream/buildstream/issues/587.

2c3a6ab8

by Chandan Singh at 2018-08-24T12:01:56Z

Merge branch 'chandan/pip-install-instructions' into 'master'

doc: Add instructions to install BuildStream via PyPI

See merge request BuildStream/buildstream!717

f4745600

by Josh Smith at 2018-08-24T12:20:05Z

setup.cfg: Show timing of the 20 slowest tests

This helps identifying what tests are slow or if the CI got stuck on
some of them

7 changed files:

HACKING.rst
buildstream/element.py
doc/source/install_linux_distro.rst
setup.cfg
tests/sources/tar.py
+ tests/sources/tar/read-only/content/a.tar.gz
+ tests/sources/tar/read-only/target.bst

Changes:

HACKING.rst

@@ -465,12 +465,12 @@ If you want to run a specific test or a group of tests, you
  can specify a prefix to match. E.g. if you want to run all of
  the frontend tests you can do::
 -  ./setup.py test --addopts '-k tests/frontend/'
 +  ./setup.py test --addopts 'tests/frontend/'
  Specific tests can be chosen by using the :: delimeter after the test module.
  If you wanted to run the test_build_track test within frontend/buildtrack.py you could do::
 -  ./setup.py test --adopts '-k tests/frontend/buildtrack.py::test_build_track'
 +  ./setup.py test --addopts 'tests/frontend/buildtrack.py::test_build_track'
  We also have a set of slow integration tests that are disabled by
  default - you will notice most of them marked with SKIP in the pytest

buildstream/element.py

@@ -1361,8 +1361,12 @@ class Element(Plugin):
              if not vdirectory.is_empty():
                  raise ElementError("Staging directory '{}' is not empty".format(vdirectory))
 -            with tempfile.TemporaryDirectory() as temp_staging_directory:
 +            # While mkdtemp is advertised as using the TMP environment variable, it
 +            # doesn't, so this explicit extraction is necesasry.
 +            tmp_prefix = os.environ.get("TMP", None)
 +            temp_staging_directory = tempfile.mkdtemp(prefix=tmp_prefix)
 +            try:
                  workspace = self._get_workspace()
                  if workspace:
                      # If mount_workspaces is set and we're doing incremental builds,
@@ -1377,6 +1381,19 @@ class Element(Plugin):
                          source._stage(temp_staging_directory)
                  vdirectory.import_files(temp_staging_directory)
++
 +            finally:
 +                # Staging may produce directories with less than 'rwx' permissions
 +                # for the owner, which will break tempfile, so we need to use chmod
 +                # occasionally.
 +                def make_dir_writable(fn, path, excinfo):
 +                    os.chmod(os.path.dirname(path), 0o777)
 +                    if os.path.isdir(path):
 +                        os.rmdir(path)
 +                    else:
 +                        os.remove(path)
 +                shutil.rmtree(temp_staging_directory, onerror=make_dir_writable)
++
          # Ensure deterministic mtime of sources at build time
          vdirectory.set_deterministic_mtime()
          # Ensure deterministic owners of sources at build time

doc/source/install_linux_distro.rst

@@ -152,8 +152,29 @@ for advice on this.
  Installing
  ~~~~~~~~~~
 -Once you have the base system dependencies, you can clone the BuildStream
 -git repository and install it as a regular user::
 +Once you have the base system dependencies, you can install the BuildStream
 +python package as a regular user.
++
 +Via PyPI (recommended)
 +++++++++++++++++++++++
 +::
++
 +  pip3 install --user BuildStream
++
 +This will install latest stable version of BuildStream and its pure python
 +dependencies into your user's homedir in ``~/.local``.
++
 +Keep following the instructions below to ensure that the ``bst``
 +command is in your ``PATH`` and to enable bash completions for it.
++
 +.. note::
++
 +  If you want a specific version of BuildStream, you can install it using
 +  ``pip install --user BuildStream==<version-number>``
++
 +Via Git checkout
 +++++++++++++++++
 +::
    git clone https://gitlab.com/BuildStream/buildstream.git
    cd buildstream
@@ -206,9 +227,19 @@ to your ``~/.bash_completion``:
  Upgrading BuildStream
  ~~~~~~~~~~~~~~~~~~~~~
 -Assuming you have followed the default instructions above, all
 -you need to do to upgrade BuildStream is to update your local git
 -checkout::
++
 +Via PyPI
 +++++++++
++
 +If you installed BuildStream from PyPI, you can update it like so::
++
 +  pip install --user --upgrade BuildStream
++
 +Via Git checkout
 +++++++++++++++++
++
 +If you installed BuildStream from a local git checkout using ``-e`` option, all
 +you need to do to upgrade BuildStream is to update your local git checkout::
    cd /path/to/buildstream
    git pull --rebase

setup.cfg

@@ -11,7 +11,7 @@ parentdir_prefix = BuildStream-
  test=pytest
  [tool:pytest]
 -addopts = --verbose --basetemp ./tmp --pep8 --pylint --pylint-rcfile=.pylintrc --cov=buildstream --cov-config .coveragerc
 +addopts = --verbose --basetemp ./tmp --pep8 --pylint --pylint-rcfile=.pylintrc --cov=buildstream --cov-config .coveragerc --durations=20
  norecursedirs = tests/integration/project integration-cache tmp __pycache__ .eggs
  python_files = tests/*/*.py
  pep8maxlinelength = 119

tests/sources/tar.py

@@ -3,6 +3,7 @@ import pytest
  import tarfile
  import tempfile
  import subprocess
 +from shutil import copyfile, rmtree
  from buildstream._exceptions import ErrorDomain
  from buildstream import _yaml
@@ -257,3 +258,47 @@ def test_stage_default_basedir_lzip(cli, tmpdir, datafiles, srcdir):
      original_contents = list_dir_contents(original_dir)
      checkout_contents = list_dir_contents(checkoutdir)
      assert(checkout_contents == original_contents)
++
++
 +# Test that a tarball that contains a read only dir works
 +@pytest.mark.datafiles(os.path.join(DATA_DIR, 'read-only'))
 +def test_read_only_dir(cli, tmpdir, datafiles):
 +    try:
 +        project = os.path.join(datafiles.dirname, datafiles.basename)
 +        generate_project(project, tmpdir)
++
 +        # Get the tarball in tests/sources/tar/read-only/content
 +        #
 +        # NOTE that we need to do this because tarfile.open and tar.add()
 +        # are packing the tar up with writeable files and dirs
 +        tarball = os.path.join(str(datafiles), 'content', 'a.tar.gz')
 +        if not os.path.exists(tarball):
 +            raise FileNotFoundError('{} does not exist'.format(tarball))
 +        copyfile(tarball, os.path.join(str(tmpdir), 'a.tar.gz'))
++
 +        # Because this test can potentially leave directories behind
 +        # which are difficult to remove, ask buildstream to use
 +        # our temp directory, so we can clean up.
 +        tmpdir_str = str(tmpdir)
 +        if not tmpdir_str.endswith(os.path.sep):
 +            tmpdir_str += os.path.sep
 +        env = {"TMP": tmpdir_str}
++
 +        # Track, fetch, build, checkout
 +        result = cli.run(project=project, args=['track', 'target.bst'], env=env)
 +        result.assert_success()
 +        result = cli.run(project=project, args=['fetch', 'target.bst'], env=env)
 +        result.assert_success()
 +        result = cli.run(project=project, args=['build', 'target.bst'], env=env)
 +        result.assert_success()
++
 +    finally:
++
 +        # Make tmpdir deletable no matter what happens
 +        def make_dir_writable(fn, path, excinfo):
 +            os.chmod(os.path.dirname(path), 0o777)
 +            if os.path.isdir(path):
 +                os.rmdir(path)
 +            else:
 +                os.remove(path)
 +        rmtree(str(tmpdir), onerror=make_dir_writable)

tests/sources/tar/read-only/content/a.tar.gz

No preview for this file type

tests/sources/tar/read-only/target.bst

 +kind: import
 +description: The kind of this element is irrelevant.
 +sources:
 +- kind: tar
 +  url: tmpdir:/a.tar.gz
 +  ref: foo

[Notes] [Git][BuildStream/buildstream][jjardon/ci_show_timings] 9 commits: element.py: chmod 777 directories if delete fails.

Javier Jardón pushed to branch jjardon/ci_show_timings at BuildStream / buildstream

Commits:

7 changed files:

Changes: