GNOME.org
 

[Notes] [Git][BuildStream/buildstream][tiagogomes/issue-500] 7 commits: Add remote source plugin

Title: GitLab

Tiago Gomes pushed to branch tiagogomes/issue-500 at BuildStream / buildstream

Commits:

24 changed files:

Changes:

  • HACKING.rst
    ... ... @@ -294,7 +294,7 @@ committed with that.
    294 294 
     To do this, first ensure you have ``click_man`` installed, possibly
    295 295 
     with::
    296 296
    297 
    -  pip install --user click_man
    297 
    +  pip3 install --user click_man
    298 298
    299 299 
     Then, in the toplevel directory of buildstream, run the following::
    300 300
    ... ... @@ -450,7 +450,7 @@ To run the tests, just type::
    450 450 
     At the toplevel.
    451 451
    452 452 
     When debugging a test, it can be desirable to see the stdout
    453 
    -and stderr generated by a test, to do this use the --addopts
    453 
    +and stderr generated by a test, to do this use the ``--addopts``
    454 454 
     function to feed arguments to pytest as such::
    455 455
    456 456 
       ./setup.py test --addopts -s
    ... ... @@ -530,7 +530,7 @@ tool.
    530 530
    531 531 
     Python provides `cProfile <https://docs.python.org/3/library/profile.html>`_
    532 532 
     which gives you a list of all functions called during execution and how much
    533 
    -time was spent in each function. Here is an example of running `bst --help`
    533 
    +time was spent in each function. Here is an example of running ``bst --help``
    534 534 
     under cProfile:
    535 535
    536 536 
         python3 -m cProfile -o bst.cprofile -- $(which bst) --help

  • buildstream/_versions.py
    ... ... @@ -23,7 +23,7 @@
    23 23 
     # This version is bumped whenever enhancements are made
    24 24 
     # to the `project.conf` format or the core element format.
    25 25 
     #
    26 
    -BST_FORMAT_VERSION = 9
    26 
    +BST_FORMAT_VERSION = 10
    27 27
    28 28
    29 29 
     # The base BuildStream artifact version

  • buildstream/plugins/sources/remote.py
    1 
    +#
    2 
    +#  Copyright Bloomberg Finance LP
    3 
    +#
    4 
    +#  This program is free software; you can redistribute it and/or
    5 
    +#  modify it under the terms of the GNU Lesser General Public
    6 
    +#  License as published by the Free Software Foundation; either
    7 
    +#  version 2 of the License, or (at your option) any later version.
    8 
    +#
    9 
    +#  This library is distributed in the hope that it will be useful,
    10 
    +#  but WITHOUT ANY WARRANTY; without even the implied warranty of
    11 
    +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
    12 
    +#  Lesser General Public License for more details.
    13 
    +#
    14 
    +#  You should have received a copy of the GNU Lesser General Public
    15 
    +#  License along with this library. If not, see <http://www.gnu.org/licenses/>.
    16 
    +#
    17 
    +#  Authors:
    18 
    +#        Ed Baunton <ebaunton1 bloomberg net>
    19 
    +
    20 
    +"""
    21 
    +remote - stage files from remote urls
    22 
    +=====================================
    23 
    +
    24 
    +**Usage:**
    25 
    +
    26 
    +.. code:: yaml
    27 
    +
    28 
    +   # Specify the remote source kind
    29 
    +   kind: remote
    30 
    +
    31 
    +   # Optionally specify a relative staging directory
    32 
    +   # directory: path/to/stage
    33 
    +
    34 
    +   # Optionally specify a relative staging filename.
    35 
    +   # If not specified, the basename of the url will be used.
    36 
    +   # filename: customfilename
    37 
    +
    38 
    +   # Specify the url. Using an alias defined in your project
    39 
    +   # configuration is encouraged. 'bst track' will update the
    40 
    +   # sha256sum in 'ref' to the downloaded file's sha256sum.
    41 
    +   url: upstream:foo
    42 
    +
    43 
    +   # Specify the ref. It's a sha256sum of the file you download.
    44 
    +   ref: 6c9f6f68a131ec6381da82f2bff978083ed7f4f7991d931bfa767b7965ebc94b
    45 
    +
    46 
    +.. note::
    47 
    +
    48 
    +   The ``remote`` plugin is available since :ref:`format version 10 <project_format_version>`
    49 
    +
    50 
    +"""
    51 
    +import os
    52 
    +from buildstream import SourceError, utils
    53 
    +from ._downloadablefilesource import DownloadableFileSource
    54 
    +
    55 
    +
    56 
    +class RemoteSource(DownloadableFileSource):
    57 
    +    # pylint: disable=attribute-defined-outside-init
    58 
    +
    59 
    +    def configure(self, node):
    60 
    +        super().configure(node)
    61 
    +
    62 
    +        self.filename = self.node_get_member(node, str, 'filename', os.path.basename(self.url))
    63 
    +
    64 
    +        if os.sep in self.filename:
    65 
    +            raise SourceError('{}: filename parameter cannot contain directories'.format(self),
    66 
    +                              reason="filename-contains-directory")
    67 
    +        self.node_validate(node, DownloadableFileSource.COMMON_CONFIG_KEYS + ['filename'])
    68 
    +
    69 
    +    def get_unique_key(self):
    70 
    +        return super().get_unique_key() + [self.filename]
    71 
    +
    72 
    +    def stage(self, directory):
    73 
    +        # Same as in local plugin, don't use hardlinks to stage sources, they
    74 
    +        # are not write protected in the sandbox.
    75 
    +        dest = os.path.join(directory, self.filename)
    76 
    +        with self.timed_activity("Staging remote file to {}".format(dest)):
    77 
    +            utils.safe_copy(self._get_mirror_file(), dest)
    78 
    +
    79 
    +
    80 
    +def setup():
    81 
    +    return RemoteSource

  • buildstream/utils.py
    ... ... @@ -802,6 +802,12 @@ def _process_list(srcdir, destdir, filelist, actionfunc, result,
    802 802 
                 os.mknod(destpath, file_stat.st_mode, file_stat.st_rdev)
    803 803 
                 os.chmod(destpath, file_stat.st_mode)
    804 804
    805 
    +        elif stat.S_ISFIFO(mode):
    806 
    +            os.mkfifo(destpath, mode)
    807 
    +
    808 
    +        elif stat.S_ISSOCK(mode):
    809 
    +            pass
    810 
    +
    805 811 
             else:
    806 812 
                 # Unsupported type.
    807 813 
                 raise UtilError('Cannot extract {} into staging-area. Unsupported type.'.format(srcpath))

  • doc/source/core_plugins.rst
    ... ... @@ -50,6 +50,7 @@ Sources
    50 50 
        :maxdepth: 1
    51 51
    52 52 
        sources/local
    53 
    +   sources/remote
    53 54 
        sources/tar
    54 55 
        sources/zip
    55 56 
        sources/git

  • doc/source/examples/flatpak-autotools.rst
    ... ... @@ -58,7 +58,7 @@ it's important to note two things:
    58 58 
     * **source**: We only want to extract the ``files/`` directory from the SDK,
    59 59
    60 60 
       This is becase Flatpak runtimes dont start at the root of the OSTree checkout,
    61 
    -  instead the actual files start in the ``files//`` subdirectory
    61 
    +  instead the actual files start in the ``files/`` subdirectory
    62 62
    63 63 
     * **target**: The content we've extracted should be staged at ``/usr``
    64 64

  • doc/source/format_declaring.rst
    ... ... @@ -400,7 +400,7 @@ dependency and that all referenced variables are declared, the following is fine
    400 400 
        It should be noted that variable resolution only happens after all
    401 401 
        :ref:`Element Composition <format_composition>` has already taken place.
    402 402
    403 
    -   This is to say that overriding ``%{version}`` at a higher priority will effect
    403 
    +   This is to say that overriding ``%{version}`` at a higher priority will affect
    404 404 
        the final result of ``%{release-text}``.
    405 405
    406 406

  • doc/source/format_project.rst
    ... ... @@ -548,7 +548,7 @@ The defaults environment for the build sandbox is defined here.
    548 548 
          PATH: /usr/bin:/bin:/usr/sbin:/sbin
    549 549
    550 550 
     Additionally, the special ``environment-nocache`` list which specifies
    551 
    -which environment variables do not effect build output, and are thus
    551 
    +which environment variables do not affect build output, and are thus
    552 552 
     not considered in the calculation of artifact keys can be defined here.
    553 553
    554 554 
     .. code:: yaml
    ... ... @@ -558,7 +558,7 @@ not considered in the calculation of artifact keys can be defined here.
    558 558
    559 559 
     Note that the ``environment-nocache`` list only exists so that we can
    560 560 
     control parameters such as ``make -j ${MAXJOBS}``, allowing us to control
    561 
    -the number of jobs for a given build without effecting the resulting
    561 
    +the number of jobs for a given build without affecting the resulting
    562 562 
     cache key.
    563 563
    564 564

  • doc/source/install_docker.rst
    ... ... @@ -38,8 +38,8 @@ be able to integrate with Bash tab-completion if you invoke it in this way.
    38 38
    39 39 
     Two Docker volumes are set up by the ``bst-here`` script:
    40 40
    41 
    - * buildstream-cache -- mounted at ``~/.cache/buildstream``
    42 
    - * buildstream-config -- mounted at ``~/.config/``
    41 
    + * ``buildstream-cache --`` mounted at ``~/.cache/buildstream``
    42 
    + * ``buildstream-config --`` mounted at ``~/.config/``
    43 43
    44 44 
     These are necessary so that your BuildStream cache and configuration files
    45 45 
     persist between invocations of ``bst-here``.

  • doc/source/install_linux_distro.rst
    ... ... @@ -77,17 +77,17 @@ setup as described `here <https://backports.debian.org/Instructions/>`_
    77 77
    78 78 
     By adding the following line to your sources.list::
    79 79
    80 
    -  deb http://ftp.debian.org/debian stretch-backports main
    80 
    +  deb http://deb.debian.org/debian stretch-backports main
    81 81
    82 82 
     And then running::
    83 83
    84 
    -  sudo apt-get update
    84 
    +  sudo apt update
    85 85
    86 86 
     At this point you should be able to get the system requirements for the default plugins with::
    87 87
    88 
    -  sudo apt-get install \
    88 
    +  sudo apt install \
    89 89 
           bzr git lzip patch python3-arpy python3-gi
    90 
    -  sudo apt-get install -t stretch-backports \
    90 
    +  sudo apt install -t stretch-backports \
    91 91 
           gir1.2-ostree-1.0 ostree
    92 92
    93 93 
     Buster or Sid
    ... ... @@ -177,8 +177,8 @@ checkout::
    177 177 
       cd /path/to/buildstream
    178 178 
       git pull --rebase
    179 179
    180 
    -If you did not specify the ``-e`` option at install time, you will
    181 
    -need to cleanly reinstall BuildStream::
    180 
    +If you did not specify the ``-e`` option at install time or the dependancies
    181 
    +have changed, you will need to cleanly reinstall BuildStream::
    182 182
    183 183 
       pip3 uninstall buildstream
    184 184 
       cd /path/to/buildstream

  • doc/source/tutorial/autotools.rst
    ... ... @@ -109,7 +109,7 @@ changes the output.
    109 109
    110 110 
     Observe where the variables are declared in the :ref:`builtin defaults
    111 111 
     <project_builtin_defaults>` and :mod:`autotools <elements.autotools>` element
    112 
    -documentation, and how overriding these affects the resolved set of variables.
    112 
    +documentation, and how overriding these effects the resolved set of variables.
    113 113
    114 114
    115 115 
     Using the project

  • tests/sources/remote.py
    1 
    +import os
    2 
    +import pytest
    3 
    +
    4 
    +from buildstream._exceptions import ErrorDomain
    5 
    +from buildstream import _yaml
    6 
    +from tests.testutils import cli
    7 
    +
    8 
    +DATA_DIR = os.path.join(
    9 
    +    os.path.dirname(os.path.realpath(__file__)),
    10 
    +    'remote',
    11 
    +)
    12 
    +
    13 
    +
    14 
    +def generate_project(project_dir, tmpdir):
    15 
    +    project_file = os.path.join(project_dir, "project.conf")
    16 
    +    _yaml.dump({
    17 
    +        'name': 'foo',
    18 
    +        'aliases': {
    19 
    +            'tmpdir': "file:///" + str(tmpdir)
    20 
    +        }
    21 
    +    }, project_file)
    22 
    +
    23 
    +
    24 
    +# Test that without ref, consistency is set appropriately.
    25 
    +@pytest.mark.datafiles(os.path.join(DATA_DIR, 'no-ref'))
    26 
    +def test_no_ref(cli, tmpdir, datafiles):
    27 
    +    project = os.path.join(datafiles.dirname, datafiles.basename)
    28 
    +    generate_project(project, tmpdir)
    29 
    +    assert cli.get_element_state(project, 'target.bst') == 'no reference'
    30 
    +
    31 
    +
    32 
    +# Here we are doing a fetch on a file that doesn't exist. target.bst
    33 
    +# refers to 'file' but that file is not present.
    34 
    +@pytest.mark.datafiles(os.path.join(DATA_DIR, 'missing-file'))
    35 
    +def test_missing_file(cli, tmpdir, datafiles):
    36 
    +    project = os.path.join(datafiles.dirname, datafiles.basename)
    37 
    +    generate_project(project, tmpdir)
    38 
    +
    39 
    +    # Try to fetch it
    40 
    +    result = cli.run(project=project, args=[
    41 
    +        'fetch', 'target.bst'
    42 
    +    ])
    43 
    +
    44 
    +    result.assert_main_error(ErrorDomain.STREAM, None)
    45 
    +    result.assert_task_error(ErrorDomain.SOURCE, None)
    46 
    +
    47 
    +
    48 
    +@pytest.mark.datafiles(os.path.join(DATA_DIR, 'path-in-filename'))
    49 
    +def test_path_in_filename(cli, tmpdir, datafiles):
    50 
    +    project = os.path.join(datafiles.dirname, datafiles.basename)
    51 
    +    generate_project(project, tmpdir)
    52 
    +
    53 
    +    # Try to fetch it
    54 
    +    result = cli.run(project=project, args=[
    55 
    +        'fetch', 'target.bst'
    56 
    +    ])
    57 
    +
    58 
    +    # The bst file has a / in the filename param
    59 
    +    result.assert_main_error(ErrorDomain.SOURCE, "filename-contains-directory")
    60 
    +
    61 
    +
    62 
    +@pytest.mark.datafiles(os.path.join(DATA_DIR, 'single-file'))
    63 
    +def test_simple_file_build(cli, tmpdir, datafiles):
    64 
    +    project = os.path.join(datafiles.dirname, datafiles.basename)
    65 
    +    generate_project(project, tmpdir)
    66 
    +    checkoutdir = os.path.join(str(tmpdir), "checkout")
    67 
    +
    68 
    +    # Try to fetch it
    69 
    +    result = cli.run(project=project, args=[
    70 
    +        'fetch', 'target.bst'
    71 
    +    ])
    72 
    +    result.assert_success()
    73 
    +
    74 
    +    result = cli.run(project=project, args=[
    75 
    +        'build', 'target.bst'
    76 
    +    ])
    77 
    +    result.assert_success()
    78 
    +
    79 
    +    result = cli.run(project=project, args=[
    80 
    +        'checkout', 'target.bst', checkoutdir
    81 
    +    ])
    82 
    +    result.assert_success()
    83 
    +    # Note that the url of the file in target.bst is actually /dir/file
    84 
    +    # but this tests confirms we take the basename
    85 
    +    assert(os.path.exists(os.path.join(checkoutdir, 'file')))
    86 
    +
    87 
    +
    88 
    +@pytest.mark.datafiles(os.path.join(DATA_DIR, 'single-file-custom-name'))
    89 
    +def test_simple_file_custom_name_build(cli, tmpdir, datafiles):
    90 
    +    project = os.path.join(datafiles.dirname, datafiles.basename)
    91 
    +    generate_project(project, tmpdir)
    92 
    +    checkoutdir = os.path.join(str(tmpdir), "checkout")
    93 
    +
    94 
    +    # Try to fetch it
    95 
    +    result = cli.run(project=project, args=[
    96 
    +        'fetch', 'target.bst'
    97 
    +    ])
    98 
    +    result.assert_success()
    99 
    +
    100 
    +    result = cli.run(project=project, args=[
    101 
    +        'build', 'target.bst'
    102 
    +    ])
    103 
    +    result.assert_success()
    104 
    +
    105 
    +    result = cli.run(project=project, args=[
    106 
    +        'checkout', 'target.bst', checkoutdir
    107 
    +    ])
    108 
    +    result.assert_success()
    109 
    +    assert(not os.path.exists(os.path.join(checkoutdir, 'file')))
    110 
    +    assert(os.path.exists(os.path.join(checkoutdir, 'custom-file')))
    111 
    +
    112 
    +
    113 
    +@pytest.mark.datafiles(os.path.join(DATA_DIR, 'unique-keys'))
    114 
    +def test_unique_key(cli, tmpdir, datafiles):
    115 
    +    '''This test confirms that the 'filename' parameter is honoured when it comes
    116 
    +    to generating a cache key for the source.
    117 
    +    '''
    118 
    +    project = os.path.join(datafiles.dirname, datafiles.basename)
    119 
    +    generate_project(project, tmpdir)
    120 
    +    assert cli.get_element_state(project, 'target.bst') == "fetch needed"
    121 
    +    assert cli.get_element_state(project, 'target-custom.bst') == "fetch needed"
    122 
    +    # Try to fetch it
    123 
    +    result = cli.run(project=project, args=[
    124 
    +        'fetch', 'target.bst'
    125 
    +    ])
    126 
    +
    127 
    +    # We should download the file only once
    128 
    +    assert cli.get_element_state(project, 'target.bst') == 'buildable'
    129 
    +    assert cli.get_element_state(project, 'target-custom.bst') == 'buildable'
    130 
    +
    131 
    +    # But the cache key is different because the 'filename' is different.
    132 
    +    assert cli.get_element_key(project, 'target.bst') != \
    133 
    +        cli.get_element_key(project, 'target-custom.bst')

  • tests/sources/remote/missing-file/target.bst
    1 
    +kind: autotools
    2 
    +description: The kind of this element is irrelevant.
    3 
    +sources:
    4 
    +- kind: remote
    5 
    +  url: tmpdir:/file
    6 
    +  ref: abcdef
    7 
    +  filename: filename

  • tests/sources/remote/no-ref/file
    1 
    +filecontent

  • tests/sources/remote/no-ref/target.bst
    1 
    +kind: autotools
    2 
    +description: The kind of this element is irrelevant.
    3 
    +sources:
    4 
    +- kind: remote
    5 
    +  url: tmpdir:/file

  • tests/sources/remote/path-in-filename/dir/file

  • tests/sources/remote/path-in-filename/target.bst
    1 
    +kind: import
    2 
    +description: test
    3 
    +sources:
    4 
    +- kind: remote
    5 
    +  url: tmpdir:/dir/file
    6 
    +  ref: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    7 
    +  filename: path/to/file

  • tests/sources/remote/single-file-custom-name/dir/file

  • tests/sources/remote/single-file-custom-name/target.bst
    1 
    +kind: import
    2 
    +description: test
    3 
    +sources:
    4 
    +- kind: remote
    5 
    +  url: tmpdir:/dir/file
    6 
    +  ref: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    7 
    +  filename: custom-file

  • tests/sources/remote/single-file/dir/file

  • tests/sources/remote/single-file/target.bst
    1 
    +kind: import
    2 
    +description: test
    3 
    +sources:
    4 
    +- kind: remote
    5 
    +  url: tmpdir:/dir/file
    6 
    +  ref: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

  • tests/sources/remote/unique-keys/dir/file

  • tests/sources/remote/unique-keys/target-custom.bst
    1 
    +kind: import
    2 
    +description: test
    3 
    +sources:
    4 
    +- kind: remote
    5 
    +  url: tmpdir:/dir/file
    6 
    +  ref: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    7 
    +  filename: some-custom-file

  • tests/sources/remote/unique-keys/target.bst
    1 
    +kind: import
    2 
    +description: test
    3 
    +sources:
    4 
    +- kind: remote
    5 
    +  url: tmpdir:/dir/file
    6 
    +  ref: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

    • —
    View it on GitLab.
    You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.

    [Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]