[Notes] [Git][BuildStream/buildstream][tiagogomes/system

Tiago Gomes pushed to branch tiagogomes/system_tests at BuildStream / buildstream

Commits:

f4fe6b6a

by Tiago Gomes at 2018-08-07T13:28:30Z

elementjob: stop calculating cache size

Calculating the cache size on the child_process_data() is racy, as this
runs in the main process which does not obtain exclusive access to the
cache resource.

This calculation also seems unnecessary, as it will triggered when
BuildQueue.done() is called.

75ee55f3

by Tiago Gomes at 2018-08-07T13:29:36Z

cascache: move tmp directory one level up

The tmp directory is filled when adding an artifact with temporary
files. This causes problems on calculate_cache_size() because we could
try to a attemp to do a stat() on a temporary file that meanwhile has
been reaped.

Handle this by not considering the tmp directory when calculating the
cache size.

8de0181a

by Tiago Gomes at 2018-08-07T13:29:36Z

WIP Add end-to-end tests

TODO:
* Uncomment when clausule.
* Remove debug information.

4 changed files:

.gitlab-ci.yml
buildstream/_artifactcache/cascache.py
buildstream/_scheduler/jobs/elementjob.py
buildstream/_scheduler/queues/queue.py

Changes:

.gitlab-ci.yml

@@ -84,6 +84,8 @@ source_dist:
    - cd ../..
    - mkdir -p coverage-linux/
    - cp dist/buildstream/.coverage.* coverage-linux/coverage."${CI_JOB_NAME}"
 +  except:
 +  - schedules
    artifacts:
      paths:
      - coverage-linux/
@@ -125,6 +127,8 @@ tests-unix:
      - cd ../..
      - mkdir -p coverage-unix/
      - cp dist/buildstream/.coverage.* coverage-unix/coverage.unix
 +  except:
 +  - schedules
    artifacts:
      paths:
      - coverage-unix/
@@ -146,10 +150,46 @@ docs:
    - make BST_FORCE_SESSION_REBUILD=1 -C doc
    - cd ../..
    - mv dist/buildstream/doc/build/html public
 +  except:
 +  - schedules
    artifacts:
      paths:
      - public/
 +.system-tests: &system-tests-template
 +  stage: test
 +  variables:
 +    XDG_CACHE_HOME: /cache
 +    bst_ext_url: git+https://gitlab.com/BuildStream/bst-external.git
 +    bst_ext_ref: 31376f750df713b3613d5348f93100c7e0ef68c0 # 0.3
 +    fd_sdk_ref: 27cb7575f57e2922331facbd252dc690807c7182 # 18.08
 +    expected_cache_key: 49816d4c
 +  before_script:
 +  - (cd dist && ./unpack.sh && cd buildstream && pip3 install .)
 +  - pip3 install --user -e ${bst_ext_url}@${bst_ext_ref}#egg=bst_ext
 +  - git clone https://gitlab.com/freedesktop-sdk/freedesktop-sdk.git
 +  - git -C freedesktop-sdk checkout ${fd_sdk_ref}
 +  #when:
 +  #- schedules
++
 +system-tests:
 +  <<: *system-tests-template
 +  script:
 +  - make -C freedesktop-sdk
 +  - pwd
 +  - cache_key=$(bst show all.bst --deps none --format %{key})
 +  - test ${cache_key} == ${expected_cache_key}
++
 +system-tests-no-cache:
 +  <<: *system-tests-template
 +  script:
 +  - sed -i '/artifacts:/,+1 d' freedesktop-sdk/bootstrap/project.conf
 +  - sed -i '/artifacts:/,+1 d' freedesktop-sdk/project.conf
 +  - make -C freedesktop-sdk
 +  - pwd
 +  - cache_key=$(bst show all.bst --deps none --format %{key})
 +  - test ${cache_key} == ${expected_cache_key}
++
  #####################################################
  #                    Post stage                     #
@@ -174,6 +214,8 @@ codequality:
          --volume "$PWD":/code
          --volume /var/run/docker.sock:/var/run/docker.sock
          "registry.gitlab.com/gitlab-org/security-products/codequality:$SP_VERSION" /code
 +  except:
 +  - schedules
    artifacts:
      paths: [codeclimate.json]
@@ -199,6 +241,8 @@ analysis:
      radon raw -s -j buildstream > analysis/raw.json
      radon raw -s buildstream
 +  except:
 +  - schedules
    artifacts:
      paths:
      - analysis/
@@ -223,6 +267,8 @@ coverage:
    - tests-fedora-27
    - tests-unix
    - source_dist
 +  except:
 +  - schedules
  # Deploy, only for merges which land on master branch.
+ #
@@ -247,3 +293,5 @@ pages:
    # See https://gitlab.com/gitlab-org/gitlab-ce/issues/35141
+   #
    - master
 +  except:
 +  - schedules

buildstream/_artifactcache/cascache.py

@@ -56,7 +56,9 @@ class CASCache(ArtifactCache):
          super().__init__(context)
          self.casdir = os.path.join(context.artifactdir, 'cas')
 -        os.makedirs(os.path.join(self.casdir, 'tmp'), exist_ok=True)
++
 +        self.tmpdir = os.path.join(context.artifactdir, 'tmp')
 +        os.makedirs(self.tmpdir, exist_ok=True)
          self._enable_push = enable_push
@@ -394,7 +396,7 @@ class CASCache(ArtifactCache):
          try:
              h = hashlib.sha256()
              # Always write out new file to avoid corruption if input file is modified
 -            with tempfile.NamedTemporaryFile(dir=os.path.join(self.casdir, 'tmp')) as out:
 +            with tempfile.NamedTemporaryFile(dir=self.tmpdir) as out:
                  # Set mode bits to 0644
                  os.chmod(out.name, stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH)
@@ -764,7 +766,7 @@ class CASCache(ArtifactCache):
              # already in local cache
              return
 -        with tempfile.NamedTemporaryFile(dir=os.path.join(self.casdir, 'tmp')) as out:
 +        with tempfile.NamedTemporaryFile(dir=self.tmpdir) as out:
              self._fetch_blob(remote, tree, out)
              directory = remote_execution_pb2.Directory()
@@ -778,7 +780,7 @@ class CASCache(ArtifactCache):
                      # already in local cache
                      continue
 -                with tempfile.NamedTemporaryFile(dir=os.path.join(self.casdir, 'tmp')) as f:
 +                with tempfile.NamedTemporaryFile(dir=self.tmpdir) as f:
                      self._fetch_blob(remote, filenode.digest, f)
                      digest = self.add_object(path=f.name)

buildstream/_scheduler/jobs/elementjob.py

@@ -109,13 +109,11 @@ class ElementJob(Job):
          data = {}
          workspace = self._element._get_workspace()
 -        artifact_size = self._element._get_artifact_size()
 -        cache_size = self._element._get_artifact_cache().calculate_cache_size()
+-
          if workspace is not None:
              data['workspace'] = workspace.to_dict()
++
 +        artifact_size = self._element._get_artifact_size()
          if artifact_size is not None:
              data['artifact_size'] = artifact_size
 -        data['cache_size'] = cache_size
          return data

buildstream/_scheduler/queues/queue.py

@@ -300,8 +300,6 @@ class Queue():
          # Update values that need to be synchronized in the main task
          # before calling any queue implementation
          self._update_workspaces(element, job)
 -        if job.child_data:
 -            element._get_artifact_cache().cache_size = job.child_data.get('cache_size')
          # Give the result of the job to the Queue implementor,
          # and determine if it should be considered as processed

[Notes] [Git][BuildStream/buildstream][tiagogomes/system_tests] 3 commits: elementjob: stop calculating cache size

Tiago Gomes pushed to branch tiagogomes/system_tests at BuildStream / buildstream

Commits:

4 changed files:

Changes: