[Notes] [Git][BuildStream/buildstream][jennis/refactor_artifact

James Ennis pushed to branch jennis/refactor_artifact_log at BuildStream / buildstream

Commits:

b6b5a638

by James Ennis at 2019-01-31T20:36:18Z

element.py: Make calculate_cache_key() API private

6b0e2900

by James Ennis at 2019-01-31T20:36:18Z

element.py: Return early in __init_defaults if no plugin config

c42bb08e

by James Ennis at 2019-01-31T21:18:20Z

metaelement.py: Provide constructor with default keyword arguments

bd4eca8f

by James Ennis at 2019-02-01T10:22:06Z

_artifactelement.py: New ArtifactElement object (derived from Element)

This object should be used when we want to handle artifact refs directly
from the command line.
An ArtifactElementError has also been added to _exceptions.py

dbe72e3c

by James Ennis at 2019-02-01T10:22:06Z

_project.py: Add create_artifact_element() method

61481582

by James Ennis at 2019-02-01T10:22:06Z

_stream.py: Add the _classify_artifacts() helper

446854fc

by James Ennis at 2019-02-01T10:22:06Z

_stream.py: Modify behaviour of _classify_artifacts()

* There is no need for this method to use a cas object.
* Search for artifact globs in the project's element path
* An artifact key is always 64 chars long

7b7d6fb0

by James Ennis at 2019-02-01T10:22:06Z

_stream.py: Allow loading to handle artifact refs

9ac33817

by James Ennis at 2019-02-01T10:22:06Z

_artifactcache.py: Add get_artifacts_log() method

A CasBasedDirectory object of an artifacts logs can be obtained
with ArtifactCache.get_artifacts_log(). This ultimately calls
CASCache.get_top_level_dir() to obtain a CasBasedDirectory
of an artifact's subdirectory (or subdirectories).

464beaae

by James Ennis at 2019-02-01T10:22:06Z

cli.py: Move artifact ref handling logic to stream

The loading of elements and the handling of artifacts does not belong
in this module. Such logic should be invoked using the Stream API

1960b25e

by James Ennis at 2019-02-01T10:22:06Z

cli.py: Remove _classify_artifacts() method

This method has been moved to Stream, where is it used there exclusively.

5c5807b8

by James Ennis at 2019-02-01T10:22:06Z

cascache.py: Allow CASCache.list_refs() to handle globs

This commit ensures that CASCache.list_refs(), and
ArtifactCache.list_artifacts(), can both handle glob expressions.

98fc37cb

by James Ennis at 2019-02-01T10:22:06Z

_stream.py: _classify_artifacts() should be able to handle globs

_classify_artifacts() no longer filters glob expressions by first
obtaining a list of ALL refs locally cached. We now only obtain refs
specified by the glob _expression_.

Furthermore, the Project.element_path is used to start searching for
globbed elements as opposed to the Project.directory.

Changes:

buildstream/_artifactcache.py

@@ -513,11 +513,14 @@ class ArtifactCache():
+     #
      # List artifacts in this cache in LRU order.
+     #
 +    # Args:
 +    #     glob (str): An option glob _expression_ to be used to list artifacts satisfying the glob
 +    #
      # Returns:
      #     ([str]) - A list of artifact names as generated in LRU order
+     #
 -    def list_artifacts(self):
 -        return self.cas.list_refs()
 +    def list_artifacts(self, *, glob=None):
 +        return self.cas.list_refs(glob=glob)
      # remove():
+     #
@@ -821,6 +824,20 @@ class ArtifactCache():
          self.cas.link_ref(oldref, newref)
 +    # get_artifact_logs():
 +    #
 +    # Get the logs of an existing artifact
 +    #
 +    # Args:
 +    #     ref (str): The ref of the artifact
 +    #
 +    # Returns:
 +    #     logsdir (CasBasedDirectory): A CasBasedDirectory containing the artifact's logs
 +    #
 +    def get_artifact_logs(self, ref):
 +        descend = ["logs"]
 +        return self.cas.get_toplevel_dir(ref, descend)
++
      ################################################
      #               Local Private Methods          #
      ################################################

buildstream/_artifactelement.py

 +#
 +#  Copyright (C) 2019 Bloomberg Finance LP
 +#
 +#  This program is free software; you can redistribute it and/or
 +#  modify it under the terms of the GNU Lesser General Public
 +#  License as published by the Free Software Foundation; either
 +#  version 2 of the License, or (at your option) any later version.
 +#
 +#  This library is distributed in the hope that it will be useful,
 +#  but WITHOUT ANY WARRANTY; without even the implied warranty of
 +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.	 See the GNU
 +#  Lesser General Public License for more details.
 +#
 +#  You should have received a copy of the GNU Lesser General Public
 +#  License along with this library. If not, see <http://www.gnu.org/licenses/>.
 +#
 +#  Authors:
 +#        James Ennis <james ennis codethink co uk>
 +from . import Element
 +from ._exceptions import ArtifactElementError
 +from ._loader.metaelement import MetaElement
++
++
 +# ArtifactElement()
 +#
 +# Object to be used for directly processing an artifact
 +#
 +# Args:
 +#    context (Context): The Context object
 +#    ref (str): The artifact ref
 +#
 +class ArtifactElement(Element):
 +    def __init__(self, context, ref):
 +        self._ref = ref
++
 +        # Ensure that the provided ref matches the form of an artifact
 +        try:
 +            _, element, key = ref.split('/', 2)
 +        except ValueError:
 +            raise ArtifactElementError("Artifact: {} is not of the expected format".format(ref))
 +        if len(key) != 64:
 +            raise ArtifactElementError("Artifact: {} is not of the expected format".format(ref))
++
 +        self._key = key
++
 +        project = context.get_toplevel_project()
 +        meta = MetaElement(project, element)  # NOTE element has no .bst suffix
 +        plugin_conf = None
++
 +        super().__init__(context, project, meta, plugin_conf)
++
 +    # Override Element.get_artifact_name()
 +    def get_artifact_name(self, key=None):
 +        return self._ref
++
 +    # Dummy configure method
 +    def configure(self, node):
 +        pass
++
 +    # Dummy preflight method
 +    def preflight(self):
 +        pass
++
 +    # Override Element._calculate_cache_key
 +    def _calculate_cache_key(self, dependencies=None):
 +        return self._key

buildstream/_cas/cascache.py

@@ -24,6 +24,7 @@ import stat
  import errno
  import uuid
  import contextlib
 +from fnmatch import fnmatch
  import grpc
@@ -32,6 +33,7 @@ from .._protos.buildstream.v2 import buildstream_pb2
  from .. import utils
  from .._exceptions import CASCacheError
 +from ..storage._casbaseddirectory import CasBasedDirectory
  from .casremote import BlobNotFound, _CASBatchRead, _CASBatchUpdate
@@ -472,22 +474,35 @@ class CASCache():
+     #
      # List refs in Least Recently Modified (LRM) order.
+     #
 +    # Args:
 +    #     glob (str) - An optional glob _expression_ to be used to list refs satisfying the glob
 +    #
      # Returns:
      #     (list) - A list of refs in LRM order
+     #
 -    def list_refs(self):
 +    def list_refs(self, *, glob=None):
          # string of: /path/to/repo/refs/heads
          ref_heads = os.path.join(self.casdir, 'refs', 'heads')
 +        path = ref_heads
++
 +        if glob is not None:
 +            globdir = os.path.dirname(glob)
 +            if not any(c in "*?[" for c in globdir):
 +                # path prefix contains no globbing characters so
 +                # append the glob to optimise the os.walk()
 +                path = os.path.join(ref_heads, globdir)
          refs = []
          mtimes = []
 -        for root, _, files in os.walk(ref_heads):
 +        for root, _, files in os.walk(path):
              for filename in files:
                  ref_path = os.path.join(root, filename)
 -                refs.append(os.path.relpath(ref_path, ref_heads))
 -                # Obtain the mtime (the time a file was last modified)
 -                mtimes.append(os.path.getmtime(ref_path))
 +                relative_path = os.path.relpath(ref_path, ref_heads)  # Relative to refs head
 +                if not glob or fnmatch(relative_path, glob):
 +                    refs.append(relative_path)
 +                    # Obtain the mtime (the time a file was last modified)
 +                    mtimes.append(os.path.getmtime(ref_path))
          # NOTE: Sorted will sort from earliest to latest, thus the
          # first ref of this list will be the file modified earliest.
@@ -587,6 +602,22 @@ class CASCache():
          reachable = set()
          self._reachable_refs_dir(reachable, tree, update_mtime=True)
 +    # get_toplevel_dir()
 +    #
 +    # Return a CasBasedDirectory object of the specified sub_directories
 +    #
 +    # Args:
 +    #     ref (str): The artifact ref
 +    #     descend (list): A list of strings of artifact subdirectories
 +    #
 +    # Returns:
 +    #     (CasBasedDirectory): The CasBasedDirectory object
 +    #
 +    def get_toplevel_dir(self, ref, descend):
 +        cache_id = self.resolve_ref(ref, update_mtime=True)
 +        vdir = CasBasedDirectory(self, cache_id).descend(descend)
 +        return vdir
++
      ################################################
      #             Local Private Methods            #
      ################################################

buildstream/_exceptions.py

@@ -344,3 +344,12 @@ class AppError(BstError):
+ #
  class SkipJob(Exception):
      pass
++
++
 +# ArtifactElementError
 +#
 +# Raised when errors are encountered by artifact elements
 +#
 +class ArtifactElementError(BstError):
 +    def __init__(self, message, *, detail=None, reason=None, temporary=False):
 +        super().__init__(message, detail=detail, domain=ErrorDomain.ELEMENT, reason=reason, temporary=True)

buildstream/_frontend/cli.py

  import os
  import sys
  from contextlib import ExitStack
 -from fnmatch import fnmatch
  from functools import partial
  from tempfile import TemporaryDirectory
@@ -895,38 +894,6 @@ def workspace_list(app):
  #############################################################
  #                     Artifact Commands                     #
  #############################################################
 -def _classify_artifacts(names, cas, project_directory):
 -    element_targets = []
 -    artifact_refs = []
 -    element_globs = []
 -    artifact_globs = []
+-
 -    for name in names:
 -        if name.endswith('.bst'):
 -            if any(c in "*?[" for c in name):
 -                element_globs.append(name)
 -            else:
 -                element_targets.append(name)
 -        else:
 -            if any(c in "*?[" for c in name):
 -                artifact_globs.append(name)
 -            else:
 -                artifact_refs.append(name)
+-
 -    if element_globs:
 -        for dirpath, _, filenames in os.walk(project_directory):
 -            for filename in filenames:
 -                element_path = os.path.join(dirpath, filename).lstrip(project_directory).lstrip('/')
 -                if any(fnmatch(element_path, glob) for glob in element_globs):
 -                    element_targets.append(element_path)
+-
 -    if artifact_globs:
 -        artifact_refs.extend(ref for ref in cas.list_refs()
 -                             if any(fnmatch(ref, glob) for glob in artifact_globs))
+-
 -    return element_targets, artifact_refs
+-
+-
  @cli.group(short_help="Manipulate cached artifacts")
  def artifact():
      """Manipulate cached artifacts"""
@@ -1105,53 +1072,30 @@ def artifact_push(app, elements, deps, remote):
  @click.pass_obj
  def artifact_log(app, artifacts):
      """Show logs of all artifacts"""
 -    from .._exceptions import CASError
 -    from .._message import MessageType
 -    from .._pipeline import PipelineSelection
 -    from ..storage._casbaseddirectory import CasBasedDirectory
+-
 -    with ExitStack() as stack:
 -        stack.enter_context(app.initialized())
 -        cache = app.context.artifactcache
 +    # Guess the element if we're in a workspace
 +    if not artifacts:
 +        guessed_target = app.context.guess_element()
 +        if guessed_target:
 +            artifacts = [guessed_target]
 -        elements, artifacts = _classify_artifacts(artifacts, cache.cas,
 -                                                  app.project.directory)
+-
 -        vdirs = []
 -        extractdirs = []
 -        if artifacts:
 -            for ref in artifacts:
 -                try:
 -                    cache_id = cache.cas.resolve_ref(ref, update_mtime=True)
 -                    vdir = CasBasedDirectory(cache.cas, cache_id)
 -                    vdirs.append(vdir)
 -                except CASError as e:
 -                    app._message(MessageType.WARN, "Artifact {} is not cached".format(ref), detail=str(e))
 -                    continue
 -        if elements:
 -            elements = app.stream.load_selection(elements, selection=PipelineSelection.NONE)
 -            for element in elements:
 -                if not element._cached():
 -                    app._message(MessageType.WARN, "Element {} is not cached".format(element))
 -                    continue
 -                ref = cache.get_artifact_fullname(element, element._get_cache_key())
 -                cache_id = cache.cas.resolve_ref(ref, update_mtime=True)
 -                vdir = CasBasedDirectory(cache.cas, cache_id)
 -                vdirs.append(vdir)
+-
 -        for vdir in vdirs:
 -            # NOTE: If reading the logs feels unresponsive, here would be a good place to provide progress information.
 -            logsdir = vdir.descend(["logs"])
 -            td = stack.enter_context(TemporaryDirectory())
 -            logsdir.export_files(td, can_link=True)
 -            extractdirs.append(td)
+-
 -        for extractdir in extractdirs:
 -            for log in (os.path.join(extractdir, log) for log in os.listdir(extractdir)):
 -                # NOTE: Should click gain the ability to pass files to the pager this can be optimised.
 -                with open(log) as f:
 -                    data = f.read()
 -                    click.echo_via_pager(data)
 +    with app.initialized():
 +        logsdirs = app.stream.artifact_log(artifacts)
++
 +        with ExitStack() as stack:
 +            extractdirs = []
 +            for logsdir in logsdirs:
 +                # NOTE: If reading the logs feels unresponsive, here would be a good place
 +                # to provide progress information.
 +                td = stack.enter_context(TemporaryDirectory())
 +                logsdir.export_files(td, can_link=True)
 +                extractdirs.append(td)
++
 +            for extractdir in extractdirs:
 +                for log in (os.path.join(extractdir, log) for log in os.listdir(extractdir)):
 +                    # NOTE: Should click gain the ability to pass files to the pager this can be optimised.
 +                    with open(log) as f:
 +                        data = f.read()
 +                        click.echo_via_pager(data)
  ##################################################################

buildstream/_loader/metaelement.py

@@ -38,20 +38,20 @@ class MetaElement():
      #    sandbox: Configuration specific to the sandbox environment
      #    first_pass: The element is to be loaded with first pass configuration (junction)
+     #
 -    def __init__(self, project, name, kind, provenance, sources, config,
 -                 variables, environment, env_nocache, public, sandbox,
 -                 first_pass):
 +    def __init__(self, project, name, kind=None, provenance=None, sources=None, config=None,
 +                 variables=None, environment=None, env_nocache=None, public=None,
 +                 sandbox=None, first_pass=False):
          self.project = project
          self.name = name
          self.kind = kind
          self.provenance = provenance
          self.sources = sources
 -        self.config = config
 -        self.variables = variables
 -        self.environment = environment
 -        self.env_nocache = env_nocache
 -        self.public = public
 -        self.sandbox = sandbox
 +        self.config = config or {}
 +        self.variables = variables or {}
 +        self.environment = environment or {}
 +        self.env_nocache = env_nocache or []
 +        self.public = public or {}
 +        self.sandbox = sandbox or {}
          self.build_dependencies = []
          self.dependencies = []
          self.first_pass = first_pass

buildstream/_project.py

@@ -26,6 +26,7 @@ from . import utils
  from . import _cachekey
  from . import _site
  from . import _yaml
 +from ._artifactelement import ArtifactElement
  from ._profile import Topics, profile_start, profile_end
  from ._exceptions import LoadError, LoadErrorReason
  from ._options import OptionPool
@@ -255,6 +256,19 @@ class Project():
          else:
              return self.config.element_factory.create(self._context, self, meta)
 +    # create_artifact_element()
 +    #
 +    # Instantiate and return an ArtifactElement
 +    #
 +    # Args:
 +    #    ref (str): A string of the artifact ref
 +    #
 +    # Returns:
 +    #    (ArtifactElement): A newly created ArtifactElement object of the appropriate kind
 +    #
 +    def create_artifact_element(self, ref):
 +        return ArtifactElement(self._context, ref)
++
      # create_source()
+     #
      # Instantiate and return a Source

buildstream/_stream.py

@@ -27,8 +27,9 @@ import shutil
  import tarfile
  import tempfile
  from contextlib import contextmanager, suppress
 +from fnmatch import fnmatch
 -from ._exceptions import StreamError, ImplError, BstError, set_last_task_error
 +from ._exceptions import StreamError, ImplError, BstError, ArtifactElementError, set_last_task_error
  from ._message import Message, MessageType
  from ._scheduler import Scheduler, SchedStatus, TrackQueue, FetchQueue, BuildQueue, PullQueue, PushQueue
  from ._pipeline import Pipeline, PipelineSelection
@@ -108,19 +109,21 @@ class Stream():
      def load_selection(self, targets, *,
                         selection=PipelineSelection.NONE,
                         except_targets=(),
 -                       use_artifact_config=False):
 +                       use_artifact_config=False,
 +                       load_refs=False):
          profile_start(Topics.LOAD_SELECTION, "_".join(t.replace(os.sep, '-') for t in targets))
 -        elements, _ = self._load(targets, (),
 -                                 selection=selection,
 -                                 except_targets=except_targets,
 -                                 fetch_subprojects=False,
 -                                 use_artifact_config=use_artifact_config)
 +        target_objects, _ = self._load(targets, (),
 +                                       selection=selection,
 +                                       except_targets=except_targets,
 +                                       fetch_subprojects=False,
 +                                       use_artifact_config=use_artifact_config,
 +                                       load_refs=load_refs)
          profile_end(Topics.LOAD_SELECTION, "_".join(t.replace(os.sep, '-') for t in targets))
 -        return elements
 +        return target_objects
      # shell()
+     #
@@ -481,6 +484,31 @@ class Stream():
              raise StreamError("Error while staging dependencies into a sandbox"
                                ": '{}'".format(e), detail=e.detail, reason=e.reason) from e
 +    # artifact_log()
 +    #
 +    # Show the full log of an artifact
 +    #
 +    # Args:
 +    #    targets (str): Targets to view the logs of
 +    #
 +    # Returns:
 +    #    logsdir (list): A list of CasBasedDirectory objects containing artifact logs
 +    #
 +    def artifact_log(self, targets):
 +        # Return list of Element and/or ArtifactElement objects
 +        target_objects = self.load_selection(targets, selection=PipelineSelection.NONE, load_refs=True)
++
 +        logsdirs = []
 +        for obj in target_objects:
 +            ref = obj.get_artifact_name()
 +            if not obj._cached():
 +                self._message(MessageType.WARN, "{} is not cached".format(ref))
 +                continue
++
 +            logsdirs.append(self._artifacts.get_artifact_logs(ref))
++
 +        return logsdirs
++
      # source_checkout()
+     #
      # Checkout sources of the target element to the specified location
@@ -912,25 +940,35 @@ class Stream():
                use_artifact_config=False,
                artifact_remote_url=None,
                fetch_subprojects=False,
 -              dynamic_plan=False):
 +              dynamic_plan=False,
 +              load_refs=False):
++
 +        # Classify element and artifact strings
 +        target_elements, target_artifacts = self._classify_artifacts(targets)
++
 +        if target_artifacts and not load_refs:
 +            detail = ''.join(target_artifacts)
 +            raise ArtifactElementError("Cannot perform this operation with artifact refs:", detail=detail)
          # Load rewritable if we have any tracking selection to make
          rewritable = False
          if track_targets:
              rewritable = True
 -        # Load all targets
 +        # Load all target elements
          elements, except_elements, track_elements, track_except_elements = \
 -            self._pipeline.load([targets, except_targets, track_targets, track_except_targets],
 +            self._pipeline.load([target_elements, except_targets, track_targets, track_except_targets],
                                  rewritable=rewritable,
                                  fetch_subprojects=fetch_subprojects)
 +        artifacts = self._load_refs(target_artifacts)
++
          # Optionally filter out junction elements
          if ignore_junction_targets:
              elements = [e for e in elements if e.get_kind() != 'junction']
          # Hold on to the targets
 -        self.targets = elements
 +        self.targets = elements + artifacts
          # Here we should raise an error if the track_elements targets
          # are not dependencies of the primary targets, this is not
@@ -987,9 +1025,9 @@ class Stream():
          # Now move on to loading primary selection.
+         #
 -        self._pipeline.resolve_elements(elements)
 -        selected = self._pipeline.get_selection(elements, selection, silent=False)
 -        selected = self._pipeline.except_elements(elements,
 +        self._pipeline.resolve_elements(self.targets)
 +        selected = self._pipeline.get_selection(self.targets, selection, silent=False)
 +        selected = self._pipeline.except_elements(self.targets,
                                                    selected,
                                                    except_elements)
@@ -1013,6 +1051,20 @@ class Stream():
          return selected, track_selected
 +    # _load_refs()
 +    #
 +    #  Create and resolve ArtifactElement objects
 +    #
 +    def _load_refs(self, refs):
 +        artifact_elements = []
 +        for ref in refs:
 +            artifact_element = self._project.create_artifact_element(ref)
 +            artifact_elements.append(artifact_element)
++
 +        self._pipeline.resolve_elements(artifact_elements)
++
 +        return artifact_elements
++
      # _message()
+     #
      # Local message propagator
@@ -1321,3 +1373,59 @@ class Stream():
                  required_list.append(element)
          return required_list
++
 +    # _classify_artifacts()
 +    #
 +    # Split up a list of targets into element names and artifact refs
 +    #
 +    # Args:
 +    #    targets (list): A list of targets
 +    #
 +    # Returns:
 +    #    (list): element names present in the targets
 +    #    (list): artifact refs present in the targets
 +    #
 +    def _classify_artifacts(self, targets):
 +        element_targets = []
 +        artifact_refs = []
 +        element_globs = []
 +        artifact_globs = []
++
 +        for target in targets:
 +            if target.endswith('.bst'):
 +                if any(c in "*?[" for c in target):
 +                    element_globs.append(target)
 +                else:
 +                    element_targets.append(target)
 +            else:
 +                if any(c in "*?[" for c in target):
 +                    artifact_globs.append(target)
 +                else:
 +                    try:
 +                        ref = target.split('/', 2)
 +                        key = ref[2]
 +                    except IndexError:
 +                        element_targets.append(target)
 +                        continue
 +                    if not len(key) == 64:
 +                        element_targets.append(target)
 +                        continue
 +                    artifact_refs.append(target)
++
 +        if element_globs:
 +            for dirpath, _, filenames in os.walk(self._project.element_path):
 +                for filename in filenames:
 +                    element_path = os.path.join(dirpath, filename)
 +                    length = len(self._project.element_path) + 1
 +                    element_path = element_path[length:]  # Strip out the element_path
++
 +                    if any(fnmatch(element_path, glob) for glob in element_globs):
 +                        element_targets.append(element_path)
++
 +        if artifact_globs:
 +            for glob in artifact_globs:
 +                artifact_refs = self._artifacts.list_artifacts(glob=glob)
 +            if not artifact_refs:
 +                self._message(MessageType.WARN, "No artifacts found for globs: {}".format(', '.join(artifact_globs)))
++
 +        return element_targets, artifact_refs

buildstream/element.py

@@ -1150,7 +1150,7 @@ class Element(Plugin):
                      e.name for e in self.dependencies(Scope.BUILD, recurse=False)
+                 ]
 -            self.__weak_cache_key = self.__calculate_cache_key(dependencies)
 +            self.__weak_cache_key = self._calculate_cache_key(dependencies)
              if self.__weak_cache_key is None:
                  # Weak cache key could not be calculated yet
@@ -1179,8 +1179,7 @@ class Element(Plugin):
              dependencies = [
                  e.__strict_cache_key for e in self.dependencies(Scope.BUILD)
+             ]
 -            self.__strict_cache_key = self.__calculate_cache_key(dependencies)
+-
 +            self.__strict_cache_key = self._calculate_cache_key(dependencies)
              if self.__strict_cache_key is None:
                  # Strict cache key could not be calculated yet
                  return
@@ -1222,7 +1221,7 @@ class Element(Plugin):
                  dependencies = [
                      e._get_cache_key() for e in self.dependencies(Scope.BUILD)
+                 ]
 -                self.__cache_key = self.__calculate_cache_key(dependencies)
 +                self.__cache_key = self._calculate_cache_key(dependencies)
              if self.__cache_key is None:
                  # Strong cache key could not be calculated yet
@@ -2064,41 +2063,7 @@ class Element(Plugin):
                  source._fetch(previous_sources)
              previous_sources.append(source)
 -    #############################################################
 -    #                   Private Local Methods                   #
 -    #############################################################
+-
 -    # __update_source_state()
 -    #
 -    # Updates source consistency state
 -    #
 -    def __update_source_state(self):
+-
 -        # Cannot resolve source state until tracked
 -        if self.__tracking_scheduled:
 -            return
+-
 -        self.__consistency = Consistency.CACHED
 -        workspace = self._get_workspace()
+-
 -        # Special case for workspaces
 -        if workspace:
+-
 -            # A workspace is considered inconsistent in the case
 -            # that its directory went missing
 -            #
 -            fullpath = workspace.get_absolute_path()
 -            if not os.path.exists(fullpath):
 -                self.__consistency = Consistency.INCONSISTENT
 -        else:
+-
 -            # Determine overall consistency of the element
 -            for source in self.__sources:
 -                source._update_state()
 -                source_consistency = source._get_consistency()
 -                self.__consistency = min(self.__consistency, source_consistency)
+-
 -    # __calculate_cache_key():
 +    # _calculate_cache_key():
+     #
      # Calculates the cache key
+     #
@@ -2107,7 +2072,7 @@ class Element(Plugin):
+     #
      # None is returned if information for the cache key is missing.
+     #
 -    def __calculate_cache_key(self, dependencies):
 +    def _calculate_cache_key(self, dependencies):
          # No cache keys for dependencies which have no cache keys
          if None in dependencies:
              return None
@@ -2146,6 +2111,40 @@ class Element(Plugin):
          return _cachekey.generate_key(cache_key_dict)
 +    #############################################################
 +    #                   Private Local Methods                   #
 +    #############################################################
++
 +    # __update_source_state()
 +    #
 +    # Updates source consistency state
 +    #
 +    def __update_source_state(self):
++
 +        # Cannot resolve source state until tracked
 +        if self.__tracking_scheduled:
 +            return
++
 +        self.__consistency = Consistency.CACHED
 +        workspace = self._get_workspace()
++
 +        # Special case for workspaces
 +        if workspace:
++
 +            # A workspace is considered inconsistent in the case
 +            # that its directory went missing
 +            #
 +            fullpath = workspace.get_absolute_path()
 +            if not os.path.exists(fullpath):
 +                self.__consistency = Consistency.INCONSISTENT
 +        else:
++
 +            # Determine overall consistency of the element
 +            for source in self.__sources:
 +                source._update_state()
 +                source_consistency = source._get_consistency()
 +                self.__consistency = min(self.__consistency, source_consistency)
++
      # __can_build_incrementally()
+     #
      # Check if the element can be built incrementally, this
@@ -2329,6 +2328,8 @@ class Element(Plugin):
          defaults['public'] = element_public
      def __init_defaults(self, plugin_conf):
 +        if plugin_conf is None:
 +            return
          # Defaults are loaded once per class and then reused
+         #

[Notes] [Git][BuildStream/buildstream][jennis/refactor_artifact_log] 13 commits: element.py: Make calculate_cache_key() API private

James Ennis pushed to branch jennis/refactor_artifact_log at BuildStream / buildstream

Commits:

9 changed files:

Changes: