Plan of action for source mirroring (client-side)

[Jonathan Maw <jonathan maw codethink co uk>]

Hi,

This E-mail is a summary of what's planned for the source mirroring 
work. I feel it's important to have because Valentin and I are working 
on this in parallel and need to be sure we're working with the same 
assumptions.

The main point of synchronisation is ensuring that the place I look for 
mirrors generated by buildstream is the same place that Valentin stores 
mirrors.

To this end, I will do the following:
* Create a new class of error, RefNotFoundError, which subclasses from 
SourceError.
* When fetching from buildstream-generated mirrors, it will have the 
format {URL}/sources/{source_kind}/{URL_directory_name}/{mirror_number}
  where {URL} is the URL of the buildstream mirror generated via config,
  {source_kind} is the kind of the source (as specified in the yaml's 
"kind" field),
  {URL_directory_name} is the URL specified for the source, mangled into 
a valid name for a directory (e.g. git://github.com becomes 
git___github_com),
  {mirror_number} as a number starting from 0, and incrementing every 
time fetching/tracking returns a RefNotFoundError instead of a general 
SourceError.
* To enforce part of this functionality / make it easier to change 
later, I will add a helper method to Source, e.g.
  Source.get_normalised_mirror_path(upstream_url, *, prefix="", 
suffix="")

Please let me know if there are any ambiguities / things I've forgotten 
to discuss.

Thanks,

Jonathan