Re: Downloading from generated mirror

[Jonathan Maw <jonathan maw codethink co uk>]

On 2018-04-30 12:56, Valentin David wrote:
> I see two specific things required for this to work. And I would to
> make sure I am correct. So please tell me if I am wrong.

Hi Valentin,

Can you confirm that my understanding of these issues is correct (when I
paraphrase them below)

>  1. Path
>  -------
>
> Path from mirror root are generated based on original URL. But really
> the plugins have some freedom about it. This path should be generated
> by the plugin in order to generate the correct URL.
>
> Since handling of URL for #328 is done as higher level, I suppose
> plugins should provide a method returning the generated path.

Currently, by convention we cache sources to 
{cache_dir}/sources/{source_type}/{URL_directory_name}

This is aided by Source.get_mirror_directory(), and 
utils.url_directory_name()

In your current MR, you have similar functionality with 
Source.find_mirror_directory()
as a helper, and a given source's _get_mirror_file() can be relied upon 
to return the
correct directory. This is close to what we want, but the numbered 
component that gets
appended by find_mirror_directory() makes it less useful for fetching.

An important note for fetching, which goes beyond the usual logic of 
finding the correct
URL by replacing the alias with a different URI is that the name of the 
repository after
the colon is also different (e.g. github:foo gets transformed into 
either github_foo or
http___github_com_foo, I'm not sure which).

>  2. Multiple directories
>  -----------------------
>
> For some plugins, in #330, there is an extra component in the path. It
> is a number incrementing from 0. There are some plugins that might get
> new incompatible repositories or files for a same URL. And when we
> detect this, we create a new directory.
>
>  - ostree and git should not suffer of this problem, because we can
> just compose repositories and collisions are unlikely.
>  - zip and tar need it when the file is changed.
>  - bzr needs it, when we detect that remote repository does not contain
> at least all the commits we have locally.
>
> The loop that tries mirrors should retry while counting on the last
> path component for those plugins requiring it.
>
> There must be now two exceptions raised from source plugins:
>  - one that says the URL works but the reference is not available,
>  - another one for any other error.
> The loop should count until it find the second error.

Here, I'd be expecting sources to exist at URLs like:

http:///mirror.com/github_foo/0
http:///mirror.con/github_foo/1

and should be trying to fetch with sources, expecting a RefNotFoundError 
when fetching
from http:///mirror.com/github_foo/0, using that as an instruction to 
try
http:///mirror.con/github_foo/1, and if that doesn't work, trying with
http:///mirror.con/github_foo/1 (which will return a SourceError instead 
of a
RefNotFoundError, which is a clear indicator to stop trying.

It should be clear by the configuration whether I should be fetching 
like it's a
buildstream mirror or a non-buildstream mirror.

Would I expect a value set by the source to indicate whether sources are 
mirrored with
numbers appended, or is it safe to assume a number is appended, even if 
multiple
repositories to store all the incompatible refs is unlikely to ever 
happen (so there'd
be "github_foo/0", but never "github_foo/1"?


Thanks for your time.

Best regards,

Jonathan