GitLab CI runners outage

[Sam Thursfield <sam thursfield codethink co uk>]

Hi,

Codethink provides some GitLab CI runners which are used by the Baserock 
and BuildStream projects.

These recently broke; the symptom was GitLab CI jobs failing with this 
error[1]:

    Running with gitlab-runner 10.1.0 (c1ecf97f)
      on baserock-manager-runner2 (f45ec36a)
    ERROR: Preparation failed: exit status 1
    Will be retried in 3s ...
    ERROR: Preparation failed: exit status 1
    Will be retried in 3s ...
    ERROR: Preparation failed: exit status 1
    Will be retried in 3s ...
    ERROR: Job failed (system failure): exit status 1

I diagnosed the issue by running `journalctl --follow` in the manager 
VM, then retrying a job. Amongst the noise was this line:

    Dec 14 12:01:06 ubuntu-1gb-lon1-01 gitlab-runner[1478]: 
time="2017-12-14T12:01:06Z" level=error msg="Error creating machine: 
Error in driver during machine creation: POST 
https://api.digitalocean.com/v2/droplets: 422 You specified an invalid 
image for Droplet creation." driver=digitalocean 
name=runner-f45ec36a-machine-1513252861-90d58fd0-baserock operation=create

The issue is that we were trying to boot the machine with the 
fedora-25-x64 image, but Fedora 25 recently hit end-of-life so 
DigitalOcean have presumably deleted it.

The fix is to edit /etc/gitlab-runner/config.toml and update the 
runners.machine.MachineOptions setting to boot the fedora-27-x64 image 
instead -- our GitLab CI pipelines seem to be working again now.

Presumably we will need to do the same thing again in just under a 
year's time :-)

Sam

[1]. such as https://gitlab.com/BuildStream/buildstream/-/jobs/44538627

--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575