Re: [BuildStream] buildbox-casd timeout issues during BuildStream startup

[Jürg Billeter <j bitron ch>]

Hi,

On Wed, 2020-03-25 at 19:28 +0000, Chandan Singh wrote:
> > An additional option that came to mind that is relatively cheap and
> > reliable (enough) would be to write a little metadata file at the root
> > of the store. casd could write this periodically (in case of crash) or
> > on shutdown. It would contain basically the current size of the CAS
> > (or whatever casd has in memory). Then on startup casd just reads this
> > file and trusts that that is the current size without having to scan
> > the whole tree. We would probably have to store the atimes in that
> > file too (which casd needs in memory), or that could be lazily
> > calculated in the LRU thread. This would make startup instantaneous
> > and both reads and writes would be permitted.

We don't currently have the object timestamps in memory, storing the
disk usage would suffice.

> > This only obvious downside to this approach is that it doesn't protect
> > against manual modification of the casd directory while casd is down
> > such that the cache and metadata gets out of sync.

I'm more concerned about crashes/SIGKILL, and missing protection
against running multiple casd instances with the same directory. I'd
suggest deleting the file on the first write operation and write it
(atomically) as part of graceful shutdown. This should avoid issues
with unexpected buildbox-casd termination.

> > Alternatively, we could attempt to speed up the operations to reduce
> > the amount of time spent in read-only mode: it doesn't seem that AIO
> > has support for stat but there is support in the very recent io_uring
> > for async statx which could be leveraged in the future. For now simply
> > multithreading this very parallelisable operation should provide
> > significant speedups.
>
> I think on slower filesystems this approach can still be quite time
> taking, so I'm leaning towards the first option you proposed above. (I
> don't have numbers to back that up just yet). Nonetheless, I guess the
> metadata update script (if we choose to write it) can leverage this
> approach as well.

Yes, the bottleneck varies. With a hot dentry cache, the bottleneck in
Linux is the stat() syscall (especially if you have a Meltdown-affected 
Intel CPU) and parallelization should help a lot, assuming you have CPU
capacity. With a cold dentry cache, the bottleneck is disk/network I/O
or the filesystem and you might not gain much with parallelization.

Cheers,
Jürg