_cas: avoid blocking calls to long grpc methods

[abderrahim]

Long-running synchronous grpc calls block the thread where they are
called, which can cause the scheduler to be unable to terminate the
job.

This commit replaces them with usage of the "asynchronous" future
API. The result() method of a grpc future will not block the thread
while it's waiting, and thus allows terminating the job without
waiting for the blocking call to return. This covers the LocalCAS
methods {Fetch,Upload}{Tree,MissingBlobs}.

As an added benefit, we try to cancel the grpc calls when
terminated. This depends on the server side (buildbox-casd) promptly
cancelling the remote calls, which it doesn't consistently do at the
moment. But that can be fixed independently.

Fixes #2157

[juergbi]

Can't we use batch_response_future.result(timeout=1) to avoid an extra delay of up to 1 second once the call is done?

[abderrahim]

I'll take a look.

[abderrahim]

pushed a change to use this. I can't say I find it more elegant. Feel free to suggest improvements.

(I'm wondering whether it makes sense to have this loop as a utility function)

[juergbi]

The updated version may not be more elegant but it should be slightly more efficient.

I think a helper function would make sense. If we don't need it beyond these two gRPC calls, I wouldn't consider it essential but if it's straight forward, why not?

[juergbi]

Looking at the gRPC Python code, we might not even need a timeout. The internal wait function seems to make sure that the thread is not blocked for longer than 0.1s. See https://github.com/grpc/grpc/blob/f167d6a879e5b3223c99170663b2cc3e791ac6ad/src/python/grpcio/grpc/_common.py#L119

I haven't verified this but we may be able to call .result() without a loop or timeout.

[abderrahim]

We could also do the same to the FetchTree calls in cascache.py, but these ones looked like the more urgent.

[abderrahim]

I haven't verified this but we may be able to call .result() without a loop or timeout.

Seems to be the case indeed. I like this much better :-)

[juergbi]

We could also do the same to the FetchTree calls in cascache.py

Yes, I think we should apply this change also to FetchTree in fetch_directory() and FindMissingBlobs in missing_blobs(). This should cover all gRPC calls involving ordinary remotes (i.e., everything except storage-service). To also cover storage-service, we might have to apply it to practically all gRPC calls.

[abderrahim]

Updated to cover all instances of {Fetch,Upload}{Tree,MissingBlobs}