This patch is unfortunately large, but it seemed necessary to make all
these changes at once to transition away from having an HTTP
ChunkStore that could allow for invalid state in the DB. Now, we have
a RemoteDataStoreClient that allows for reading and writing of Values,
and performs validation on the server side before persisting chunks.
The semantics of DataStore are that written values can be read back
out immediately, but are not guaranteed to be persistent until after
Commit() The semantics are now that Put() blocks until the Chunk is
persisted, and the new PutMany() can be used to write a number of
Chunks all at once.
From a command-line tool point of view, -h and -h-auth still work as
expected.
With this patch, DynamoStore will now hold off writing anything to
the backend until it's got a full batch or someone calls UpdateRoot().
When it's time to write, DynamoStore will now fire off a new goroutine
to build the request (including compressing chunks), send it, and wait
for a response. It will keep up to dynamoWriteConcurrency concurrent
batch writes in flight.
Includes statKeeper, which provides a way for concurrent goroutines to
keep stats like write count, bytes written, etc. Also includes a refactor
to make unwrittenPutCache a separate type that both HTTPStore and
DynamoStore use instead of using copypasted code.
Chris Masone