LocalDatabase generally uses a BatchStoreAdaptor, which is a kinda
dumb wrapper around ChunkStore. During a Pull(), though, this would
cause Chunks to be Put in a top-down fashion, meaning that Chunks
wound up in the backing store _before_ other Chunks that they
reference. This means that LocalDatabases were transiently invalid,
and that cancelling an in-progress pull could lead to an invalid DB.
Now, calling validatingBatchStore() on a LocalDatabase returns a
BatchStore that uses the same strategy as RemoteDatabaseClient,
caching chunks as the come in and putting them into the backing store
bottom-up when Flush() is called.
Fixes#1915
This reverts commit ba909929c2, but
tweaks the size of the memory cache so that, hopefully, Travis
doesn't barf. Maybe once travis is dead we can size up.
Fixes#1817
The basic approach here is to take the max of the heights of the
source and sink queues, then grab all the refs of that height from
both and sort them into three sets: refs in the source, refs in the
sink, and refs in both. These are then processed in parallel and the
reachable refs are all added to the appropriate queue. Repeat as long
as stuff still shows up in the source queue.
Fixes#1564
The bigger change here is having chunks.DeserializeToChan take a
channel of *Chunk, instead of Chunk. It saves a couple of seconds
when committing the sfcrime dataset.
Also, get rid of a d.Chk.Equal in favor of a d.Chk.True and increase
the batch size that ValidatingBatchSink uses to reduce stalls due to
putting into a ChunkStore.
In datas/put_cache.go, we need to serialize one chunk at a time. With
the old API, this entailed creating a serializer (which creates some
channels and a goroutine) only to use it for one Chunk.
Instead, just expose Serialize() and use it directly.
We determined that gzipping the entire writeValue payload at Commit-
time was extremely time-expensive. After doing some experiments, it
seems that a couple of changes make Commit go a lot faster:
1) Use snappy instead of gzip
2) Compress chunks individually instead of delaying until the whole
stream is ready.
This does result in more data needing to be sent over the network.
However, we get to the point of being able to send data _much_ sooner.
Since Chunks are compressed individually, the orderedChunkCache code
now compresses them before storing them, also saving disk space.
Rafael Weinstein