We determined that gzipping the entire writeValue payload at Commit-
time was extremely time-expensive. After doing some experiments, it
seems that a couple of changes make Commit go a lot faster:
1) Use snappy instead of gzip
2) Compress chunks individually instead of delaying until the whole
stream is ready.
This does result in more data needing to be sent over the network.
However, we get to the point of being able to send data _much_ sooner.
Since Chunks are compressed individually, the orderedChunkCache code
now compresses them before storing them, also saving disk space.
At some point, the Go ValueStore code started caching the hash of a
Chunk as a validation 'hint' for the Chunk itself. JS never did
this. This must have addressed some edge case back when it was fatal
for validation code to run across a Chunk it hadn't seen and didn't
have a hint for. The downside is that doing this can cause us to send
a hint for every novel Chunk present in a writeValue payload in the
worst case.
Since I can't remember the edge case, and that edge case will no
longer be fatal anyway, removing this to avoid the (potentially
terrible) downside makes sense.
Change Dataset.Pull to use a single algorithm to pull data from a
source to a sink, regardless of which (if any) is local. The basic
algorithm is described in the first section of pulling.md. This
implementation is equivalent but phrased a bit differently. The
algorithm actually used is described in the second section of
pulling.md
The main changes:
- datas.Pull(), which implements the new pulling algorithm
- RefHeap, a priority queue that sorts types.Ref by ref-height and
then by ref.TargetHash()
- Add has() to both Database implementations. Cache has() checks.
- Switch Dataset to use new datas.Pull(). Currently not concurrent.
Toward #1568
Mostly, prune reachableChunks
Rafael Weinstein