In httpBatchStore.Get() and Has(), I add 1 to a WaitGroup, so that
Flush() and Close() can wait for all outstanding requests to complete.
As each Get or Has is processed, it's added to a ReadBatch. Once the
batch is processed, the WaitGroup count is decreased by the size of
the ReadBatch... but ReadBatch is a map! So if the same object is
passed to Get or Has twice, it will (by design) only show up in the
ReadBatch once. That means that we'd add 2 to the WaitGroup, but only
remove 1!
This will lead to waiting forever on Flush() or Close(), and is no
good.
One side effect of the way that Go's flag library works is that any
flag that's defined at the global scope by any library that's built
into your binary shows up in your help -- whether you honor it or
not. Arguably, including the library and calling its entry points
should be tantamount to honoring the flags. This change makes the
profiling flags behave that way -- there's one function that you call
to turn on whichever kinds of profiling the user specifies on the
command line. It returns an object with a Stop() method that you call
to shut down profiling and flush data, like so:
defer profile.MaybeStartProfile().Stop()
The bigger change here is having chunks.DeserializeToChan take a
channel of *Chunk, instead of Chunk. It saves a couple of seconds
when committing the sfcrime dataset.
Also, get rid of a d.Chk.Equal in favor of a d.Chk.True and increase
the batch size that ValidatingBatchSink uses to reduce stalls due to
putting into a ChunkStore.
In datas/put_cache.go, we need to serialize one chunk at a time. With
the old API, this entailed creating a serializer (which creates some
channels and a goroutine) only to use it for one Chunk.
Instead, just expose Serialize() and use it directly.
We determined that gzipping the entire writeValue payload at Commit-
time was extremely time-expensive. After doing some experiments, it
seems that a couple of changes make Commit go a lot faster:
1) Use snappy instead of gzip
2) Compress chunks individually instead of delaying until the whole
stream is ready.
This does result in more data needing to be sent over the network.
However, we get to the point of being able to send data _much_ sooner.
Since Chunks are compressed individually, the orderedChunkCache code
now compresses them before storing them, also saving disk space.
At some point, the Go ValueStore code started caching the hash of a
Chunk as a validation 'hint' for the Chunk itself. JS never did
this. This must have addressed some edge case back when it was fatal
for validation code to run across a Chunk it hadn't seen and didn't
have a hint for. The downside is that doing this can cause us to send
a hint for every novel Chunk present in a writeValue payload in the
worst case.
Since I can't remember the edge case, and that edge case will no
longer be fatal anyway, removing this to avoid the (potentially
terrible) downside makes sense.
Change Dataset.Pull to use a single algorithm to pull data from a
source to a sink, regardless of which (if any) is local. The basic
algorithm is described in the first section of pulling.md. This
implementation is equivalent but phrased a bit differently. The
algorithm actually used is described in the second section of
pulling.md
The main changes:
- datas.Pull(), which implements the new pulling algorithm
- RefHeap, a priority queue that sorts types.Ref by ref-height and
then by ref.TargetHash()
- Add has() to both Database implementations. Cache has() checks.
- Switch Dataset to use new datas.Pull(). Currently not concurrent.
Toward #1568
Mostly, prune reachableChunks
Chris Masone