At some point, the Go ValueStore code started caching the hash of a
Chunk as a validation 'hint' for the Chunk itself. JS never did
this. This must have addressed some edge case back when it was fatal
for validation code to run across a Chunk it hadn't seen and didn't
have a hint for. The downside is that doing this can cause us to send
a hint for every novel Chunk present in a writeValue payload in the
worst case.
Since I can't remember the edge case, and that edge case will no
longer be fatal anyway, removing this to avoid the (potentially
terrible) downside makes sense.
Change Dataset.Pull to use a single algorithm to pull data from a
source to a sink, regardless of which (if any) is local. The basic
algorithm is described in the first section of pulling.md. This
implementation is equivalent but phrased a bit differently. The
algorithm actually used is described in the second section of
pulling.md
The main changes:
- datas.Pull(), which implements the new pulling algorithm
- RefHeap, a priority queue that sorts types.Ref by ref-height and
then by ref.TargetHash()
- Add has() to both Database implementations. Cache has() checks.
- Switch Dataset to use new datas.Pull(). Currently not concurrent.
Toward #1568
Mostly, prune reachableChunks
Mike Gray