We want to better pipeline I/O when pulling from a remote, and moving a
streaming interface where storage can see more of the addresses that are needed
at once will allow us to achieve it.
For now, we implement the streaming interface just by calling the existing
batch get interface.
Use a buffered channel for pending table file writes. Make the upload
management thread more transparently responsible for just accumulating manifest
updates and calling AddTableFilesToManifest.
The PullChunkTracker is an optimization which can concurrently call HasMany on
the destination database while chunks are being pulled from the source
database.
A long-standing bug in the remotesapi which the sql-server exposes could cause
a `dolt clone` to fail when running against a database which had been garbage
collected. This change fixes the bug in the server. It also patches the client
behavior so that it will tolerate responses from older versions of dolt.
Before this fix, |dolt clone| against a sql-server remote can fail with a
confusing error message if the sql-server has a chunk journal. Adding the chunk
journal to our destination ChunkStore causes us to update the root hash, which
the Clone code was not expecting.
This change updates the Clone code to look for the possible update to the
destination's root hash and to not try to set the root hash on the ChunkStore
if it has already been done by the code handling the chunk journal.
pull.Clone uses the TableFileStore interface to transit whole table files
without needing to follow chunk references or do any reconciliation with the
destination database regarding what it already has. It is much faster against
every time of remote.
Adds a paranoid mode where we noisely detect unclosed table files. The mode can
be enabled by setting an environment variable.
Fixes some unit tests, including all of go/store/... to run cleanly under the
paranoid mode.
Changes the quota interface to:
* Release |sz int| bytes instead of requiring a []byte with the correct length
to show up.
* Work with |int| instead of |uint64|, since MaxUint64 is never allocatable and
MaxInt32+z is only allocatable on 64-bit platforms.
* Not return an error on Release(). Implementations should not fail to release
quota.
