Aaron Son
6f646a7f01
go/store/datas/pull/clone.go: Fix to clone to interact with sql-server remotes better.
...
Before this fix, |dolt clone| against a sql-server remote can fail with a
confusing error message if the sql-server has a chunk journal. Adding the chunk
journal to our destination ChunkStore causes us to update the root hash, which
the Clone code was not expecting.
This change updates the Clone code to look for the possible update to the
destination's root hash and to not try to set the root hash on the ChunkStore
if it has already been done by the code handling the chunk journal.
2023-05-09 15:25:07 -07:00
Aaron Son
6edc6fd54a
go: env/actions: remotes.go: In SyncRoots, if we are syncing to an empty destination repository, use pull.Clone instead of pull.Pull.
...
pull.Clone uses the TableFileStore interface to transit whole table files
without needing to follow chunk references or do any reconciliation with the
destination database regarding what it already has. It is much faster against
every time of remote.
2023-05-02 17:12:03 -07:00
Aaron Son
7406c4658a
go/store/nbs: Fix some quota leaks in conjoin, GC.
...
Adds a paranoid mode where we noisely detect unclosed table files. The mode can
be enabled by setting an environment variable.
Fixes some unit tests, including all of go/store/... to run cleanly under the
paranoid mode.
Changes the quota interface to:
* Release |sz int| bytes instead of requiring a []byte with the correct length
to show up.
* Work with |int| instead of |uint64|, since MaxUint64 is never allocatable and
MaxInt32+z is only allocatable on 64-bit platforms.
* Not return an error on Release(). Implementations should not fail to release
quota.
2023-02-16 16:01:20 -08:00
AndyA
c113089de2
Merge pull request #5282 from dolthub/andy/batch-stack
...
go/store/datas/pull: try to avoid unbounded growth of outstanding abs…
2023-02-02 13:00:12 -08:00
Aaron Son
f41cd010aa
Merge pull request #5270 from dolthub/aaron/online-gc-prune-table-files-change
...
go/store/types: value_store.go: Change GC implementation to call TableFileStore.PruneTableFiles after the copy is complete.
2023-02-02 12:19:05 -08:00
Andy Arthur
7ae021dd6a
go/store/datas/pull: refactored puller code again
2023-02-02 10:22:29 -08:00
Andy Arthur
002641e5ab
go/store/datas/pull: fix bug in visited set update
2023-02-01 17:36:27 -08:00
Andy Arthur
0bd07d8252
go/store/datas/pull: unref visited set in pull earlier
2023-02-01 16:50:21 -08:00
Andy Arthur
83c369e47b
go/store/datas/pull: some renaming in puller
2023-02-01 16:47:30 -08:00
Andy Arthur
d7b58abc8b
go/store/datas/pull: try to avoid unbounded growth of outstanding absent set
2023-02-01 16:45:07 -08:00
Andy Arthur
c782af7a4a
go/store/datas/pull: also reduce nextAbsent footprint
2023-02-01 15:29:16 -08:00
Andy Arthur
1c39fea3c1
go/store/datas/pull: trim memory footprint for puller hash sets
2023-02-01 15:13:01 -08:00
Brian Hendriks
56706c0826
add another layer of batching
2023-01-31 15:51:11 -08:00
Aaron Son
ff63732b49
go/store/types: value_store.go: Change GC implementation to call TableFileStore.PruneTableFiles after the copy is complete.
2023-01-31 15:48:39 -08:00
Taylor Bantle
cee7e15eb7
Remove datas/pull
2023-01-12 10:23:58 -08:00
Taylor Bantle
546cca8f0c
Fix gc unit tests
2023-01-12 09:26:25 -08:00
Taylor Bantle
f7d2a767f9
Use blobstore for MemFactory
2023-01-12 09:26:25 -08:00
Taylor Bantle
3339cf34c2
Remove PutMany
2023-01-12 09:26:02 -08:00
Taylor Bantle
7b759f2c75
More fixes
2023-01-12 09:26:02 -08:00
Taylor Bantle
98b073070c
Fix some test failures
2023-01-12 09:26:02 -08:00
Taylor Bantle
98871db953
First attempt at PutMany
2023-01-12 09:26:02 -08:00
Taylor Bantle
3b06cb373b
Uncomment getAddrs code that breaks tests
2023-01-12 09:26:02 -08:00
Taylor Bantle
18fedd79ad
Add sanity check to Put
...
This reverts commit b1de143a16
2023-01-12 09:26:02 -08:00
Aaron Son
d957daf6b6
go/libraries/doltcore/remotestorage: chunk_store: Fix semantics of Rebase and Root. Root stays fixed until a Rebase call.
...
When remotestorage.DoltChunkStore was originally written, it forwarded all
ChunkStore calls to the remotesapi. In reality, a client should expect a
snapshot of the database on open, and the root should only change when Rebase()
or Commit() is called. This fixes up that initial flaw. Rebase() becomes a
refresh of the Root from the gRPC service, and Root() itself is just a read of
the cached root hash.
This reduces unnecessary traffic to the remotestorage service and improves
performance of things like sql-server read replicas.
2023-01-06 16:41:26 -08:00
Taylor Bantle
6b9d2faa73
Move TableFileStore back to nbs
2022-12-30 11:02:16 -08:00
Zach Musgrave
2f187be138
Merge pull request #5005 from dolthub/zachmu/replica-perf
...
Perf improvements for remote read replicas
2022-12-20 09:46:53 -08:00
Taylor Bantle
464a3b7c63
Merge pull request #5012 from dolthub/taylor/table-file-store
...
Move TableFileStore from nbs to chunks
2022-12-20 08:12:06 -08:00
zachmu
ad55526c3a
[ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh
2022-12-20 01:07:35 +00:00
Zach Musgrave
09ce74d905
Merge aaron changes
2022-12-19 16:46:01 -08:00
Aaron Son
df85ba3cc8
[no-release-notes] go/store/datas/pull: Update Pull interfaces to take a slice of hashes to pull, not just one.
...
This allows client code to be more efficient when it needs to pull multiple
hashes.
2022-12-19 15:21:27 -08:00
Taylor Bantle
b6848b105a
Move TableFileStore from nbs to chunks
2022-12-16 10:56:51 -08:00
zachmu
2dd7624054
[ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh
2022-12-15 22:46:05 +00:00
Zach Musgrave
52a5163b00
Got rid of no-op pull funcs, more experimental changes
2022-12-12 17:15:52 -08:00
Andy Arthur
97443d8e8c
added pull tests for chunk journal
2022-12-06 15:51:09 -08:00
Taylor Bantle
9becfc9d02
Cleanup
2022-11-21 10:03:33 -08:00
Taylor Bantle
98120eb31d
online prune table files
2022-11-21 10:03:33 -08:00
Dhruv Sringari
69713346eb
if the BPS is less than 1, report 0 ( #4467 )
2022-10-05 14:34:18 -07:00
Aaron Son
e648d9bceb
go/libraries/doltcore/remotestorage: Migrate to backoff/v4, fix some misusage of the backoff API.
2022-09-27 14:45:57 -07:00
Aaron Son
cd971d9a17
go/store/datas/pull: Puller retains byte_sink_ temp file name, instead of flushing file to possibly-conflicting table file name.
2022-08-15 11:47:12 -07:00
coffeegoddd
b68814c796
[ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh
2022-08-05 17:21:26 +00:00
coffeegoddd
60fd6472f8
/go/{store,libraries}: attempt to fix some comments for formatting
2022-08-05 10:20:04 -07:00
Aaron Son
642f2e0ba7
go: More migration away from shim into NodeStore.
2022-07-05 14:16:36 -07:00
Aaron Son
0558d9a99a
go/store/datas: Remove some usage of Format_Default.
2022-05-10 16:20:10 -07:00
Aaron Son
be27368901
go/store/datas: GetCommitValue -> GetCommittedValue.
2022-04-04 13:26:01 -07:00
Aaron Son
a00f430f29
go/store/types: WalkRefsForChunkStore -> WalkAddrsForChunkStore.
2022-04-04 13:14:28 -07:00
Aaron Son
f8ca345ad0
Merge pull request #3123 from dolthub/aaron/pull-stats-rewrite
...
go/store/datas/pull: Use errgroup. Rewrite Stats publishing.
2022-04-04 11:23:30 -07:00
Aaron Son
a94c1421f4
go/store/datas/pull: puller.go: Avoid writing empty table files.
2022-03-31 14:59:08 -07:00
Aaron Son
096269b291
go/store/datas/pull: Remove PullerEvent status updates. Publish pull stats through a Stats struct instead.
2022-03-29 16:37:58 -07:00
Dhruv Sringari
b9f29fd347
Merge pull request #3080 from dolthub/dhruv/memory-quota
...
go/libraries/doltcore/dbfactory: Thread memory quota provider through store layer to onHeapTableIndex
2022-03-29 13:28:25 -07:00
Aaron Son
e0abfac106
go/store/datas/pull: Iterate on new stats.
2022-03-29 12:25:45 -07:00
