Commit Graph

28674 Commits

Author SHA1 Message Date
Brian Hendriks 7900980c32 pr feedback 2025-12-09 15:51:03 -08:00
Brian Hendriks 18b4accaf8 PR feedback 2025-12-09 15:18:23 -08:00
Brian Hendriks 044bb017b7 Update go/cmd/dolt/commands/sqlserver/metrics_auth.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-09 15:11:48 -08:00
Brian Hendriks 05300b9165 header fix 2025-12-09 14:41:16 -08:00
Brian Hendriks a2dae20341 fix command line config 2025-12-09 14:07:36 -08:00
Brian Hendriks 74ba3e71f0 remove dead code 2025-12-09 13:55:36 -08:00
Brian Hendriks 4514b456f1 add jwt auth for metrics requests 2025-12-09 13:53:58 -08:00
coffeegoddd 4fd223dfda [ga-bump-release] Update Dolt version to 1.78.8 and release v1.78.8 v1.78.8 2025-12-09 04:40:11 +00:00
Aaron Son 7dcacba0f4 Merge pull request #10183 from dolthub/aaron/backup-sync-memory-opt
go: store/datas/pull: pull_chunk_tracker.go: Optimize memory use when backing up to an AWS remote.
2025-12-08 20:06:51 -08:00
Aaron Son e63f912e55 go: store/datas/pull: pull_chunk_tracker.go: PR feedback. 2025-12-08 19:28:48 -08:00
Aaron Son 43da391c52 go: store/datas/pull: pull_chunk_tracker.go: Optimize memory use when backing up to an AWS remote.
PullChunkTracker is responsible for making the HasMany calls against the
destination and batching up absent hashes into HashSets which will be delivered
to GetManyCompressed and eventually written into table files which are
uploaded. This code is used for both pull and push, when the destination is the
"local" database or when destination is the remote database respectively. It is
used when the remote is both doltremoteapi, thus every HasMany call is an RPC,
and when the remote is something like file:// or aws://, thus the table file
indexes for the remote are in memory and HasMany calls are very quick.

Different operational characteristics of the various dependencies mean that
sometimes a Pull is prone to build up large sets of hashes waiting for HasMany
calls, whereas other times it is prone to build up large sets of absent hashes
which are waiting for the fetcher thread(s) to take them.

Previously, PullChunkTracker was structured to accumulate HasMany responses and
wait to batch them into appropriately-sized batches for GetManyCompressed until
the fetcher threads asked for them. This meant that if HasMany batches were
very small, because HasMany was very fast, we would accumulate a large number
of very small HashSets. These HashSets would take up large amounts of memory.
Accumulating the batches as the HasMany responses come in is more memory
efficient and should be no slower - we will always accumulate the full batches,
and in basically the same order.

Tested by pushing a large database to an AWS remote and memory profiling the
result.
2025-12-08 19:26:44 -08:00
Tim Sehn 172098be14 Merge pull request #10182 from dolthub/timsehn-46280834
[auto-bump] [no-release-notes] dependency by timsehn
2025-12-08 17:06:59 -08:00
timsehn 243d5e1190 [ga-bump-dep] Bump dependency in Dolt by timsehn 2025-12-09 00:36:40 +00:00
Elian a4087f5709 Merge pull request #10164 from dolthub/elian/10136
dolthub/dolt#10136: Fix `dolt_backup` to work in non-Dolt directories
2025-12-08 13:17:49 -08:00
coffeegoddd 628b9dcce5 [ga-bump-release] Update Dolt version to 1.78.7 and release v1.78.7 v1.78.7 2025-12-05 23:21:23 +00:00
Nick Tobey ef273f60d6 Merge pull request #10178 from dolthub/nicktobey/json-null
Report merge conflict when merging a JSON document that is NULL in the common ancestor.
2025-12-05 14:39:12 -08:00
Nick Tobey a896b44cdc Merge pull request #10179 from dolthub/nicktobey-d4cc0dab
[auto-bump] [no-release-notes] dependency by nicktobey
2025-12-05 14:26:11 -08:00
nicktobey 9912094ce2 [ga-bump-dep] Bump dependency in Dolt by nicktobey 2025-12-05 21:13:48 +00:00
Nick Tobey 984eeba88d Report conflicts in JSON merge when one value is NULL 2025-12-05 13:03:03 -08:00
Nick Tobey 09b5037afe Add json merge tests with NULLs 2025-12-05 13:02:38 -08:00
Dustin Brown f5562654c9 Merge pull request #10172 from dolthub/db/fix-workflow
[no-release-notes]: don't close failing auto bumps
2025-12-05 09:49:33 -08:00
Nick Tobey 99c2c4b65d Merge pull request #10120 from dolthub/nicktobey/commit_diff
Allow dolt_commit_diff_ to diff against HEAD
2025-12-04 18:40:38 -08:00
coffeegoddd☕️✨ 613d8159cd /.github/workflows/bump-dependency.yaml: fix typo 2025-12-04 17:06:11 -08:00
coffeegoddd☕️✨ c9776bb229 /.github/workflows/bump-dependency.yaml: ensure failing bumps are not autoclosed 2025-12-04 17:03:03 -08:00
angelamayxie f73ec2d0e9 Merge pull request #10155 from dolthub/angela/datetime
Update `datetime` types to correct precision for dolt system tables
2025-12-04 15:58:05 -08:00
angelamayxie ca02be3078 Merge pull request #10170 from dolthub/angelamayxie-f6c4ebdf
[auto-bump] [no-release-notes] dependency by angelamayxie
2025-12-04 15:28:21 -08:00
angelamayxie 6ef776e452 Merge branch 'angela/datetime' into angelamayxie-f6c4ebdf 2025-12-04 15:28:05 -08:00
angelamayxie 3f66120757 [ga-bump-dep] Bump dependency in Dolt by angelamayxie 2025-12-04 23:26:10 +00:00
angelamayxie e896f3515b update datetime precision for tags and branch activity 2025-12-04 14:31:47 -08:00
elianddb d9cd4eea4c add backup.bats test to make sure command is allowed to run in invalid repo 2025-12-04 11:54:07 -08:00
angelamayxie a44000fb3d Merge branch 'angela/datetime' of https://github.com/dolthub/dolt into angela/datetime 2025-12-04 11:33:57 -08:00
coffeegoddd 3c8b436917 [ga-bump-release] Update Dolt version to 1.78.6 and release v1.78.6 v1.78.6 2025-12-04 18:48:10 +00:00
elianddb 298c4ee0a3 amend dolt_backup impl to work in non-dolt directories 2025-12-04 10:18:38 -08:00
Jason Fulghum 93bd0854d2 Merge pull request #10163 from dolthub/fulghum/bugfix
Bug fix: correctly copy check constraints
2025-12-04 10:13:47 -08:00
Jason Fulghum 408f1793aa Bug fix for Copy() implementation that wasn't returning a copy 2025-12-04 09:38:12 -08:00
angelamayxie d14588ffda update datetime type 2025-12-03 15:39:27 -08:00
angelamayxie 36b7fbc297 Merge branch 'main' into angela/datetime 2025-12-02 16:22:12 -08:00
angelamayxie 2ff68bb99c update datetime type in log table function test 2025-12-02 16:20:30 -08:00
Dustin Brown 8c2a083399 [ga-bump-dep] Bump dependency in Dolt by jycor (#10154)
Co-authored-by: jycor <jcor@ucsd.edu>
2025-12-02 16:03:54 -08:00
angelamayxie 6d94494ee0 Merge branch 'angela/datetime' of https://github.com/dolthub/dolt into angela/datetime 2025-12-02 15:54:59 -08:00
angelamayxie 25f521b5da update datetime type to datetime3 2025-12-02 15:54:53 -08:00
angelamayxie 8f6b47c7ee [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh 2025-12-02 23:45:43 +00:00
angelamayxie 8e63921cef update datetime type to datetime3 for dolt tables 2025-12-02 15:35:40 -08:00
James Cor 8dfb36bb44 Change nodeCache from array to slice 2025-12-02 15:21:40 -08:00
Aaron Son b8ba076afc Merge pull request #10126 from dolthub/aaron/add-rpms-to-release-artifacts
.github,go/utils/{publishrelease,rpmbuild}: Add a basic RPM build to the published release artifacts for Dolt.
2025-12-02 14:52:52 -08:00
Dustin Brown 37204f5be6 [ga-bump-dep] Bump dependency in Dolt by jycor (#10152) 2025-12-02 14:40:39 -08:00
Elian aae600a213 Merge pull request #10147 from dolthub/elian/10138
dolthub/dolt#10138: Fix `dolt_backup` sync and sync-url not taking working set changes in transaction
2025-12-02 13:03:36 -08:00
elianddb 4231e7a901 mv CommitWorkingSet inside of syncRemote() 2025-12-02 11:35:02 -08:00
elianddb d4d6dc0c2d fix dolt_backup sync and sync-url not working in transaction 2025-12-02 10:25:36 -08:00
Elian 6fc0e63acc Merge pull request #10146 from dolthub/elian/7628b
dolthub/dolt#7628: Amend `dolt_backup` DoltgresSQL privilege check for server only
2025-12-02 10:19:39 -08:00