Commit Graph

28695 Commits

Author SHA1 Message Date
James Cor
0873caea36 [no-release-notes] Handle new sql.ConvertInRange values (#10189) 2025-12-10 15:47:04 -08:00
Dustin Brown
839a62d04f Merge pull request #10193 from dolthub/coffeegoddd-73e40081
[auto-bump] [no-release-notes] dependency by coffeegoddd
2025-12-10 13:56:36 -08:00
coffeegoddd
0b3bfa1ad2 [ga-bump-dep] Bump dependency in Dolt by coffeegoddd 2025-12-10 20:39:20 +00:00
Elian
a1d078810e Merge pull request #10159 from dolthub/elian/7628c
Add `adapters.TableAdapter` to handle `dolt_status` and other table conversions for integrators (a.k.a. Doltgres)
2025-12-10 11:57:31 -08:00
elianddb
a354d667a0 add copyright in adapters/table.go 2025-12-10 10:16:38 -08:00
Elian
7cff94757a Merge pull request #10097 from dolthub/codeaucafe/1430
dolthub/dolt#10030: `--filter` contribution for `dolt diff`
2025-12-09 15:14:32 -08:00
elianddb
904f8e281c [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh 2025-12-09 22:30:28 +00:00
elianddb
d956afb08a rm privs check in dolt_purge_dropped_databases as it is already handled 2025-12-09 03:30:19 -08:00
coffeegoddd
4fd223dfda [ga-bump-release] Update Dolt version to 1.78.8 and release v1.78.8 v1.78.8 2025-12-09 04:40:11 +00:00
Aaron Son
7dcacba0f4 Merge pull request #10183 from dolthub/aaron/backup-sync-memory-opt
go: store/datas/pull: pull_chunk_tracker.go: Optimize memory use when backing up to an AWS remote.
2025-12-08 20:06:51 -08:00
Aaron Son
e63f912e55 go: store/datas/pull: pull_chunk_tracker.go: PR feedback. 2025-12-08 19:28:48 -08:00
Aaron Son
43da391c52 go: store/datas/pull: pull_chunk_tracker.go: Optimize memory use when backing up to an AWS remote.
PullChunkTracker is responsible for making the HasMany calls against the
destination and batching up absent hashes into HashSets which will be delivered
to GetManyCompressed and eventually written into table files which are
uploaded. This code is used for both pull and push, when the destination is the
"local" database or when destination is the remote database respectively. It is
used when the remote is both doltremoteapi, thus every HasMany call is an RPC,
and when the remote is something like file:// or aws://, thus the table file
indexes for the remote are in memory and HasMany calls are very quick.

Different operational characteristics of the various dependencies mean that
sometimes a Pull is prone to build up large sets of hashes waiting for HasMany
calls, whereas other times it is prone to build up large sets of absent hashes
which are waiting for the fetcher thread(s) to take them.

Previously, PullChunkTracker was structured to accumulate HasMany responses and
wait to batch them into appropriately-sized batches for GetManyCompressed until
the fetcher threads asked for them. This meant that if HasMany batches were
very small, because HasMany was very fast, we would accumulate a large number
of very small HashSets. These HashSets would take up large amounts of memory.
Accumulating the batches as the HasMany responses come in is more memory
efficient and should be no slower - we will always accumulate the full batches,
and in basically the same order.

Tested by pushing a large database to an AWS remote and memory profiling the
result.
2025-12-08 19:26:44 -08:00
Tim Sehn
172098be14 Merge pull request #10182 from dolthub/timsehn-46280834
[auto-bump] [no-release-notes] dependency by timsehn
2025-12-08 17:06:59 -08:00
timsehn
243d5e1190 [ga-bump-dep] Bump dependency in Dolt by timsehn 2025-12-09 00:36:40 +00:00
elianddb
16069d8889 rename table_adapter.go -> table.go 2025-12-08 16:22:04 -08:00
Elian
a4087f5709 Merge pull request #10164 from dolthub/elian/10136
dolthub/dolt#10136: Fix `dolt_backup` to work in non-Dolt directories
2025-12-08 13:17:49 -08:00
elianddb
a17a3564b3 fix 'NewStatusTable' func doc 2025-12-08 12:42:04 -08:00
elianddb
f9ff5ed52f add test for Dolt table adapter registry 2025-12-08 12:01:03 -08:00
elianddb
6990d48af4 fix generated system table name set to use adapter names conditionally 2025-12-08 11:50:38 -08:00
elianddb
921d8a8909 amend func docs 2025-12-08 04:55:10 -08:00
elianddb
cd084b0cf1 rm extra default adapter and normalize internal alias instead 2025-12-08 04:49:38 -08:00
elianddb
752db82b84 add dolt table specific adapter registry for integrators 2025-12-05 16:32:41 -08:00
coffeegoddd
628b9dcce5 [ga-bump-release] Update Dolt version to 1.78.7 and release v1.78.7 v1.78.7 2025-12-05 23:21:23 +00:00
Nick Tobey
ef273f60d6 Merge pull request #10178 from dolthub/nicktobey/json-null
Report merge conflict when merging a JSON document that is NULL in the common ancestor.
2025-12-05 14:39:12 -08:00
Nick Tobey
a896b44cdc Merge pull request #10179 from dolthub/nicktobey-d4cc0dab
[auto-bump] [no-release-notes] dependency by nicktobey
2025-12-05 14:26:11 -08:00
elianddb
d7cf4b6d3b add dolt adapter to standardize table overwrites by integrators 2025-12-05 13:27:00 -08:00
nicktobey
9912094ce2 [ga-bump-dep] Bump dependency in Dolt by nicktobey 2025-12-05 21:13:48 +00:00
Nick Tobey
984eeba88d Report conflicts in JSON merge when one value is NULL 2025-12-05 13:03:03 -08:00
Nick Tobey
09b5037afe Add json merge tests with NULLs 2025-12-05 13:02:38 -08:00
Dustin Brown
f5562654c9 Merge pull request #10172 from dolthub/db/fix-workflow
[no-release-notes]: don't close failing auto bumps
2025-12-05 09:49:33 -08:00
Nick Tobey
99c2c4b65d Merge pull request #10120 from dolthub/nicktobey/commit_diff
Allow dolt_commit_diff_ to diff against HEAD
2025-12-04 18:40:38 -08:00
coffeegoddd☕️✨
613d8159cd /.github/workflows/bump-dependency.yaml: fix typo 2025-12-04 17:06:11 -08:00
coffeegoddd☕️✨
c9776bb229 /.github/workflows/bump-dependency.yaml: ensure failing bumps are not autoclosed 2025-12-04 17:03:03 -08:00
angelamayxie
f73ec2d0e9 Merge pull request #10155 from dolthub/angela/datetime
Update `datetime` types to correct precision for dolt system tables
2025-12-04 15:58:05 -08:00
angelamayxie
ca02be3078 Merge pull request #10170 from dolthub/angelamayxie-f6c4ebdf
[auto-bump] [no-release-notes] dependency by angelamayxie
2025-12-04 15:28:21 -08:00
angelamayxie
6ef776e452 Merge branch 'angela/datetime' into angelamayxie-f6c4ebdf 2025-12-04 15:28:05 -08:00
angelamayxie
3f66120757 [ga-bump-dep] Bump dependency in Dolt by angelamayxie 2025-12-04 23:26:10 +00:00
angelamayxie
e896f3515b update datetime precision for tags and branch activity 2025-12-04 14:31:47 -08:00
elianddb
d9cd4eea4c add backup.bats test to make sure command is allowed to run in invalid repo 2025-12-04 11:54:07 -08:00
angelamayxie
a44000fb3d Merge branch 'angela/datetime' of https://github.com/dolthub/dolt into angela/datetime 2025-12-04 11:33:57 -08:00
coffeegoddd
3c8b436917 [ga-bump-release] Update Dolt version to 1.78.6 and release v1.78.6 v1.78.6 2025-12-04 18:48:10 +00:00
elianddb
298c4ee0a3 amend dolt_backup impl to work in non-dolt directories 2025-12-04 10:18:38 -08:00
Jason Fulghum
93bd0854d2 Merge pull request #10163 from dolthub/fulghum/bugfix
Bug fix: correctly copy check constraints
2025-12-04 10:13:47 -08:00
Jason Fulghum
408f1793aa Bug fix for Copy() implementation that wasn't returning a copy 2025-12-04 09:38:12 -08:00
angelamayxie
d14588ffda update datetime type 2025-12-03 15:39:27 -08:00
elianddb
73428104fc rm UserHasSuperAccess and move to doltgresql 2025-12-03 12:50:40 -08:00
elianddb
3655b40fd0 mv priv check to var 2025-12-03 11:57:28 -08:00
angelamayxie
36b7fbc297 Merge branch 'main' into angela/datetime 2025-12-02 16:22:12 -08:00
angelamayxie
2ff68bb99c update datetime type in log table function test 2025-12-02 16:20:30 -08:00
Dustin Brown
8c2a083399 [ga-bump-dep] Bump dependency in Dolt by jycor (#10154)
Co-authored-by: jycor <jcor@ucsd.edu>
2025-12-02 16:03:54 -08:00