Commit Graph

454 Commits

Author SHA1 Message Date
Zach Musgrave 9136202b51 More type changes 2022-05-11 17:39:49 -07:00
James Cor 2f0338dd17 merge james/collate into auto-bump 2022-05-06 11:12:25 -07:00
Dhruv Sringari 69bde477cf [no-release-notes] implement cell-wise merges for new storage format (#3346)
* implement cell-wise merges and on the fly secondary index corrections

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* move old storage format fix to new PR

* pr feedback

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* pr feedback 2, no pointers to interfaces, and use val.OrdinalMapping

* copyright header

* need for loop around select, buffer cellWiseChan, pr edits

Co-authored-by: druvv <druvv@users.noreply.github.com>
2022-05-05 11:03:06 -07:00
James Cor 9bf8742557 fixing tests 2022-05-05 10:53:30 -07:00
Dhruv Sringari 4e36408bfd mergeTableData for new storage format, with cell-wise merge (no conflicts) 2022-04-27 17:42:50 -07:00
Maximilian Hoffman 122cf63553 skip -race enginetests until merge (#3286)
* skip prepared tests when -race testing

* test.short

* missing env var

* skip race enginetests

* race on merge main

* put new format skip back

* simplify matrix

* aaron's comment
2022-04-23 09:21:49 -07:00
Vinai Rachakonda 3aad420681 Enable unique indexes on keyless tables (#3245) 2022-04-21 12:23:18 -04:00
James Cor 03a9a5e68f dumps working 2022-04-15 11:25:12 -07:00
James Cor d6088a084f procedures dumping 2022-04-14 16:18:57 -07:00
James Cor f746da30ef first pass at dumping views and triggers 2022-04-14 13:24:33 -07:00
Maximilian Hoffman 2fd67bfbf7 Bulk IEA memory improvements (#3191)
Partials map takes ~90% less space. Edit maps flush to disk periodically
to curb unbounded growth.
2022-04-11 11:05:22 -07:00
Daylon Wilkins 8516471e2c Moved the bulk of foreign key processing into the SQL engine 2022-04-10 04:00:17 -07:00
Andy Arthur 107e8511b0 Merge remote-tracking branch 'origin/main' into andy/new-auto-increment 2022-03-25 13:41:26 -07:00
Andy Arthur 99a16ef90b pr feedback 2022-03-25 13:38:07 -07:00
Maximilian Hoffman 2049664ff9 Simplify RebuildIndex (#3053)
* Simplify RebuildIndex

Two improvements with index rewriting:

1) Brian's `TupleFactory` optimizations were being skipped, and 70% of
the runtime was spent building `newBinaryNomsWriter` instances for tuple
construction. Removing this restores index inserts as the only memory
pressure.

2) Brian's `BulkEditAccumulator` lets the final Map sort skip
work materializing intermediate prolly trees.

On a 2000 row table, fix (1) reduces the runtime by 50% (80ms -> 40ms),
and fix (2) reduces the runtime by another 40% (40ms -> 10ms)%, for a
cumulative ~90% improvement.

* fixes
2022-03-22 07:07:45 -07:00
Zach Musgrave e0f60063d3 Fixed bug in automatic index creation on referenced table during foreign key creation. Two bugs: the index was being incorrectly created as unique in some cases; and index was being incorrectly permitted when it wasn't a prefix of the key. 2022-03-18 17:19:59 -07:00
Andy Arthur 43a884af57 reimplemented auto increment using new GMS interface 2022-03-17 15:01:41 -07:00
Aaron Son c6b08323e7 Merge pull request #2929 from dolthub/aaron/datas-new-format-nonchunking-refmap
go/store/datas: Begin work for moving top-of-DAG chunks to flatbuffers messages.
2022-03-07 14:29:55 -08:00
James Cor 06892bb6ce removing nil row at end 2022-03-07 11:24:33 -08:00
James Cor eb5650cd39 Update go/libraries/doltcore/table/untyped/sqlexport/batch_sqlwriter.go
Co-authored-by: AndyA <andy@dolthub.com>
2022-03-07 11:03:16 -08:00
James Cor 53e79106d9 merge with main 2022-03-04 10:02:35 -08:00
Aaron Son e0b8c4f46b go/store/types: Create a __DOLT_DEV__ format.
Moves go/store/datas to always use ReadValue/WriteValue, and creates a
SerialMessage types.Value type whose implementation is a lot like InlineBlob.
Implements WalkRefs for SerialMessage.

This lets the full stack of dolt functionality work while we transition types
from the top of the DAG.
2022-03-03 16:23:35 -08:00
JCOR11599 282811526d [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh 2022-03-03 23:53:05 +00:00
James Cor 4493a5b9dc Merge branch 'james/dump-batch' of https://github.com/dolthub/dolt into james/dump-batch 2022-03-03 15:45:17 -08:00
James Cor 2dd1db1ed8 batch dolt rows too 2022-03-03 15:45:14 -08:00
JCOR11599 12a3676a26 [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh 2022-03-03 23:36:52 +00:00
James Cor 517eb2bb56 fixing tests 2022-03-03 15:35:38 -08:00
James Cor 22ce2a26bc moving option inside of datamover 2022-03-03 13:40:04 -08:00
James Cor f491b47c79 restructured batch to use prefix then tuple 2022-03-03 13:23:10 -08:00
James Cor b86dc647d8 moving error check 2022-03-03 12:43:26 -08:00
James Cor 194fad24ed renamed constant 2022-03-03 12:22:52 -08:00
James Cor 75faa91bcb removing batchrow from other writers 2022-03-02 15:29:36 -08:00
James Cor 6a88482d95 removing unused variables, and increasing batch size limit 2022-03-02 15:03:09 -08:00
JCOR11599 6b1105124e [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh 2022-03-02 22:51:12 +00:00
James Cor e583f67d41 Merge branch 'james/dump-batch' of https://github.com/dolthub/dolt into james/dump-batch 2022-03-02 14:50:00 -08:00
James Cor 5f81928b96 working, probably 2022-03-02 14:49:54 -08:00
James Cor 165b10928c compiles 2022-03-02 14:06:19 -08:00
JCOR11599 bf95ec2de2 [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh 2022-03-02 21:25:28 +00:00
James Cor a324a05d6c removing now useless bool 2022-03-02 13:24:05 -08:00
James Cor cec4733ccb added ability to limit number of inserts per row 2022-03-02 13:21:48 -08:00
James Cor 0cadc528e5 renamed WriteNoNewLine to WriteWithoutNewLine 2022-03-02 12:16:56 -08:00
JCOR11599 9669c95f9f [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh 2022-03-01 23:49:09 +00:00
James Cor 88b6a6f197 working for sql format; not sure if i need any other formats 2022-03-01 15:42:33 -08:00
James Cor 1cc1266f44 quick fix; need to write tests 2022-02-28 16:54:36 -08:00
Maximilian Hoffman bdab3358f1 Remove index filters (#2828)
* GMS bump

* eliminate OR filters

* fix range filters and datetime formatting

* GMS bump

* cleanup

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* remove some plan bats

* simplify all checks into is not null

* better comments

* daylon's comments

Co-authored-by: max-hoffman <max-hoffman@users.noreply.github.com>
2022-02-24 22:32:22 -08:00
Aaron Son d5a157dc92 Merge remote-tracking branch 'origin/main' into aaron/database-cleanup 2022-02-08 13:27:23 -08:00
Andy Arthur e9d86aa992 pr feedback 2022-02-07 14:23:19 -08:00
Andy Arthur 78156bb780 added val.OrdinalMapping 2022-02-07 12:16:28 -08:00
Andy Arthur 2fad103f8e cleanup 2022-02-06 12:03:31 -08:00
Andy Arthur e751789b1b Merge branch 'main' into andy/prolly-indexes-2 2022-02-06 10:58:15 -08:00