Commit Graph

131 Commits

Author SHA1 Message Date
Zach Musgrave b8f2ece723 More SQL type changes 2022-05-11 17:22:13 -07:00
Zach Musgrave ab793216b3 Fixed more SQL types 2022-05-11 17:14:07 -07:00
Aaron Son 108390b179 go/store/types: TupleRowStorage: Implement walkRefs. 2022-05-05 15:54:42 -07:00
Aaron Son afab5f544c doltdb: Commit: Store a *datas.Commit intsead of rootV, CommitMeta, stref, etc. 2022-04-04 13:26:01 -07:00
Maximilian Hoffman 2049664ff9 Simplify RebuildIndex (#3053)
* Simplify RebuildIndex

Two improvements with index rewriting:

1) Brian's `TupleFactory` optimizations were being skipped, and 70% of
the runtime was spent building `newBinaryNomsWriter` instances for tuple
construction. Removing this restores index inserts as the only memory
pressure.

2) Brian's `BulkEditAccumulator` lets the final Map sort skip
work materializing intermediate prolly trees.

On a 2000 row table, fix (1) reduces the runtime by 50% (80ms -> 40ms),
and fix (2) reduces the runtime by another 40% (40ms -> 10ms)%, for a
cumulative ~90% improvement.

* fixes
2022-03-22 07:07:45 -07:00
Lars Lehtonen ce4744c9f7 libraries/doltcore/dtestutils/testcommands: fix dropped errors 2022-03-16 00:15:02 -07:00
Aaron Son 478b8312b7 go/store/datas: Move CommitMeta and TagMeta to datas. 2022-02-28 11:29:25 -08:00
Andy Arthur d243032e0e merged main 2022-02-01 13:11:02 -08:00
Andy Arthur a729a63f76 first pass at secondary indexes 2022-01-31 15:00:45 -08:00
Jason Fulghum 53d18f895d Bug fix for dropped table check constraints when altering a table.
Resolves: https://github.com/dolthub/dolt/issues/2672

Confirmed that tests now cover all schema properties.
2022-01-26 16:09:38 -08:00
Andy Arthur 7805819bb2 Merge branch 'main' into andy/prolly-indexes 2022-01-19 18:24:50 -08:00
Vinai Rachakonda ea6815adc3 Modify Add Column to apply defaults in engine (#2580) 2022-01-19 17:48:37 -08:00
Andy Arthur 761510e4d5 threading prolly indexes 2022-01-11 14:54:54 -08:00
Andy Arthur 7a82e0c6e9 renamed doltdb.Table.GetRowData() to GetNomsRowData() 2022-01-06 14:14:48 -08:00
Andy Arthur bd517a6be2 replaced index types.Map with durable.IndexSet 2022-01-05 15:58:02 -08:00
Aaron Son 034d49629c Bump GMS, pick up ctx parameter for RowIter.Next(). 2021-12-16 15:00:18 -08:00
Andy Arthur 11f9aa1795 repo fmt 2021-12-13 15:50:44 -08:00
Andy Arthur 3466021f98 Merge branch 'main' into andy/remove-noms-map-creator 2021-12-13 15:48:31 -08:00
AndyA 47f3030ef5 Merge pull request #2497 from dolthub/andy/durable
/go/libraries/doltcore/doltdb: Extracted Persistent Object from `doltdb.Table`
2021-12-13 15:42:52 -08:00
Andy Arthur 3e800180af removed noms map creator 2021-12-13 15:23:51 -08:00
Andy Arthur 499d3488ab refactored conflict schemas 2021-12-13 11:08:06 -08:00
Maximilian Hoffman 0cbd7e780c Async push for replication (#2369)
* starter code

* async replication prototype

* benchmark async push

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* delete hanging line

* add comment for concurrent map access

* move replication into databaseProvider

* top-level wait group for async threads

* fix testsg

* standardize wg format

* missing header

* address data race

* make async test shorter

* delete unnecessary lines

* starter code for new sqlEngine interface

* new engine interface with background threads

* brian's comments

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* missed a datarace

* fix bats

* bad go.sum

* add latest GMS commit

* bump GMS

Co-authored-by: max-hoffman <max-hoffman@users.noreply.github.com>
2021-12-11 13:41:58 -08:00
jennifersp 6dff0cac41 fix missed func 2021-11-29 17:30:16 -08:00
Maximilian Hoffman 45a3374808 DOLT_REPLICATE_HEADs_STRATEGY option, fetch all branches from remote before every read (#2346)
* save version with defaultConfig in session -- todo remove and make PersistedSession

* add PersistedSession, and DoltSession interface to switch

* fmt

* fixup test for persistedSession

* PersistedSession progress, interfaces and enginetests

* All DsessFromSess references fixed

* delete unnuecessary lines

* Comments for dolt session

* Dolt session tests

* save bats progress

* formatting

* GMS version

* Bump GMS version

* add license header

* fix multi-db errors

* try copyright again

* make test deterministic

* higher connection timeout default

* sever config ordering

* PR fixes

* PR fixes

* config rewrite progress

* server config refactor

* PR cleanup

* delete lines

* add go tests, refactor persistent global initialization

* PR fixes

* delete more lines

* small PR cleanup

* try to fix data race

* better constructor names

* zach don't like --server flag

* edge cases around disabling replication

* missed line in merge

* Skip two tests i fixed in follow-on PR

* commit hook warning progress

* merge two config prs

* fix nondeterministic bats

* zach's comments

* bump bats sql-shell expect timeout

* non-server progress

* replication warnings

* Get and set config bool types

* More bats, small PR fixes

* missing header

* fix bats panic

* PR touch ups

* flag, logic and tests for pulling multiple heads to replica

* change name because enginetest, fix impl typo

* missed file commit, config var name changed to strategy

* delete unnecessary lines

* progress, need to merge other PR

* prototype working

* delete some lines

* don't make a new branch until after fetch

* zach's comments

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* fix import cycle

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* fix import cycle, sys vars in sqle

* style fixes, one more bats

* split heads spec and all heads into two variables

* remove unnecessary rebase

* bad merge duplicated bats

* fix on demand fetch after multi head pull changes

* correctness, bats and zach fixes

* drop unnecessary server test

* bad bats title

* small PR fixes

* correctness fix for detached head / fetch branch

* fix comment

Co-authored-by: max-hoffman <max-hoffman@users.noreply.github.com>
2021-11-09 10:37:09 -08:00
Zach Musgrave 9f177fec26 Finished fixing compile errors 2021-11-04 10:53:15 -07:00
Maximilian Hoffman ca5ae29dd1 replica benchmark, no event progress for server (#2258)
* small push on write perf benchmark with no prog channel

* remove unused prog funcs

* remove unused prog funcs

* fix bench name

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

Co-authored-by: max-hoffman <max-hoffman@users.noreply.github.com>
2021-10-28 15:34:04 -07:00
Eng Zer Jun 8164a625c7 refactor: move from io/ioutil to io and os package
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-10-05 01:40:59 +08:00
Maximilian Hoffman 8a5f3f54be config, clone, backups refactor (#2196)
* shittiest read replica imaginable is kind of working

* import cycle progress

* delete unecessary files and fix  db type switch bug

* Add bats test

* delete comments

* fix working set updates for cli

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* clean comments

* comment racy server test

* move env variables to local dolt config

* refactor clone, fix multienv test

* cleanup comments

* missing copyright

* brian's comments, add compile time checks for config interfaces

* format

* fix windows filepaths issue

* file:/// with three slashes should work on windows

* more windows problems

* three slashes didn't work for clone, do chdir to ref local dolt db

Co-authored-by: max-hoffman <max-hoffman@users.noreply.github.com>
2021-09-29 19:10:37 -07:00
Maximilian Hoffman a0f6799a2c FF head on read replica (#2182)
* shittiest read replica imaginable is kind of working

* import cycle progress

* delete unecessary files and fix  db type switch bug

* Add bats test

* delete comments

* fix working set updates for cli

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* clean comments

* comment racy server test

* zach's comments

* add copyright headers

Co-authored-by: max-hoffman <max-hoffman@users.noreply.github.com>
2021-09-29 10:23:07 -07:00
Dustin Brown b5904ee069 /{go,integration-tests}: comment on recycled pids, use exported default branch where possible 2021-09-24 12:37:18 -07:00
Dustin Brown baeb4fd656 /{go,integration-tests}: address pr feedback 2021-09-23 17:37:04 -07:00
Dustin Brown 52782575fd /go/{store,performance,libraries,cmd}: fix go tests after branch rename 2021-09-23 14:29:33 -07:00
Brian Hendriks 775ac0af51 Import Performance and Garbage Generation improvements. 2021-09-08 14:52:42 -07:00
Maximilian Hoffman f5d63f4e3d move common merge/push/pull code into doltcore/env (#2063)
* Start moving code out of cmd

* hacky cmd -> action/evn refactor

* progress

* String verbose error from lib, mostly not reconstructed on cli side

* Missing verbose errors in lib, still not handled in cli. Refactor vinai's doc diffs to not depend on merge

* Small progress commit on other laptop

* Merge bats passing

* format

* progress

* Remotes bats tests working

* More error fixes

* Fix cyclic foreign keys bats

* Fix import cycle

* Start cleaning up code

* Fix a couple bats

* Clean up comments and println

* Brian's comments
2021-08-31 15:23:23 -07:00
Zach Musgrave 00624bcad9 Merge branch 'zachmu/cleanup' of github.com:dolthub/dolt into zachmu/cleanup 2021-08-05 12:51:35 -07:00
Zach Musgrave bfe976375f PR feedback 2021-08-05 12:51:29 -07:00
zachmu 4a75e8bbd2 [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh 2021-08-05 03:11:56 +00:00
Zach Musgrave f8b2c234e5 Fix error in test 2021-08-04 20:10:33 -07:00
Zach Musgrave 55e094bef0 Removed PutTableToWorking 2021-08-04 19:17:21 -07:00
Zach Musgrave e67a480c86 Cleaned up docs staging code 2021-08-04 18:48:23 -07:00
Zach Musgrave 29565615fe Made StageAllTables functional 2021-08-04 17:50:34 -07:00
Zach Musgrave f60ee9ca0d Fix commit for active merge bug I introduced 2021-08-04 15:58:58 -07:00
Zach Musgrave 2c4b068f66 Cleared out the merge state in a few more places it was needed 2021-08-04 15:31:03 -07:00
Zach Musgrave 0a2521bdc8 Got rid of the final two misplaced methods on RepoStateReader 2021-08-04 15:08:15 -07:00
Zach Musgrave 75dd2a58c7 Merge branch 'master' into zachmu/cleanup 2021-08-04 09:58:07 -07:00
Daylon Wilkins 6148bded9d Changed commit force to ignore conflicts and constraint violations 2021-07-29 14:12:59 -07:00
Zach Musgrave 797a2ac9ae Moved MergeWouldStompWorkingChanges out of the repo_state.go file 2021-07-15 11:46:56 -07:00
Zach Musgrave 22e0ad8365 Fixed a bunch more tests 2021-06-24 16:24:07 -07:00
Zach Musgrave 532d1dd7fe Passing roots into more places 2021-06-24 09:50:44 -07:00
Zach Musgrave e27f60d2c9 Merge state refactoring 2021-06-22 18:31:06 -07:00