Commit Graph

25 Commits

Author SHA1 Message Date
Aaron Son 7b777e921f go/libraries/doltcore/sqle: ReadReplicaDatabase: Add variable for force pulling remote branches. 2022-09-20 15:46:53 -07:00
Zach Musgrave 3d47f77c4f Merge pull request #4276 from dolthub/zachmu/auto-clone
Automatically push and pull new databases in replicated servers
2022-09-07 15:43:38 -07:00
Zach Musgrave 32a97e385b Fixed error message in bats test 2022-09-07 14:01:15 -07:00
jennifersp 6910ac96b7 allow init with empty working set (#4255) 2022-09-02 13:21:27 -07:00
Tan Yong Zhi 178f541f34 Fix bats tests 2022-08-26 12:14:05 +08:00
Tan Yong Zhi 9b4b805268 Fix more failing bats tests 2022-08-26 12:14:05 +08:00
Maximilian Hoffman e69cd3c457 replicate delete branch (#3816)
* replicate delete branch

* fixes and tests

* zach's comments
2022-07-14 14:31:30 -07:00
Maximilian Hoffman 3ec0e48b0d dolt_checkout and read replication working set fixes (#3808)
* dolt_checkout and read replication working set fixes

Sql path 'dolt_checkout' used to create a new branch from HEAD
if the checkout name did not have a remote tracking ref. The CLI,
on the other hand, will initialize a local branch if a remote tracking
ref with the same name in the default remote exists. The SQl path
now mirrors the CLI checkout: if we checkout a new branch, like
'new_feature', and we have previously fetched that branch from our
default remote, `dolt_checkout('feature')` will initialize a branch
using the head commit from the remote `feature`.

Second, read replication was creating a local tracking branch for
remote branches (sourced from the master remote), but not creating
corresponding working sets. This let users attach to new branches with
`use db/feature` syntax, but prevented users from
`dolt_checkout('feature')` both for the bug in dolt checkout, but also
because we created an intermediary state with a local branch, but
without a corresponding working set. SQL replication will now fetch,
create the new branch, and create the appropriate working set.

* jennifer's feedback

* unskip
2022-07-12 14:17:28 -07:00
Aaron Son 178c3bf2b5 go: doltdb: commit_hooks,hooksdatabase: Ensure that replication hooks work for replicating HEADs when the commit itself is already on the remote. 2022-07-08 11:33:46 -07:00
Dhruv Sringari 24cac1f6c4 [no-release-notes] unskip more bats (#3737)
* Fix merge test for __DOLT_1__

* tree.ThreeWayMerge always calls callback when key collision occurs

* add keyless support for new format merge and other merge related tables

* newline of doom

* unskip more bats

* empty

* skip blame.bats and remotes-aws.bats
2022-07-01 17:00:42 -07:00
Daylon Wilkins 178259995b Unskipped bats tests 2022-06-21 04:03:39 -07:00
Maximilian Hoffman 4a994d31c6 Read replica update working set fixes (#3471)
* Read replica update working set fixes

Read replica pull updates the session working set after pulling a
filtered set of branches from the tracking database. The original
(buggy) implementation updates the working set to the branch specified
at server-start time. The identity of that branch was fixed for the
duration of the server, so dolt_checkout would (appear to) have no
effect on the new branch's working set. What actually happened was more
pernicious: the working set was updated to the value of the
incorrect branch.

The fix no longer statically sets the active branch for a read replica
database. The active branch is pulled from the `*sql.Context`, so the
correct working set will be updated.

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* add test

* todos

* delete extraeneous line

Co-authored-by: max-hoffman <max-hoffman@users.noreply.github.com>
2022-05-23 15:00:11 -07:00
Andy Arthur d5bd000790 more BATS skipping for __DOLT_1__ 2022-05-17 11:44:11 -07:00
James Cor 7a3c80f7a9 removing todo and fixing test 2022-04-13 13:52:47 -07:00
James Cor 6cff16b348 adding a test 2022-04-12 13:30:16 -07:00
Andy Arthur 3bf3cf43c8 unified merge() and dolt_merge(), updated BATS to stage changes before committing, some sql-server tests still failing 2022-03-14 11:46:06 -07:00
Maximilian Hoffman 0cbd7e780c Async push for replication (#2369)
* starter code

* async replication prototype

* benchmark async push

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* delete hanging line

* add comment for concurrent map access

* move replication into databaseProvider

* top-level wait group for async threads

* fix testsg

* standardize wg format

* missing header

* address data race

* make async test shorter

* delete unnecessary lines

* starter code for new sqlEngine interface

* new engine interface with background threads

* brian's comments

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* missed a datarace

* fix bats

* bad go.sum

* add latest GMS commit

* bump GMS

Co-authored-by: max-hoffman <max-hoffman@users.noreply.github.com>
2021-12-11 13:41:58 -08:00
Maximilian Hoffman 45a3374808 DOLT_REPLICATE_HEADs_STRATEGY option, fetch all branches from remote before every read (#2346)
* save version with defaultConfig in session -- todo remove and make PersistedSession

* add PersistedSession, and DoltSession interface to switch

* fmt

* fixup test for persistedSession

* PersistedSession progress, interfaces and enginetests

* All DsessFromSess references fixed

* delete unnuecessary lines

* Comments for dolt session

* Dolt session tests

* save bats progress

* formatting

* GMS version

* Bump GMS version

* add license header

* fix multi-db errors

* try copyright again

* make test deterministic

* higher connection timeout default

* sever config ordering

* PR fixes

* PR fixes

* config rewrite progress

* server config refactor

* PR cleanup

* delete lines

* add go tests, refactor persistent global initialization

* PR fixes

* delete more lines

* small PR cleanup

* try to fix data race

* better constructor names

* zach don't like --server flag

* edge cases around disabling replication

* missed line in merge

* Skip two tests i fixed in follow-on PR

* commit hook warning progress

* merge two config prs

* fix nondeterministic bats

* zach's comments

* bump bats sql-shell expect timeout

* non-server progress

* replication warnings

* Get and set config bool types

* More bats, small PR fixes

* missing header

* fix bats panic

* PR touch ups

* flag, logic and tests for pulling multiple heads to replica

* change name because enginetest, fix impl typo

* missed file commit, config var name changed to strategy

* delete unnecessary lines

* progress, need to merge other PR

* prototype working

* delete some lines

* don't make a new branch until after fetch

* zach's comments

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* fix import cycle

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* fix import cycle, sys vars in sqle

* style fixes, one more bats

* split heads spec and all heads into two variables

* remove unnecessary rebase

* bad merge duplicated bats

* fix on demand fetch after multi head pull changes

* correctness, bats and zach fixes

* drop unnecessary server test

* bad bats title

* small PR fixes

* correctness fix for detached head / fetch branch

* fix comment

Co-authored-by: max-hoffman <max-hoffman@users.noreply.github.com>
2021-11-09 10:37:09 -08:00
Max Hoffman faf2e45f91 merge replication warnings PR 2021-11-08 10:26:21 -08:00
Maximilian Hoffman d11f5ec41b default session implements persistable interface, can save SQL variables (#2270)
* save version with defaultConfig in session -- todo remove and make PersistedSession

* add PersistedSession, and DoltSession interface to switch

* fmt

* fixup test for persistedSession

* PersistedSession progress, interfaces and enginetests

* All DsessFromSess references fixed

* delete unnuecessary lines

* Comments for dolt session

* Dolt session tests

* save bats progress

* formatting

* GMS version

* Bump GMS version

* add license header

* fix multi-db errors

* try copyright again

* make test deterministic

* higher connection timeout default

* sever config ordering

* PR fixes

* PR fixes

* config rewrite progress

* server config refactor

* PR cleanup

* delete lines

* add go tests, refactor persistent global initialization

* PR fixes

* delete more lines

* small PR cleanup

* try to fix data race

* better constructor names

* zach don't like --server flag

* edge cases around disabling replication

* missed line in merge

* Skip two tests i fixed in follow-on PR

* merge two config prs

* fix nondeterministic bats

* zach's comments

* bump bats sql-shell expect timeout

* GMS and integrator race to access system vars; need to decide whether to force integrator to initialize, or make GMS init private; leaving in GMS for now
2021-11-01 16:49:00 -07:00
Maximilian Hoffman f12196df03 dolt_branches INSERT/UPDATE commit replication (#2301)
* branch table replicates

* Change param type
2021-10-28 15:28:31 -07:00
Maximilian Hoffman 9a7a49f516 initial dolt backup cli (#2208)
* prototype

* backup pushes noms root

* backup pushes root value, basic restore still needs origin drop

* bats tests, no origin on restore, only head ref on restore

* cleanup code, up to date bats test

* make backup restore a root sync also, to catch branches and remote refs

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* couple of tim's comments

couple of tim's comments

* zach's comments

* loosen restrictions for conflicting remote addresses

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

Co-authored-by: max-hoffman <max-hoffman@users.noreply.github.com>
2021-10-04 11:53:05 -07:00
Maximilian Hoffman 8a5f3f54be config, clone, backups refactor (#2196)
* shittiest read replica imaginable is kind of working

* import cycle progress

* delete unecessary files and fix  db type switch bug

* Add bats test

* delete comments

* fix working set updates for cli

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* clean comments

* comment racy server test

* move env variables to local dolt config

* refactor clone, fix multienv test

* cleanup comments

* missing copyright

* brian's comments, add compile time checks for config interfaces

* format

* fix windows filepaths issue

* file:/// with three slashes should work on windows

* more windows problems

* three slashes didn't work for clone, do chdir to ref local dolt db

Co-authored-by: max-hoffman <max-hoffman@users.noreply.github.com>
2021-09-29 19:10:37 -07:00
Maximilian Hoffman a0f6799a2c FF head on read replica (#2182)
* shittiest read replica imaginable is kind of working

* import cycle progress

* delete unecessary files and fix  db type switch bug

* Add bats test

* delete comments

* fix working set updates for cli

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* clean comments

* comment racy server test

* zach's comments

* add copyright headers

Co-authored-by: max-hoffman <max-hoffman@users.noreply.github.com>
2021-09-29 10:23:07 -07:00
Maximilian Hoffman b0a75ccbc0 replicate HEAD on commit (#2151)
* save progress

* progress before stash  pop

* hacky prototype

* bad merge

* simple bats

* delete unnecessary code

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* format

* refactor commitHook command structure

* progress commit, local testing works

* Fix head update issue

* re: aaron, fix ctx cleanup

* commit hooks recieve error handler from user facing packages

* Add storage level replication hook test

* Simplify replication parameters

* Add bats test for server commit replication

* rename backup key, was missing the stop progress channel call

* small fixes

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* Attach server Logger.Out to commit hooks

* formatting

* zach's comments

* missed sql-server test

* accidentally commented bats

Co-authored-by: max-hoffman <max-hoffman@users.noreply.github.com>
2021-09-23 11:15:36 -07:00