Commit Graph

76 Commits

Author SHA1 Message Date
Aaron Son 8021bc5f02 sqle: cluster: Set the engine to read-only when a replica is in standby mode. Set it back to read-write when it becomes primary.
This prevents standby replicas from running some DDL which they were previously
erroneously allowed to run, including CREATE USER, GRANT, CREATE DATABASE and
DROP DATABASE.
2023-09-01 10:47:23 -07:00
Aaron Son 48cf308d05 Merge pull request #6454 from dolthub/aaron/cluster-replication-users-and-grants
go/libraries/doltcore/sqle/cluster: Add replication of user and grant changes from primary to standby replicas.
2023-08-22 13:28:35 -07:00
Aaron Son 6d7c73575d dolt sql-server: Add a system_variables: key to config.yaml, which allows setting global system variables.
For example, this can be used to set `secure_file_priv` when the server starts
up. That variable is non-dynamic, so it cannot be set with `SET @@GLOBAL...`
2023-08-15 15:53:18 -07:00
Aaron Son 08f91f74a0 go/libraries/doltcore/sqle/cluster: Add replication of user and grant changes from primary to standby replicas. 2023-08-04 11:33:57 -07:00
Stephanie You d178f0dd6f update go-sql-driver for integration-tests 2023-07-23 17:10:11 -07:00
Aaron Son 3f9e8d6995 integration-tests/go-sql-server-driver: go.mod: go mod tidy. 2023-07-13 16:11:05 -07:00
Aaron Son fab46e22ed integration-tests/go-sql-server-driver/go.mod: go mod tidy. 2023-07-10 14:40:36 -07:00
Aaron Son c0b0cc42d1 dolt clone: Fix dolt clone run against a sql-server where the database has been GCd.
A long-standing bug in the remotesapi which the sql-server exposes could cause
a `dolt clone` to fail when running against a database which had been garbage
collected. This change fixes the bug in the server. It also patches the client
behavior so that it will tolerate responses from older versions of dolt.
2023-06-27 16:41:22 -07:00
Aaron Son 6314e27de9 Merge pull request #6204 from dolthub/aaron/cluster-assume-role-standby-changes
sqle: cluster: Add dolt_cluster_transition_to_standby stored procedure.
2023-06-22 14:03:13 -07:00
Aaron Son 155884711e sqle: cluster: dolt_cluster_transition_to_standby: Add logic to respect and enforce minCaughtUpReplicas. 2023-06-21 16:29:45 -07:00
Aaron Son 0a7cd7e033 sqle: cluster: dolt_cluster_transition_to_standby: Register the implementation (without transition transition thresholds) and test it. 2023-06-21 16:03:59 -07:00
Zach Musgrave 0b52f6c424 Revert "Revert "Merge pull request #5968 from dolthub/zachmu/tx-6""
This reverts commit bb03b8cb25.
2023-06-08 11:09:54 -07:00
Zach Musgrave bb03b8cb25 Revert "Merge pull request #5968 from dolthub/zachmu/tx-6"
This reverts commit 51c2129e51, reversing
changes made to 9692f6ef92.
2023-06-07 17:27:06 -07:00
Zach Musgrave e672c304ab Merge main 2023-06-06 15:17:07 -07:00
Taylor Bantle ea2e87eac8 Fix go-sql-server-driver tests 2023-06-05 13:39:58 -07:00
Taylor Bantle ed8f23481c Integration tests for load data ignore/replace 2023-06-05 13:23:35 -07:00
Zach Musgrave bff42f81fc Merge main 2023-06-02 13:08:13 -07:00
Maximilian Hoffman 0bfd57ae4d [no-release-notes] Pushdown refactor, deprecate sql.FilteredTable, ColumnDiffTable correctness (#6024)
* [no-release-notes] Pushdown refactor, deprecate sql.FilteredTable,
ColumnDiffTable correctness

Pushdown splitting into two phases uncovered bugs. sql.FilteredTable
is now deprecated. ColumnDiffTable correctness tests were never run
through the indexing path before, which are broken. This disables the
indexing path and runs the proper test suites.

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* bump

* tidy

* missing interface method

* mod tidy

* formatter

---------

Co-authored-by: max-hoffman <max-hoffman@users.noreply.github.com>
2023-05-24 16:32:20 -07:00
Zach Musgrave 9b2cd87c57 Added a README 2023-05-22 13:49:32 -07:00
Zach Musgrave ae2b616add Added debug timeouts and a run single test method 2023-05-22 11:41:27 -07:00
Zach Musgrave a46ebe1f9f Added a debug port option to launch dlv, and fixed problems with graceful shutdown on windows 2023-05-19 18:38:38 -07:00
Aaron Son 6ea915ca4d Merge remote-tracking branch 'origin/main' into aaron/block-on-cluster-replication 2023-05-16 16:57:09 -07:00
Aaron Son 6ccae75a8d integration-tests/go-sql-server-driver: Add TestCluster/last_updated_heartbeats test which asserts that last_updated does not grow while there are no writes to replicate. 2023-05-16 14:21:41 -07:00
Aaron Son 535e59141d integration-tests/go-sql-server-driver: Cluster, dolt_cluster_ack_writes_timeout_secs, add a failing test to assert that dolt_branch modifications are waited on. 2023-05-15 14:00:04 -07:00
Aaron Son dfdb4a4b2a go: sqle: cluster: Implement a first pass at dolt_cluster_ack_writes_timeout_secs.
Setting this system variable to a non-zero value on a primary replica in a
sql-server cluster will cause dolt to block a SQL client performing a commit
until that client's commit is fully replicated to the replicas.

If there is a timeout, currently a warning is logged in the logs.
2023-05-11 15:50:43 -07:00
Aaron Son 2674c3cf2a sql-server: Allow dolt_log_level system variable to receive writes and dynamically update the log level. 2023-05-10 14:46:45 -07:00
Aaron Son 663d2e3181 sql-server: Expose the current server log level through the dolt_log_level system variable. 2023-05-10 14:25:41 -07:00
Aaron Son 6e6ed1b77b Fix result ordering in integration test. 2023-05-09 14:08:19 -07:00
Aaron Son acf99f1edf go: sqle: dolt_gc.go: Disable CALL dolt_gc() on a standby replica, where it is not currently safe. Shallow GC is still available. 2023-05-09 13:41:16 -07:00
Zach Musgrave 75b1395510 New integration test ala aaron 2023-05-05 07:42:32 -07:00
Aaron Son 1d0a6001ac go/libraries/doltcore/sqle: database_provider.go: When cloning a remote into a sql-server, always run the InitDatabaseHook on it.
Failing to run the hook means that the cloned database is not appropriately
configured for replication, for example.

Fixes: #5850
2023-05-01 12:42:02 -07:00
Aaron Son b51bdc2cff integration-tests/go-sql-server-driver: go mod tidy. 2023-04-06 16:12:20 -07:00
Zach Musgrave bd0dc834b4 Better failure message for test failure 2023-03-30 11:51:07 -07:00
Aaron Son d1bb4f6807 integration-tests/go-sql-server-driver: Add a test which fails when a non-chunk journal repo first switches to chunk journal and runs call dolt_gc. 2023-03-24 09:43:54 -07:00
Aaron Son d024ea8ea6 go/store/nbs: store.go: Conjoin if we need to when we add new table files to a manifest.
This allows us to avoid huge manifests with lots of table files when we run
multiple full GCs over a period of time.
2023-03-20 16:39:52 -07:00
Aaron Son 7105650970 integration-tests/go-sql-server-driver: concurrent_gc_test.go: Small change to always remove dolt_gc connection from the connection pool. 2023-03-20 11:29:35 -07:00
Aaron Son 774c100678 go/store/types: gc: Establish safepoint after we are finalizing but before we swap tables. 2023-03-17 15:40:01 -07:00
Aaron Son 622b65264d integration-tests/go-sql-server-driver: Add a concurrent gc test where we create dolt commits. 2023-03-17 12:28:03 -07:00
Aaron Son cf4b81217a dolt_gc support: Make waitForGC unblock with error on context canceled. Use more agressive retry-with-backoff policy on awaiting process quiesence in dolt_gc. 2023-03-17 11:48:18 -07:00
Aaron Son b9fbf6d733 integration-tests/go-sql-server-driver: concurrent_gc_test.go: Add an initial test case. 2023-03-16 16:12:48 -07:00
Aaron Son 20aef1ab89 integration-tests/go-sql-server-driver: Add some cleanup for the tmpdirs created as part of running these tests. 2023-03-14 11:58:30 -07:00
Zach Musgrave c02ec6dd50 Fixed go.mod problems in integration test 2023-02-20 09:41:08 -08:00
Zach Musgrave 5f5dae3ccf Squash merge from zachmu/session-transaction to get rid of spurious test files 2022-12-21 13:12:31 -08:00
Dustin Brown baedb70771 [auto-bump] [no-release-notes] dependency by max-hoffman (#4888)
* [ga-bump-dep] Bump dependency in Dolt by max-hoffman

* go mod update

Co-authored-by: max-hoffman <maximilian.wolfgang1@gmail.com>
Co-authored-by: Max Hoffman <max@dolthub.com>
2022-11-29 19:52:18 -08:00
Aaron Son f7c69bf881 Merge pull request #4749 from dolthub/aaron/cluster-jwks-auth
go/libraries/doltcore/sqle/cluster: Add JWT authentication for peer communication to cluster replicas.
2022-11-15 12:21:55 -08:00
Aaron Son 6d7a569e61 integration-tests/go-sql-server-driver: Use 385{1,2} instead of 5005{1,2} for cluster remotesapi ports. 2022-11-10 16:34:49 -08:00
Maximilian Hoffman 08167dfd5d Redo import perf (#4689)
* import perf harness prog

* more test isolation

* rework common dep

* fix server driver

* delete old files, fix readme

* more readme fixes

* batching edits

* edits

* skip import tests by def

* GA starter

* rename

* fix arg name

* change input name again

* remove espaced character from input desc

* fix env definition

* fix env definition

* fix workflow_dispatch name

* CI test files

* remove os spec

* runs on

* different set variable

* defaults

* defaults

* remove local GMS

* edits

* run bench from go/ directory

* dropped def equals

* go run, not go build

* fix output command name

* configure dolt

* fix query file path

* fix query file path

* switch query and inputs

* fix format <<EOF

* reformat yaml

* debug output

* more debug output

* fix echo -> cat

* fix to json flag

* fix yaml spacing

* yaml spacing

* yaml spacing

* revert html spacing

* json format

* env var expansion

* echo not cat

* another json string printing error

* no echo

* log template

* no template string

* wrong parameter, template should be templateName

* remove console.log

* pass file, not string, to ses

* rely on preexisting template? email action interface

* fix yaml

* fix html print

* fix html header

* change ci script

* fix sql-server connection pass

* mysql no password, setup default db

* put password back

* missed port for default database

* expanded CI tests

* shorter test file

* extend ci tests

* change default query

* try to push to dolthub

* modify push to dolthub

* duplicate test names

* typo

* dolt-action version

* invalid param for dolt_action

* specify feature branch

* specify main branch dolt-action

* -b flag first

* dont need -q flag for batch EOF

* combine results and dolthub push

* missing curly brace

* no auth

* new creds

* new cred fun

* use the cred key

* try again

* log table

* move push out of batch

* more logging

* new vs old branch

* fix branch rule

* more test

* all tests

* smaller tests

* smaller tests

* double env: key

* fix yaml error

* more yaml errors

* more yaml errors

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* linting fixes, remove other import nightly

* linting

* go vet

* licenses

* fix compile errorrs

* warn don't panic on init()

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* no external package

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* unused dep

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* Revert perf scripts

* refactor to repo dispatch

Co-authored-by: max-hoffman <max-hoffman@users.noreply.github.com>
2022-11-10 12:56:43 -08:00
Aaron Son b22fbf11f2 go/libraries/doltcore/sqle/cluster: Add JWT authentication for peer communication to cluster replicas.
A remotesapi server running on a cluster replica publishes a JWKS.

Every outbound GRPC call the cluster replica makes includes a JWT signed with a
private key.

remotesapi servers running on cluster replicas require and validate incoming
JWTs for cluster traffic. The set of valid signing keys is taken from the
JWKSes which are published at /.well-known/jwks.json on the standby replica
hosts.

It is possible to configure tls_ca on cluster remotesapi to configure the set
of trusted roots for outbound TLS connections. Because the JWKSes are served
over the same connection, and becuase signed JWTs are not replay resistent, TLS
is recommended for all deployment topologies.
2022-11-10 09:26:38 -08:00
Jason Fulghum a62e12686c [no-release-notes] Merge pull request #4686 from dolthub/fulghum/orm-testing
Prisma compatibility smoke testing
2022-11-08 15:43:29 -08:00
Aaron Son 57523a9323 go/libraries/doltcore/sqle/cluster: Add the ability to generate a keypair, serve a JWKS, and sign outbound RPCs with a JWT. 2022-11-02 09:54:09 -07:00