Commit Graph

163 Commits

Author SHA1 Message Date
Aaron Son
edf0485148 go/libraries/utils/pipeline/stage.go: Fix how we finalize a stage so that every consumer can still see nil in the success case. 2022-05-18 14:31:09 -07:00
Aaron Son
ec7af0c507 go/libraries/utils/pipeline: Fix unit test for new semantics. 2022-05-18 14:07:14 -07:00
Aaron Son
a8f5e44786 go/libraries/utils/pipeline: stage.go: Another pass at finalization strategy. 2022-05-18 12:17:35 -07:00
Aaron Son
f4b6b6fe50 go/libraries/utils/pipeline: Fix racey finalization of a pipeline.
Before this change, when an initial stage function returned a non-nil err, it
would return non-nil, resulting in closing its output channel and leaving the
errgroup responsible for canceling the egCtx.

The next transform stage would read `nil, false` on its inCh. That stage would
forward the `nil` down the transform pipeline. If the inCh selects all resolved
down the transform pipeline before any of the `<- ctx.Done()` selects, then the
pipeline would see the exact same behavior for the error case as it would see
for an `io.EOF` being reached in the first stage.

Instead, transform stages in the pipeline should abort without forwarding data
if they see a closed input channel. In turn, input stages should forward one
last `nil` sentinel value to their output channel when the see an `io.EOF`
before they close the channel.
2022-05-18 10:11:27 -07:00
Dhruv Sringari
c0c54db5d2 Fix bad upload table file retries 2022-03-25 14:09:46 -07:00
James Cor
0cadc528e5 renamed WriteNoNewLine to WriteWithoutNewLine 2022-03-02 12:16:56 -08:00
James Cor
88b6a6f197 working for sql format; not sure if i need any other formats 2022-03-01 15:42:33 -08:00
Lars Lehtonen
bf727113cc libraries/utils/editor: fix dropped error 2022-02-21 18:54:48 -08:00
Dhruv Sringari
009bb85d8d fix bats 2022-02-17 14:22:46 -08:00
Dhruv Sringari
c662e2fa8c self-review 2022-02-16 16:29:00 -08:00
Dhruv Sringari
a18bfc6c27 wip 2022-02-16 14:52:34 -08:00
Zach Musgrave
576753d38c Added failsafe values to dolt CLI config, only in sql context for now 2021-11-08 12:26:52 -08:00
Zach Musgrave
c76f4bde26 Fixed bug in config hierarchy construction -- it was being constructed backwards, so globals were being returned instead of locals 2021-11-08 11:43:13 -08:00
Zach Musgrave
37c2137557 merge main 2021-11-04 14:12:38 -07:00
Zach Musgrave
4a236171e7 More clean up of session setup 2021-11-04 09:51:30 -07:00
jennifersp
efef83b167 dolt dump: Add dolt dump command and bats tests. 2021-11-03 15:21:46 -07:00
Zach Musgrave
92f175a37a Merge branch 'main' into zachmu/create-db 2021-11-02 10:46:50 -07:00
Zach Musgrave
037f249e91 First pass at getting something working for create db 2021-11-02 10:46:12 -07:00
Maximilian Hoffman
d11f5ec41b default session implements persistable interface, can save SQL variables (#2270)
* save version with defaultConfig in session -- todo remove and make PersistedSession

* add PersistedSession, and DoltSession interface to switch

* fmt

* fixup test for persistedSession

* PersistedSession progress, interfaces and enginetests

* All DsessFromSess references fixed

* delete unnuecessary lines

* Comments for dolt session

* Dolt session tests

* save bats progress

* formatting

* GMS version

* Bump GMS version

* add license header

* fix multi-db errors

* try copyright again

* make test deterministic

* higher connection timeout default

* sever config ordering

* PR fixes

* PR fixes

* config rewrite progress

* server config refactor

* PR cleanup

* delete lines

* add go tests, refactor persistent global initialization

* PR fixes

* delete more lines

* small PR cleanup

* try to fix data race

* better constructor names

* zach don't like --server flag

* edge cases around disabling replication

* missed line in merge

* Skip two tests i fixed in follow-on PR

* merge two config prs

* fix nondeterministic bats

* zach's comments

* bump bats sql-shell expect timeout

* GMS and integrator race to access system vars; need to decide whether to force integrator to initialize, or make GMS init private; leaving in GMS for now
2021-11-01 16:49:00 -07:00
Aaron Son
86307b6f8e Merge pull request #2230 from abmyii/arm-dolt-push
Fixes for dolt on arm devices
2021-10-07 09:05:55 -07:00
Aaron Son
97ea3b8749 go/libraries/utils/iohelp: ReaderWithStats: Support > 4GB read stats. 2021-10-06 14:39:47 -07:00
Abdurrahmaan Iqbal
7e760c4e31 Fix for dolt push on arm devices 2021-10-06 21:04:50 +01:00
Zach Musgrave
23df5c88fe Merge pull request #2211 from Juneezee/deprecate-ioutil
refactor: move from io/ioutil to io and os package
2021-10-04 14:10:04 -07:00
Maximilian Hoffman
9a7a49f516 initial dolt backup cli (#2208)
* prototype

* backup pushes noms root

* backup pushes root value, basic restore still needs origin drop

* bats tests, no origin on restore, only head ref on restore

* cleanup code, up to date bats test

* make backup restore a root sync also, to catch branches and remote refs

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* couple of tim's comments

couple of tim's comments

* zach's comments

* loosen restrictions for conflicting remote addresses

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

Co-authored-by: max-hoffman <max-hoffman@users.noreply.github.com>
2021-10-04 11:53:05 -07:00
Eng Zer Jun
8164a625c7 refactor: move from io/ioutil to io and os package
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-10-05 01:40:59 +08:00
Maximilian Hoffman
8a5f3f54be config, clone, backups refactor (#2196)
* shittiest read replica imaginable is kind of working

* import cycle progress

* delete unecessary files and fix  db type switch bug

* Add bats test

* delete comments

* fix working set updates for cli

* [ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/update.sh

* clean comments

* comment racy server test

* move env variables to local dolt config

* refactor clone, fix multienv test

* cleanup comments

* missing copyright

* brian's comments, add compile time checks for config interfaces

* format

* fix windows filepaths issue

* file:/// with three slashes should work on windows

* more windows problems

* three slashes didn't work for clone, do chdir to ref local dolt db

Co-authored-by: max-hoffman <max-hoffman@users.noreply.github.com>
2021-09-29 19:10:37 -07:00
Brian Hendriks
3070d3af9f merge rows on dolt table import -u when import data only represents a subset of the columns (#2105) 2021-09-16 14:06:45 -07:00
Brian Hendriks
775ac0af51 Import Performance and Garbage Generation improvements. 2021-09-08 14:52:42 -07:00
Maximilian Hoffman
f5d63f4e3d move common merge/push/pull code into doltcore/env (#2063)
* Start moving code out of cmd

* hacky cmd -> action/evn refactor

* progress

* String verbose error from lib, mostly not reconstructed on cli side

* Missing verbose errors in lib, still not handled in cli. Refactor vinai's doc diffs to not depend on merge

* Small progress commit on other laptop

* Merge bats passing

* format

* progress

* Remotes bats tests working

* More error fixes

* Fix cyclic foreign keys bats

* Fix import cycle

* Start cleaning up code

* Fix a couple bats

* Clean up comments and println

* Brian's comments
2021-08-31 15:23:23 -07:00
Brian Hendriks
0a0ce2181c Bh/gen cs (#1938) 2021-08-05 18:20:19 -07:00
Daylon Wilkins
1d33bde91b Reworked verify-constraints and renamed to constraints verify 2021-07-20 18:16:38 -07:00
Daylon Wilkins
8d971dcd35 Added fast forward handling and fleshed out constraint violation messages 2021-07-15 17:16:16 -07:00
Daylon Wilkins
6b2b68923b Further attempts to fix ACCESS_DENIED error on Windows 2021-07-15 13:26:59 -07:00
Zach Musgrave
475a6c4787 Merge pull request #1890 from dolthub/zachmu/tx-enabled
Enable SQL transactions by default, and begin writing working set to database instead of repo_state.json file
2021-07-06 18:12:47 -07:00
Brian Hendriks
88fd93e7cc Bh/push pull temptf (#1885) 2021-07-06 17:09:16 -07:00
Zach Musgrave
e78ed3b263 Merge master 2021-07-04 09:30:24 -07:00
Brian Hendriks
86bd20a4cd dolt roots command (#1891) 2021-07-04 07:51:07 -07:00
Zach Musgrave
d8dfd68383 Migrating workingset from existing repo state file on load 2021-06-23 17:33:40 -07:00
Aaron Son
9b763af5a6 libraries/utils/async/ring_buffer.go: PR feedback. Return error on epoch mismatch. 2021-03-24 15:45:58 -07:00
Aaron Son
b77f2bad8c go/libraries/doltcore/sqle/async_indexed_lookups.go: Fix leaking of results from one index lookup to another.
When a query with an index lookup as a LIMIT, it's possible for the result
iterator to be closed before all the index results have been streamed. When
that happens, we can end up with queueRows goroutine which is enqueuing rows
into a ring buffer that is going to be used by a different query later.

This PR adds two mechanisms, one to stop the wrong results from being seen and
to cancel the unnecessary work.

1. RingBuffer gets an epoch, which is incremented on each Reset(). If a given
   Push does not match the current epoch, then nothing is added to the ring
   buffer.

2. queueRows is run with a cancelable (sub-)Context, and that Context is
   canceled when the corresponding iterator is Closed().
2021-03-24 14:11:09 -07:00
Brian Hendriks
c8264ef610 add file copy helper method to iohelp package (#1403) 2021-03-09 16:46:50 -08:00
Brian Hendriks
04d189937e chunk store in a file no manifest needed (#1401) 2021-03-09 16:46:08 -08:00
Brian Hendriks
69de83a757 Fixed worker pool used for index lookups (#1304) 2021-02-16 17:28:20 -08:00
Aaron Son
2ed61f501f go/libraries/utils/tracing: Add pluggable context-aware Tracing function. 2021-01-07 11:59:18 -08:00
Aaron Son
a1bf8b62fd repofmt: Hygiene. 2021-01-06 15:02:24 -08:00
Aaron Son
a654f02e14 go/cmd/dolt: Add debugging flag to run with Jaeger span reporting to localhost.
This adds a --jaeger flag to dolt CLI which installs a Jaeger Tracer as the
global opentracing Tracer. The Tracer is configured to report to an HTTP
collector running on http://localhost:14268, which is the port that docker
image `jaegertracing/all-in-one` listens on.

Also adds some parameters in places where Dolt constructs sql.Contexts to pass
the correct Tracer through.

Also adds a few new Span points, in things like nbs.Get and
metaSequenceImpl.getChildSequence.
2021-01-06 14:50:06 -08:00
Andy Arthur
b635b72e7b match keyless tables by column intersection, ddl diff bats 2020-12-17 11:54:00 -08:00
Brian Hendriks
602f0ae2c0 partition ranges, covering indexes, smarter iterators (#1116) 2020-12-15 18:44:57 -08:00
Brian Hendriks
ffab301d8c PR feedback 2020-11-24 15:39:54 -08:00
Brian Hendriks
d26d82065b Merge remote-tracking branch 'remotes/origin/master' into bh/pipelinev2 2020-11-05 11:43:18 -08:00