Noms SDK users frequently shoot themselves in the foot because they're
holding onto an "old" Database object. That is, they have a Database
tucked away in some internal state, they call Commit() on it, and
don't replace the object in their internal state with the new Database
returned from Commit.
This PR changes the Database and Dataset Go API to be in line with the
proposal in Issue #2589. JS follows in a separate patch.
* Implement noms cli configuration support
- Introduce .nomsconfig
- Supports a default db to use when no explicit db is given
- Supports defining db aliases to use as short hand for db urls
- See samples/cli/nomsconfig for more info
fix: #2131
* Use sampling for a better bytes-written estimate for noms sync
* Confirmed that remaining overestimate of data written is consistent with leveldb stats and opened #2567 to track
These were previously intertwined into one writer that was
embedded in and only usable by the 'noms' command.
This commit separates them into to separate writers that
can be used independently or combined. I also moved them
into go/utils/writers so they can be used by other code.
The main impetus to do this was to fix Bug #2593.
* Add "noms commit" command
* Updated csv-import, json-import, xml-import and url-fetch to (optionally) not commit results
* Added helpers for creating commit meta-data struct through command line or function calls
Fixes are based on Go report card output:
- `gofmt -s` eliminates some duplication in struct/slice initialization
- `golint` found some issues like: `warning: should drop = nil from declaration of var XXX; it is the zero value`
- `golint` found some issues like: `warning: receiver name XXX should be consistent with previous receiver name YYY for ZZZ`
- `golint` says not to use underscores for function/variable names
- `golint` found several issues like: `warning: if block ends with a return statement, so drop this else and outdent its block`
No functional changes are included - just source code quality improvements.
fix misspellings; fix code that was not gofmt'd - plus take advantage of gofmt -s too; couple of unreachable golint reported fixes; reference go report card results and tests
This isn't triggering any problem in particular, I just noticed it. All
it meant was that noms log might take unnecessarily long with list diff
when there are a lot of changes.
In 7c103344 I changed noms log to use left-right diff so that it
completes faster, but it changed noms diff to use left-right as well.
Noms diff is supposed to use best diff.
* Fix noms_sync status message to indicate when new dataset is created
Originally, syncing a commit to a non-existent dataset and an already sync'ed dataset resulted in the same message: "<src> is up to date"
This fix distinguishes 2 cases:
- Syncing to a new dataset prints "<dest> created"
- Syncing to an identical dataset prints "<dest> already up to date"
Resolves: #2053
* Make message when noms sync creates new dataset more fun
Addresses: #2053
The Dataset.Commit() code pathway still enforces fast-forward-only
behavior, but a new SetHead() method allows the HEAD of a Dataset to
be forced to any other Commit.
noms sync detects the case where the source Commit is not a descendent
of the provided sink Dataset's HEAD and uses the new API to force the
sink to the desired new Commit, printing out the now-abandoned old
HEAD.
Fixes#2240
Previously there were 3 ways to construct a Path: via methods like
AddIndex, via parsing/AddPath, and simply via using append() (since Path
is a slice).
This patch deletes the former and leaves only ParsePath and append(),
with the ability to manually create the various path parts (I've made
them public). I also added documentation for the public types/fields.
It (a) had a spurious blank line, (b) should print the dataset path not
just its ID, (c) should include the # for the hash. Now it says:
```
> noms ds -d path/to/db::my-dataset
Deleted path/to/db::my-dataset (was #lvnbmpj94r9h3nuaj1t0m4jeqp25o1ve)
>
```
This fixes https://github.com/attic-labs/noms/issues/2235 where set/map
diff was still running when noms log has finished, and already closed
its database. Commonly this would crash.
I've removed panic-based control flow to make error handling explicit,
and added a "DiffLeftRight" method to set/map so that noms log completes
ASAP. See the issue for details.
For Blobs we print:
```
- Blob (42 kB)
+ Blob (1 B)
```
For Refs we just print the hashes:
```
- abcdeabcdeabcdeabcde
+ defghdefghdefghdefgh
```
Fixes#2213
Earlier I'd used 2 channels and selected over them, but as I found
elsewhere, this doesn't scale very well. It's simpler to just use a
close channel with a buffer size of 1.
Shows number of changes between two top level values.
```
noms diff -summarize $l::#t5p4im6uug7n5m72frr0dnjnkm04e4ph.value $l::#ueo0utduuqsf9vrntrhn25lnc19m848l.value
```
Prints:
```
13,636 insertions, 0 deletions, 107 changes, (1,919,894 values vs 1,933,530 values)
```
Where the numbers are updated as more data is computed from the diff.
Towards #2031
- Too have same API as all the other diff methods
- Send changes to channel without intermediary slices and without the
need to union and sort the fields
Previously a header was only printed once per entire map. This led to a
confusing diff where if a map like:
```
{
"a": {
"b": 1
}
}
```
changed to:
```
{
"a": {
"b": 2
},
"c": {
"d": 3
}
}
```
then the diff would be:
```
/["a"]
- "b": 1
+ "b": 2
+ "c": {
+ "d": 3
+ }
```
which makes it look like the "c" change is part of the ["a"]["b"] one.
After this change, the diff will be:
```
/["a"]
- "b": 1
+ "b": 2
/
+ "c": {
+ "d": 3
+ }
```
I also fixed up a bit of the diff formatting in this change.
This changes so that all commit struct types have a meta field
(which might be an empty struct).
Increment the serialization version since the old data does not
necessarily have the meta field.
Fixes ##2112
Currently they return immediately, but run internal goroutines which
stream results back on a channel. Now they must be run on their own
goroutine from outside the function.
This makes the code easier to reason about, and a future patch to write
a top-down and left-right multiplexing diff easier to write.
