demo-server was using a read-through cache to allow it to serve more
concurrent requests more quickly by reducing disk I/O. As seen in issue
across Databases, leading to incorrect sync results in some instances.
Since we're not worried about demo-server load right now, simply delete
the cache.
Fixes#2688
The subsequent runs of url-fetch on jenkins are way faster, and this
appears to be because commiting is much faster on subsequent runs. The
perf tests now use a new database each time.
This patch implements evolving support for configuring aliases and defaults for the noms cli (started with #2131)
For an introduction, please take a look at the sample code here: https://github.com/attic-labs/noms/blob/master/samples/cli/nomsconfig/README.md
Improvements include:
- All go samples now work with .nomsconfig
- Absolute paths in ldb specs are now properly handled
- Add -v|--verbose flag to commands to debug expansion
- Make default just another alias and change [default] section to [db.default]
- Introduce the `.` shorthand to refer to a previously mentioned dataset/object
This just involves changing types.NewBlob(io.MultiReader(files...)) to
types.NewBlob(files...). On my laptop it improves
Test01ImportSfCrimeBlobFromTestdata from 21s to 16s - though much of
this is dominated by commit, which wouldn't be affected by this change.
Blob.Concat is a simple use of the sequence concat code that List.Concat uses.
NewBlob uses Blob.Concat to construct a Blob in parallel.
Perf tests for parallel NewBlob write N temporary files then constructs a Blob
from them, so there is some I/O, but it appears to be mostly CPU bound. NewBlob
doesn't get much more than 50% faster with any P >= 2.
Noms SDK users frequently shoot themselves in the foot because they're
holding onto an "old" Database object. That is, they have a Database
tucked away in some internal state, they call Commit() on it, and
don't replace the object in their internal state with the new Database
returned from Commit.
This PR changes the Database and Dataset Go API to be in line with the
proposal in Issue #2589. JS follows in a separate patch.
* Add "noms commit" command
* Updated csv-import, json-import, xml-import and url-fetch to (optionally) not commit results
* Added helpers for creating commit meta-data struct through command line or function calls
This patch modifies merge.ThreeWay() to take a callback that allows
for custom conflict resolution. The noms-merge command-line tool uses
this to inject a callback that accepts input from the console
dictating whether to accept the value from the 'left' or 'right' merge
candidates.
Toward #2445
Fixes are based on Go report card output:
- `gofmt -s` eliminates some duplication in struct/slice initialization
- `golint` found some issues like: `warning: should drop = nil from declaration of var XXX; it is the zero value`
- `golint` found some issues like: `warning: receiver name XXX should be consistent with previous receiver name YYY for ZZZ`
- `golint` says not to use underscores for function/variable names
- `golint` found several issues like: `warning: if block ends with a return statement, so drop this else and outdent its block`
No functional changes are included - just source code quality improvements.
Currently we only have a perf test for CSV list import, which uses the
sf-crime dataset. This test uses the 43MB sf-registered-businesses
dataset instead, since sf-crime is too slow. Which is ironic, since we
normally parse sf-crime into a map.
I've also tightened up some of the other perf tests.
- Fixed a bug where Database was shared between runs.
- Make the pure CSV parsing test use a smaller dataset, it doesn't need
to use something as large as ny-vehicle-registrations.
This is a first pass at exposing the new merge package to users. The
tool is very basic, and currently only works on datasets in the same
database. It requires the 'parent' (i.e. a common ancestor of the two
datasets being merged) to be provided by a commandline option; a
follow-on patch will make the code discover this ancestor automatically.
Toward #2445
fix misspellings; fix code that was not gofmt'd - plus take advantage of gofmt -s too; couple of unreachable golint reported fixes; reference go report card results and tests
Aaron Boodman