Commit Graph

2903 Commits

Author SHA1 Message Date
Erik Arvidsson
70004e1699 Let splore use versioned URLs too (#2530) 2016-09-07 18:05:14 -07:00
Erik Arvidsson
3d83e4e72f Allow stage.py to create the staging dir (#2528)
Previously, it was required to already exist
2016-09-07 14:24:16 -07:00
Erik Arvidsson
dbe0556c63 Rename files to their hashes when staging (#2515)
This add two keyword arguments to the GlobCopier:
- rename - if True, will rename the files to include their hash
- index_file - if present, then the paths in this file here will be
  updated to the renamed paths.
2016-09-07 13:34:14 -07:00
cmasone-attic
1c69c6b891 Update merge.ThreeWay() to allow very basic custom conflict resolution (#2505)
This patch modifies merge.ThreeWay() to take a callback that allows
for custom conflict resolution. The noms-merge command-line tool uses
this to inject a callback that accepts input from the console
dictating whether to accept the value from the 'left' or 'right' merge
candidates.

Toward #2445
2016-09-07 13:21:22 -07:00
zcstarr
ef817db179 Adds error for invalid skip records argument and exits csv importer (#2522) 2016-09-06 17:37:05 -07:00
zcstarr
b1c0aeb9c5 Adds checks for bad column-type and header csv-import flag values (#2525) 2016-09-06 17:12:21 -07:00
zcstarr
ef11062cab Fixes bug where delimiter applied to header arguments (#2523) 2016-09-06 16:50:05 -07:00
zcstarr
65131432c7 Fix cmdline help message color argument (#2520) 2016-09-06 14:59:01 -07:00
cmasone-attic
652613d09e Go: Add List.IteratorAt() and use in list merging (#2512)
When merge.ThreeWay() merges list splices, it frequently needs to
extract a slice of a List or check whether a given range of values is
exactly equal in two different Lists. Repeatedly Getting elements from
a List is expensive, because it creates a new cursor internally every
time. Adding IteratorAt(idx uint64) allows code to iterate over a
range of a List while only creating a cursor once. This allows a slice
of Values to be extracted or used in comparisons efficiently.

Toward #2445
2016-09-06 13:50:08 -07:00
Erik Arvidsson
061f7694ad JS: Fix subtype issue with unions as elem types (#2518)
For compound types (List, Set, Map, Ref) the concrete type may be a
union. If that is the case all the types in the union must be a
subtype of the concrete type's element type.

`C<T | V>` is a subtype of `C<S>` iff T is a subtype of S and V is a
subtype of S.
2016-09-06 13:38:51 -07:00
Mike Gray
47565f39d1 Improve code based on tool analysis feeback (#2521)
Fixes are based on Go report card output:
- `gofmt -s` eliminates some duplication in struct/slice initialization
- `golint` found some issues like: `warning: should drop = nil from declaration of var XXX; it is the zero value`
- `golint` found some issues like: `warning: receiver name XXX should be consistent with previous receiver name YYY for ZZZ`
- `golint` says not to use underscores for function/variable names
- `golint` found several issues like: `warning: if block ends with a return statement, so drop this else and outdent its block`

No functional changes are included - just source code quality improvements.
2016-09-06 16:35:25 -04:00
zcstarr
3cdebb7e77 Add run safe method that reads stderr and stdout regardless of panic (#2475)
Run method will now always return stdout,stderr, and a recoveredErr
on Exit or Panic. MustRun will Panic with recoveredErr.
2016-09-06 11:30:57 -07:00
Erik Arvidsson
fd4c52acef stage.py: import os too (#2514) 2016-09-02 14:00:15 -07:00
Erik Arvidsson
893d0fa360 stage.py needs to import sys before using it (#2513)
Also changes the tabs to spaces. Thanks pylint.
2016-09-02 13:38:43 -07:00
Erik Arvidsson
f465b0bd3c Fix import of noms.staging in stage.py (#2510)
Also fix unit test that was not updated when the functions were
renamed.
2016-09-02 13:09:07 -07:00
Eric Halpern
48854dc999 Move datas.descendsFrom and prune search using Ref.Height (#2480)
- Move to commit.CommitDescendsFrom
 - When searching for an ancestor in the commit history, prune
   whenever the commit.Height() < ancestor.Height()
 - Add new test TestCommitDescendsFrom to verify correctness and pruning.
2016-09-02 11:44:38 -07:00
Dan Willhite
57f653dd39 Disable perf suite check for nomsRevision (#2508)
This needs to be disabled so that test cases succeed when
building demo-server. Currently this fails because we build
with CodePipeline so the source code is not in a git repo
when getting built.
2016-09-02 10:53:18 -07:00
Erik Arvidsson
017d599b1d Fix subtype issue with unions as elem types (#2506)
For compound types (List, Set, Map, Ref) the concrete type may be a
union. If that is the case all the types in the union must be a
subtype of the concrete type's element type.

`C<T | V>` is a subtype of `C<S>` iff T is a subtype of S and V is a
subtype of S.
2016-09-01 18:40:51 -07:00
cmasone-attic
e5fcfd6ebf Make poke hang on to parent's Commit metadata (#2504)
Before this, poke would drop any commit metadata from the dataset being modified. Now, it just pulls it forward.
2016-09-01 17:21:31 -07:00
Ben Kalman
ace91a323d Update CONTRIBUTING.md (#2468)
Add a bit more info on squash merges and perf tests.
2016-09-01 16:22:46 -07:00
Ben Kalman
0f6cc2023c Print skipped perf tests in verbose mode (#2497) 2016-09-01 16:22:23 -07:00
Erik Arvidsson
b9f15526ba JS Docs: Redirect to versioned sub directory (#2503) 2016-09-01 16:10:38 -07:00
Ben Kalman
8eba76a2ec Check before calling mkdirs on staging directory (#2502)
Also run pylint on staging.py.
2016-09-01 15:19:13 -07:00
Ben Kalman
6393e7c6e7 Fix "and and" comment from last patch (#2500) 2016-09-01 13:39:08 -07:00
Ben Kalman
b734a80cb1 Improvements to the perf viewer (#2499)
* Increase number of results to 20.
* See newer results at the end of the graph, not start.
* Make point radius show standard deviation to scale.
* Try not to draw datasets over each other.
* Invert light colours to make them darker.
2016-09-01 13:36:53 -07:00
cmasone-attic
9f080a2fa7 Fix test assertion in noms-merge::TestLose() (#2498)
This was caused by me changing an error string and failing
to update the test.
2016-09-01 11:34:04 -07:00
cmasone-attic
771eb092da Add failure tests for noms-merge (#2484)
Tests command line validation for noms-merge

Toward #2445
2016-09-01 11:31:29 -07:00
Erik Arvidsson
456f403b8a Build js docs correctly (#2495)
There was a typo (missing comma) in the command.
2016-09-01 10:38:29 -07:00
Ben Kalman
d1c90d11cd Add prepublish step to perf/viewer so that out.js gets built (#2496) 2016-09-01 10:33:03 -07:00
Aaron Boodman
42fd80be2e Add a super quick indeterminite progress meter to noms-merge (#2488) 2016-09-01 10:24:30 -07:00
Daniel Krech
01bdeab025 Add progress reporting to json_importer (#2494)
Fixes #2494
2016-09-01 00:04:26 -07:00
Aaron Boodman
63f4127aac Update CONTRIBUTING.md 2016-08-31 23:43:22 -07:00
Aaron Boodman
25fedb51fc Update cli-tour.md 2016-08-31 21:37:56 -07:00
Erik Arvidsson
eb42d99950 JS Doc builder (#2493)
Trying to work around issue where `npm run build-docs` fails
2016-08-31 18:27:58 -07:00
Erik Arvidsson
9452898ff7 Revert "Work around python popen bug for js docs" (#2492) 2016-08-31 18:18:16 -07:00
Erik Arvidsson
0a53ee7e3d Work around python popen bug for js docs (#2491) 2016-08-31 18:10:10 -07:00
Erik Arvidsson
ad66582f44 npm run did not work (#2489) 2016-08-31 17:56:12 -07:00
Ben Kalman
9c694f024b Add a perf test for CSV map import (#2461)
Currently we only have a perf test for CSV list import, which uses the
sf-crime dataset. This test uses the 43MB sf-registered-businesses
dataset instead, since sf-crime is too slow. Which is ironic, since we
normally parse sf-crime into a map.

I've also tightened up some of the other perf tests.
- Fixed a bug where Database was shared between runs.
- Make the pure CSV parsing test use a smaller dataset, it doesn't need
  to use something as large as ny-vehicle-registrations.
2016-08-31 17:05:00 -07:00
Erik Arvidsson
b857b95bb4 Drive js docs from npm script (#2486)
Turns out that documentationjs reads the babelrc file from js/noms
which leads to us having to do npm install in js/noms. Therefore
putting the dev deps in there and running the script from there
makes things cleaner.

It also allows doing:

  npm run build-docs

Towards #1471
2016-08-31 16:57:31 -07:00
Ben Kalman
815ca1586f Add perf build.py and util (#2487) 2016-08-31 16:50:34 -07:00
Ben Kalman
1e8ae3f214 Make perf viewer point radius show standard deviation (#2485) 2016-08-31 16:42:59 -07:00
Mike Gray
733a1fd97a Update spelling.md with more syntax (#2450) 2016-08-31 19:25:08 -04:00
Erik Arvidsson
7c47547110 Add node to the PATH for documentation build (#2482) 2016-08-31 15:43:43 -07:00
Erik Arvidsson
adff0da9c3 Build script for building JS SDK Docs (#2479)
This script npm installs documentationjs and runs it on noms.js and
puts the output in $WORKSPACE/build

It is intended to be used with Jenkins.
2016-08-31 14:51:23 -07:00
Ben Kalman
70c43f4e28 Add stage.py for perf viewer (#2478) 2016-08-31 14:29:17 -07:00
cmasone-attic
49ea5ec3c0 Introduce noms-merge, a standalone noms tool for merging datasets (#2470)
This is a first pass at exposing the new merge package to users.  The
tool is very basic, and currently only works on datasets in the same
database. It requires the 'parent' (i.e. a common ancestor of the two
datasets being merged) to be provided by a commandline option; a
follow-on patch will make the code discover this ancestor automatically.

Toward #2445
2016-08-31 14:20:21 -07:00
Ben Kalman
aaad3601db Tweak number of reps and timeout of perf bot script (#2462) 2016-08-31 11:27:44 -07:00
Aaron Boodman
b279f08057 Update CONTRIBUTING.md 2016-08-31 11:15:24 -07:00
Aaron Boodman
10944bf23b Update CONTRIBUTING.md 2016-08-31 11:05:51 -07:00
Aaron Boodman
4227aae20c Update CONTRIBUTING.md 2016-08-31 11:04:10 -07:00