Commit Graph

3694 Commits

Author SHA1 Message Date
Dan Willhite
10ec10dc00 Add ability to register HRSCommenters on Structs. (#3609)
Clients can register HRSCommenters to cause additional info
to be included as comments when generating the human readable
encoding for Noms Structs.
2017-09-13 17:21:08 -07:00
Benjamin Kalman
26eb9e3713 Don't write a sequence chunk if there is no parent (#3699)
In most cases this will avoid writing the root chunk of a prolly tree,
which is the behavior we're aiming for: a prolly tree might be used
inline in which case the root never needs to be written.

The solution in this patch is imperfect because it may unnecessarily
write chunks, but this is rare.

Fixes https://github.com/attic-labs/noms/issues/3645
2017-09-13 15:52:31 -07:00
Erik Arvidsson
3db7be5062 Clean up Type WalkValues (#3700)
Use good practice OO design 😝
2017-09-13 15:45:52 -07:00
Erik Arvidsson
8f95c25403 Remove some printf debugging from tests (#3701) 2017-09-13 15:36:42 -07:00
Erik Arvidsson
5ff6432c7b Add support for parsing values (#3688)
This allows parsing all Noms values from the string representation
used by human readable encoding:

```
v, err := nomdl.Parse(vrw, `map {"abc": 42}`)
```

Fixes #1466
2017-09-13 15:02:01 -07:00
cmasone-attic
d3be53d164 csv-invert: Perf improvments! (#3698)
Tweaking the main loop that processes list entries to avoid some
map assignments, lookups, and allocations saves 15% or so, resulting
in an overall savings of about 1m on the 6m runtime of our test
workload (as run on my laptop).

Towards #3690
2017-09-12 15:48:22 -07:00
cmasone-attic
1b5ac05793 csv-invert command line tool (#3689)
Takes the output of a CSV file imported as a List of Struct and
"inverts" it so that it's now a Struct of Lists.

Example:

List<Struct Row {
  Base?: String,
  DOLocationID?: String,
}>

becomes

Struct Columnar {
  base: List<String>,
  dolocationid: List<String>,
}
2017-09-12 15:05:31 -07:00
cmasone-attic
41f63a5a6a Stop noms sync from destroying locality (#3659)
This patch implements a new strategy for Pull() that pulls the chunks
from a given level of the graph over in the order they'll be
encountered by clients reading the graph.

Fixes #2968
2017-09-11 16:04:13 -07:00
cmasone-attic
14e95379af NBS: Fragmentation tool using new estimate of locality (#3658)
The new version of this tool now estimates the locality of a DB
written using the "grandchild" strategy implemented by
types.ValueStore. It does do by dividing each level of the graph
up into groups that are roughly the size of the branching factor
of that level, and then calculating how many physical reads are
needed to read each group.

In the case of perfect locality, each group could be read in a
single physical read, so that's what the tool uses as its estimate
of the optimal case.

Toward #2968
2017-09-11 15:34:17 -07:00
Aaron Boodman
a18bd984d1 Update importer.go 2017-09-09 19:59:09 -07:00
wardn
b213146428 fix csv-importer dest-type description (#3682) 2017-09-09 19:58:06 -07:00
wardn
af6000ee16 add vi .swp files to the gitignore (#3685) 2017-09-09 19:57:34 -07:00
wardn
79e285e5d5 explicit collection types (#3683) 2017-09-09 19:56:30 -07:00
Aaron Boodman
484587a2b4 Fix broken build (#3680) 2017-09-08 02:54:45 -07:00
Aaron Boodman
f0568592b8 Add import to p2p sample (#3679) 2017-09-08 02:10:01 -07:00
phritz
025609828e request set & list elements in batch (#3660)
When requesting a range of values read all the chunks ahead of time.

This works for indexed sequences. Does not include support for ordered sequences.

Work towards https://github.com/attic-labs/noms/issues/3619
2017-09-07 16:23:22 -10:00
phritz
4d7ac48736 move note about commutativity close to its mention 2017-09-07 15:54:58 -10:00
phritz
eeed68d598 add nav header to whats next 2017-09-07 15:51:04 -10:00
phritz
d6455016b5 add nav header to vision 2017-09-07 15:50:46 -10:00
phritz
794e9f1f9c add nav header to demo app 2017-09-07 15:50:06 -10:00
phritz
18fe206714 add nav header to how to use noms 2017-09-07 15:49:45 -10:00
phritz
829dc3b939 move nav above header 2017-09-07 15:49:13 -10:00
phritz
d72d0e8344 add nav header to about noms 2017-09-07 15:48:18 -10:00
phritz
f0077ccbf6 add example code
Hasn't been tested yet, none of this probably works.
2017-09-07 15:35:10 -10:00
Aaron Boodman
a25ea915a7 add a simpler p2p ipfs-chat (#3678) 2017-09-07 18:32:47 -07:00
phritz
6845633f16 break instructions down by storage type
And lots of other changes. Still WIP.
2017-09-07 14:31:13 -10:00
Dan Willhite
51541db629 Add instructions for running ipfs-chat demo 2017-09-07 17:09:17 -07:00
Ben Kalman
03b7221c36 Use stretchr/testify not attic-labs/testify (#3677)
stretchr has fixed a bug with the -count flag. I could merge these
changes into attic-labs, but it's easier to just use strechr.

We forked stretchr a long time ago so that we didn't link in the HTTP
testing libraries into the noms binaries (because we were using d.Chk in
production code). The HTTP issue doesn't seem to happen anymore, even
though we're still using d.Chk.
2017-09-07 15:01:03 -07:00
Eric Halpern
5d1723674b Add --lowercase option to map column names to lowercase struct names (#3675)
* Add --lowercase option to map column names to lowercase struct names

By default, each column name maps to a struct field preserving the original case.
If --lowercase is specified the resulting struct fields will always be lowercase.
2017-09-07 11:21:31 -07:00
Aaron Boodman
f01e7afa9e Update vision.md 2017-09-06 23:27:43 -07:00
Aaron Boodman
8b3c14ebd5 Update vision.md (#3673) 2017-09-06 23:27:08 -07:00
Aaron Boodman
d35c95bf7e Update faq.md 2017-09-06 22:15:34 -07:00
Aaron Boodman
80f77c9994 Update faq.md 2017-09-06 22:07:35 -07:00
phritz
3b4cd4084b indent list 2017-09-06 16:36:10 -10:00
phritz
819fb86173 fix formatting 2017-09-06 16:35:33 -10:00
phritz
29e8bf6ef2 fix link to ipfs-chat dir 2017-09-06 15:05:28 -10:00
phritz
a7ffaa5fab minor tweaks to about noms text 2017-09-06 14:43:04 -10:00
phritz
f6bf81c099 update spacing 2017-09-06 14:39:02 -10:00
phritz
e7c8b8d86d update vision 2017-09-06 14:27:45 -10:00
phritz
43d25c9116 first draft of d12db landing site docs (#3672) 2017-09-06 14:10:58 -10:00
Aaron Boodman
759cf4fe58 gofmt (#3671) 2017-09-05 18:47:56 -07:00
Dan Willhite
071ba838d2 Modifications to ipfs-chat can ipfs chunkstore (#3666) 2017-09-05 18:35:50 -07:00
Ian Davis
8c4fa02c6e Fix godoc strings for several functions (#3665) 2017-09-05 11:50:30 -07:00
Ian Davis
63a72d3bfb Use ETag/If-None-Match in url-fetch (#3664) 2017-09-04 01:02:21 -07:00
Ian Davis
652f0eed9a Bring noms-tour up to date with recent changes 2017-09-03 22:57:10 -07:00
Ian Davis
052d3d73c8 Don't ignore spec split error in ForDatasetOpts 2017-09-03 09:06:21 -07:00
Ian Davis
99f696edb8 Rename --summarize flag to --stat 2017-09-03 07:26:38 -07:00
Erik Arvidsson
84b4aba5a6 HRS: Prefix map/set values with map/set (#3655)
We now print map and set values as:

```
map {
  "string": 42,
  "set": set {
    true,
    false,
  },
}
```

Towards #1466
2017-08-31 13:31:21 -07:00
Dan Willhite
aa65868741 Changes to accommodate new version of ipfs 2017-08-31 10:48:27 -07:00
Dan Willhite
c0aa9507e4 New version of ipfs and dependencies 2017-08-31 10:48:27 -07:00