* Modifications to ipfs-chat and ipfs chunkstore
* Change ipfs paths to include directory where ipfs repo is stored.
* Rework ipfs-chat to create ipfs chunkstores manually rather than
relying on Spec.ForDataset. This enables creating two chunkstores
(one local and one network) using the same IpfsNode (ipfs repo).
* Create separate replicate function for daemon and mergeMessage
function for client to experiment with slightly different behaviors
for each.
* Re-organization of code to remove duplication.
The main points are:
* added event loop to process events synchronously
* more agressive about not re-processing msgs from other nodes
that we've already processed
* fixed bug in ipfs chunkstore HasMany()
* Add go-base58 library
This makes all but types.Type be backed by a []byte.
The motivation is to reduce the allocations and the work needed to be
done when we read parts of a value (especially prolly trees).
Towards #2270
In most cases this will avoid writing the root chunk of a prolly tree,
which is the behavior we're aiming for: a prolly tree might be used
inline in which case the root never needs to be written.
The solution in this patch is imperfect because it may unnecessarily
write chunks, but this is rare.
Fixes https://github.com/attic-labs/noms/issues/3645
This allows parsing all Noms values from the string representation
used by human readable encoding:
```
v, err := nomdl.Parse(vrw, `map {"abc": 42}`)
```
Fixes#1466
This patch implements a new strategy for Pull() that pulls the chunks
from a given level of the graph over in the order they'll be
encountered by clients reading the graph.
Fixes#2968
The new version of this tool now estimates the locality of a DB
written using the "grandchild" strategy implemented by
types.ValueStore. It does do by dividing each level of the graph
up into groups that are roughly the size of the branching factor
of that level, and then calculating how many physical reads are
needed to read each group.
In the case of perfect locality, each group could be read in a
single physical read, so that's what the tool uses as its estimate
of the optimal case.
Toward #2968
When requesting a range of values read all the chunks ahead of time.
This works for indexed sequences. Does not include support for ordered sequences.
Work towards https://github.com/attic-labs/noms/issues/3619
stretchr has fixed a bug with the -count flag. I could merge these
changes into attic-labs, but it's easier to just use strechr.
We forked stretchr a long time ago so that we didn't link in the HTTP
testing libraries into the noms binaries (because we were using d.Chk in
production code). The HTTP issue doesn't seem to happen anymore, even
though we're still using d.Chk.
Print Ref values as #123 instead of 123
Since our hashes are SHA-512 and we write them using Base32 there are a lot of overlaps with other parts of NomDL. This makes them unambiguous.
Towards #1466
This removes the type tagged version of the human readable encoding.
Motivation: Simplify this in preparation to make the HRS unambiguous so that we can write a parser.
Towards #1466
Introduce Sloppy - an estimating compression function for snappy - which allows for the rolling hash to better produce a given target chunk size after compression.
When we added GetMany and HasMany, we didn't realize that requests
could then be larger than the allowable HTTP form size. This patch
makes the body of getRefs and hasRefs be serialized as binary instead,
which addresses this issue and actually makes the request body more
compact.
Fixes#3589
* use kingpin for help and new commands, set up dummy command for noms blob
* document existing commands using kingpin
* remove noms-get and noms-set in favor of new noms blob command
* normalize bool flags in tests, remove redundant cases that kingpin now handles
* add kingpin to vendor files
* make profile flags global
* move --verbose and --quiet to global flags
Erik Arvidsson