There were races in both manifest implementations, between cache usage
in ParseIfExists() and Update(). It was possible for Update() calls to
occur between the time that ParseIfExists() read the manifest data and
the time it updated the cache. This led to newer manifest content
getting clobbered in the cache by out-of-date manifest content.
The new approach takes caching out of the hands of the manifest impls,
instead wrapping a shared caching layer around the outside.
Fixes#3551
Add a method to chunks.Factory that allows the caller to
signal that it's willing to try to make forward progress
using an out-of-date ChunkStore. This allows AWSStoreFactory
and LocalStoreFactory to vend NBS instances without hammering
persistent storage every time.
Towards #3491
1. In MarshalType() allow Noms collections without element types, e.g.:
type Foo struct {
A types.List
}
marshal.MustMarshalType(Foo{}) => struct Foo { A List<Value> }
2. There were cases where we would fail to marshal structs containing
non-comparable Go types (e.g. slices). Fix those using DeepEqual().
This is done by implementing StructNameMarshaler interface. For example
```go
type TestStruct struct {
...
}
func (ts TestStruct) MarshalNomsStructName() string {
return "HelloWorld"
}
```
Fixes#2972
This makes it possible to include/exclude multiple directories when rolling src into a vendor directory.
For example, to roll the `cmd` and `go` directories from github.com/attic-labs/noms
into your vendor directory, you could run the following:
```
cd $MY_REPO_HOME
$NOMS/tools/roll.py --incl cmd --incl go https://github.com/attic-labs/noms
```
This patch also strips '.git' if it's present on the end of the repo url.
In an NBS world, bulk 'has' checks are waaaay cheaper than they used
to be. In light of this, we can toss out the complex logic we were
using in Pull() -- which basically existed for no reason other than to
avoid doing 'has' checks. Now, the code basically just descends down a
tree of chunks breadth-first, using HasMany() at each level to figure
out which chunks are not yet in the sink all at once, and GetMany() to
pull them from the source in bulk.
Fixes#3182, Towards #3384
Previously, every NomsBlockStore instance decided when to conjoin
tables (and which to conjoin) entirely on its own, which led to A LOT
of concurrent conjoining that would mostly be wasted effort, as one
instance would win the race and then all the rest would drop their
work on the floor, rebase, and continue. This patch introduces a
'conjoiner' that is either process-global, or owned by one of the NBS
factory objects you can create. Now, NBS instances vended by a given
factory call this single conjoiner during Commit(), asking it to
perform a conjoin if necessary. If a conjoin is already underway, the
conjoiner blocks the caller until it's finished and then
returns. Whether the conjoin was triggered at the caller's request, or
the caller got to opportunistically piggyback on a conjoin already in
progress, the caller must rebase after Conjoin() returns.
Fixes#3422
