* use kingpin for help and new commands, set up dummy command for noms blob
* document existing commands using kingpin
* remove noms-get and noms-set in favor of new noms blob command
* normalize bool flags in tests, remove redundant cases that kingpin now handles
* add kingpin to vendor files
* make profile flags global
* move --verbose and --quiet to global flags
Prior to this patch, whenever we created a chunkSource for a table
persisted to AWS, awsTablePersister::Open() would hit DynamoDB to
check whether the table data was stored there. That's how it knew
whether to create a dynamoTableReader or an s3TableReader. This
results in consulting Dynamo (or the in-memory small-table cache)
every time we go to open a table. Most of the time, this isn't
necessary, as we separately cache table indices and really only
need that data at Open() time.
This patch defers reading table data if possible.
Fixes#3607
This will likely bloat the cache with tables no one's going to read
data from, BUT doing this also means that most checks to see if a table
is in Dynamo at all can proceed locally. Stopgap until #3607 lands
Fail over to fully-consistent reads if no result. This means that
misses will get more expensive, but hits will cost us half what they
were costing in the initial version of the code.
Fixes#3604
Looking at metrics on staging today, there are frequent spikes of
tens of thousands of throttled DynamoDB reads. One explanation is
that we're constantly evicting 'hot' tables from the in-memory
cache because the working set is larger than the space we've
allotted for the cache.
There seems to be a floor on the amount of time required to persist
small objects to S3. For workloads that generate lots of small tables,
this can really add up. DynamoDB is much faster to read/write, and can
hold items of up to 400k. This patch stores tables with < 64 chunks
that are < 400k in DynamoDB, caching them in memory on persist and
open to reduce load on the back end.
Fixes#3559
Right now, the only kinds of Commits that the server will retry are those
to different datasets. That is, if another client concurrently landed a change
to some other dataset, and that is the only thing that causes your Commit
attempt to fail, the server will retry.
Fixes#3582
This patch uses process-wide per-store locking to ensure that only one
NomsBlockStore instance is ever trying to update the upstream NBS
manifest at a time. It also locks out attempts to fetch the manifest
contents during that window.
Conjoining is now much simpler. Since only one instance can ever be in
the critical path of Commit at a time, and conjoining is triggered on
that critical path, we now simply perform the conjoin while excluding
all other in-process NBS instances. Hopefully, locking out instances
who want to fetch the manifest contents during a conjoin won't cripple
performance.
Fixes issue #3583
All messages emitted from non-cli code are now output through the standard
logger rather than directly to stdout. This gives code that embeds noms control
over how messages are logged (using log.SetFlags and log.SetOutput).
When the noms cli is used, the logger is initialized so that it only prints the
string logged. In other cases, the log setup is left to the embedding code.
toward: #3590
By adding per-entry locking to indexCache, this patch allows both NBS
storage backends to ensure that only one goroutine is loading a given
table index at a time. Previoudly, if multiple readers tried to open
the same table at the same time, it was possible for both to get a
cache-miss and then read in the table index at the same time, which
wastes memory. Now, one will win the race, load the index, and then
the other will see a cache hit.
Fixes#3581
If NomsBlockStore can assume that its manifest is a cachingManifest,
it can pre-emptively check to see if someone else in-process has
already moved the manifest forward and, if so, fail early.
Fixes#3574
In httpChunkStore, calls to Get/Has and friends put a request object
with a 'return channel' onto a queue channel, and then block on the
return channel. The queue channel was buffered, which made it
impossible to cause Get, Has et al to terminate reliably when the
store was closed.
This patch removes the buffering on the channel so we can
deterministically bail from Get/Has et al when closing the store. I
don't think we were actually seeing any benefit from the buffer on the
queue channels, because everywhere we write to one of them we
immediately block on another channel, waiting for the result of the
request.
Fixes#3566
There were races in both manifest implementations, between cache usage
in ParseIfExists() and Update(). It was possible for Update() calls to
occur between the time that ParseIfExists() read the manifest data and
the time it updated the cache. This led to newer manifest content
getting clobbered in the cache by out-of-date manifest content.
The new approach takes caching out of the hands of the manifest impls,
instead wrapping a shared caching layer around the outside.
Fixes#3551
Add a method to chunks.Factory that allows the caller to
signal that it's willing to try to make forward progress
using an out-of-date ChunkStore. This allows AWSStoreFactory
and LocalStoreFactory to vend NBS instances without hammering
persistent storage every time.
Towards #3491
Rafael Weinstein