Introduce a 'compactingChunkStore', which knows how to compact itself
in the background. It satisfies get/has requests from an in-memory
table until compaction is complete. Once compaction is done, it
destroys the in-memory table and switches over to using solely the
persistent table.
Fixes#2879
Add GetMany(), which most ChunkStores implement by repeated calls to their own Get(), but creates the opportunity for stores to optimize reads of larger blocks of potentially sequential chunks (e.g. NBS).
Add RemoteBatchStore getRefs endpoint support for calling GetMany() rather than Get()
Remove ReadThroughChunkStore which was dead code.
Adds the ability for SequenceCursors to eagerly load all child sequences when the first child is requested.
The effect is for uses where noms tends to forward scan, it will read in batches of ~150 chunks (e.g. ~512k) rather than one chunk at a time.
On my MBP, this improves the raw blob read perf over http from ~60K/s to ~5MB/s.
When we read a chunk and create structs we were validating that the
struct was of the type it claimed to be.
This now, no longer does that validation, which matches Go.
This lets you do foo.bar@at(n) to get the nth element of a list, set, or
map (for lists, this is equivalent to foo.bar[n]). This patch also adds
support for negative indices to @at(-n) and [-n] to get the nth element
relative to the back of collections.
This patch introduces/expands the 'manifest' and 'tableSet'
abstractions, so that NomsBlockStore is no longer explicitly using any
file system operations
Towards issue #2877
These get the set/map element at a specific index.
I haven't implemented it in JS yet because the JS code has no method to
create a cursor at an index. This exists in Go because a refactor was
done a few months ago to add it, but it hasn't been ported to JS.
* Support marshaling from and unmarshaling to *types.Type
* Incorporate code review suggestions.
- Use fallthrough in switch
- Update decode godoc to spec *types.Type handling. Godoc for encode is fine as is.
toward: #2889
-
Encode chunk counts consistently as uint32 until #2873 is addressed. This also fixes an error in passing chunkCounts resulting from compaction that don't account for dropped (duplicate) chunks.
