Though Raf and I can't figure out how, it's clear that the method we
initially used for calculating the max amount of space for
snappy-compressed chunk data was incorrect. That's the root cause of
of all the chunks to be written and summing the snappy.MaxEncodedLen()
for each.
Fixes#3156
Apparently, there's some issue running demo-server with --verbose
in prod, so we don't do it. This means that the logging info I
added isn't showing up. Change the logging code to use fmt.Fprintf()
Also, add unit test for errata functionality.
Towards #3156
Add support for unions containing scalar values by "boxing" those scalars in that context. Also add a hash field to Struct so that empty structs have at least one field
There's some case that causes chunks that compress to more than about
55k (we think these are quite big, chunks that are many hundreds of K
in size) not to wind up correctly inserted into tables. It looks like
the snappy library believes the buffer we've allocated may not be
large enough, so it allocates its own space and this screws us up.
This patch changes two things:
1) The CRC in the NBS format is now the CRC of the _compressed_ data
2) Such chunks will be manually copied into the table, so they won't
be missing anymore
Also, when the code detects a case where the snappy library decided to
allocate its own storage, it saves the uncompressed data off to the
side, so that it can be pushed to durable storage. Such chunks are
stored on disk or in S3 named like "<chunk-hash>-errata", and logging
is dumped out so we can figure out which tables were supposed to
contain these chunks.
Towards #3156
We need to fully map the noms types into graphql even though the mapping isn't useful, otherwise the endpoint may die when attempting to create a schema for types that aren't mapped.
This changes to use Jest. Benefits of jest is parallel tests and jest can run the minimal set of tests that are affected by a change (by looking at dependencies).
The main work was to disentangle our cyclic dependencies. To do this I had to remove some runtime assertions in encode value as well as expose the values of a struct without going through a struct mirror.
Panics on background goroutines take down the server. This patch
hacks in a mechanism to pipe failures during NBS tableReader.extract
back to the main goroutine so the server doesn't die on this failure
and I can diagnose it.
* First pass at compaction
The first cut at compaction blocks UpdateRoot() while it compacts n/2
tables down into a single, large table (where n == number of tables
named in the NBS manifest). It then attempts to update the manifest
with one referencing the compacted table, the novel tables from the
client, and the remaining upstream tables that were not compacted.
If the update fails, probably due to an optimistic lock failure, the
client drops the compacted table it just created, pulls in the tables
from the newly-discovered upstream manifest, and tries again.
Known flaws:
- may explode RAM (#3130)
- doesn't handle novel tables > max tables (#3142)
- may handle optimistic-lock-failures suboptimally (#3141)
Fixes#3132
Also, fixes#2944 because doing so simplifies some code.
It was using a file watcher which was causing us to run out of inodes
on our Jenkins build. Instead, use flow-copy-source which does the
same copying without a file wathcer (we no longer used the --watch flag
anyway).
DynamoDB provides eventual read-after-write consistency, unless you
ask explicitly for strong consistency. When committing over HTTP, the
code writes a bunch of Values and then attempts to UpdateRoot to point
to one of those novel chunks. The UpdateRoot call _must_ see the
result of the prior WriteValue, or it may fail.
Fixes#3084
* Fixes a long standing bug in which the RemoteBatchStore is accidentally caching all chunks
* ValueStore's value cache now stores `Promise<?Value>` so that concurrent `readValues` of the same value can share a single decoding
* Removes the debug-only chunk hash check which keeps tripping up perf investigations
This code initially panicked in this case, because there didn't used
to be a reasonable way that a caller might wind up trying to update
away from a Root that didn't match NomsBlockStore's internal
bookkeeping. Now, given the new Flush() behavior there is. So, just
return false and allow the caller to take appropriate action.
Towards #3089
This patch adds a static function which can walk graphs looking for (and diffing) two structs. It uses type information to avoid traversing sub-values which can't contain structs. It also uses a similar approach as sync to avoid visiting common sub-chunk-graphs.
