Under load, our server can exhaust the number of file descriptors it's
allowed to have open at one time. Part of this is because of how many
incoming connections it's handling, but we believe that handling lots
of simultaneous reads to leveldb is the larger part of the issue.
This patch applies the rate limit we were using for writing to both
read and write operations.
Fixes#2227
1) ValueStore now maintains validation cache info across Commit() calls,
so we can remove a workaround in Dataset.Commit()
2) Update comment for setNewHead
It (a) had a spurious blank line, (b) should print the dataset path not
just its ID, (c) should include the # for the hash. Now it says:
```
> noms ds -d path/to/db::my-dataset
Deleted path/to/db::my-dataset (was #lvnbmpj94r9h3nuaj1t0m4jeqp25o1ve)
>
```
This fixes https://github.com/attic-labs/noms/issues/2235 where set/map
diff was still running when noms log has finished, and already closed
its database. Commonly this would crash.
I've removed panic-based control flow to make error handling explicit,
and added a "DiffLeftRight" method to set/map so that noms log completes
ASAP. See the issue for details.
I need this in the short term because CSV raw files are too large to
check into github, so they need to be split up. It's easier to just do
"cat * | url-fetch -stdin" than recombining them on the file system.
In the server side of the Remote Databse, the handler for
UpdateRoot now verifies that the new proposed Root is of a
legal type: empty map OR Map<String, Ref<Commit-like>>
Fixes#2116
When running loadtest on a number of machines all at once, machines that
start at the same time tend to wind up with the same random seed, leading
to the same sequence of operations.
Mixing in the IP address perturbs the seed enough to change things up.
This patch creates a new kind of chunks.Factory that demo-server
uses to vend ChunkStore instances that all share the same
MemoryStore-based Chunk cache. This cache _will_ grow without bound,
but the current RAM/data ratio on demo.noms.io means that, in practice,
we will be fine for a bit.
This will need to be removed in favor of a real solution in Issue #2227Fixes#2009
It turns out Pull() was making some bad assumptions about how the Go
heap package used its backing storage. Since it wasn't really relying
on heap guarantees anyway, this changes the code to use a slice of Ref
that's sorted in order of increasing ref-height: RefByHeight.
Erik Arvidsson