It's already propagated for List.
The problem occurs when accessing/modifying chunked Sets and Maps that
have read from the DB then modified. For example:
set := ds.Head().(Set)
set = set.Remove(42)
// These will both crash:
set.Has(43)
set.Insert(43)
If |set| is a compoundSet then the new Set returned from Remove won't
have a ChunkSource.
- When Has is called, the set will attempt to read in chunks, but there
is no longer a ChunkSource to read from.
- When Insert is called, the set will re-chunk, which may require
reading in more chunks.
Instead of writing sequence chunks as soon as they're are created (as a
result of hitting chunk boundaries), only write them once they're
referenced - which only happens if those chunks are themselves chunked.
The effect of this is root chunks of collections/blobs aren't written
until they're committed, which makes the XML importer run twice as fast
on a month of MLB data - 60s instead of 120s, with --ldb-dump-stats
showing a PutCount of 21,272 instead of 342,254.
In the future it should be possible to avoid writing *any* chunks until
the root is committed, which will improve incremental update
performance, but that's a larger change (issue #710). This change fixes
issue #832.
Ref values use the TargetRef to get the internal implementation
RegisterFromValFunction and ToNomsValueFromTypeRef were only used by
ref values at this point so these were renamed and simplified to be
more specific for ref values
We now do a recursive call which bottoms out with a ref.Ref for RefKind
Values. This means that we traverse into nested structures consistently.
The effect of this is that we get all the refs that the current chunk
references.
The implementation of structs is a Map and we used to reserve a key
with the name "$type" for the TypeRef. This is no longer needed since
the TypeRef is a constant per struct and needs no storage.
Fixes#450
This makes the new typed serialization the default (the old
serialization is not used but the code has not been cleaned up yet).
Some things are no working in the new world:
Chunking - The compound list is not working correctly any more. The
Chunks method is having issues because it assumed things based on the
old implicit chunking.
Commit - uses a `Set(Commit)` which means that the parent commit is
embedded. We need to change that to be `Set(Ref(Commit))` so that the
parent commit is referenced instead.
The new serialization format use "t " as in typed. The rest of the
message is a JSON array describing the typed data. The type is
described by types.TypeRef
Fixes#384
Issues #281, #304
This will enable us to walk the chunk graph without having to go
through weird contortions to figure out which values don't have
chunks in any chunkstore (because they were inlined).
Towards issue #82
It turns out these functions aren't actually generally useful,
because they deal in Values and Refs, while what we actually
want is to deal in chunks. So...
Revert "Settle big/small vs to/from ONCE AND FOR ALL"
This reverts commit f51c575cd2.
Revert "Add a naive Diff() function between two Values."
This reverts commit 662bc5e3c4.
Given two refs, we need to be able to determine the set of chunks
reachable from one that cannot be reached from the other. This patch
adds a naive implementation of this functionality that essentially walks
the value tree down from the values pointed to by two refs and compares them.
It also has a little command-line driver, but this may be removed.
Fixes issue #82
1) Get rid of temp file usage in FileStore. Instead, write to a buffer and then
dump to disk on Close().
2) Chk.Equals() uses reflection even if you call it on primitive types, which
can be surprisingly costly. Switch to a Chk.True() in a couple of hot paths.
