This patch is unfortunately large, but it seemed necessary to make all
these changes at once to transition away from having an HTTP
ChunkStore that could allow for invalid state in the DB. Now, we have
a RemoteDataStoreClient that allows for reading and writing of Values,
and performs validation on the server side before persisting chunks.
The semantics of DataStore are that written values can be read back
out immediately, but are not guaranteed to be persistent until after
Commit() The semantics are now that Put() blocks until the Chunk is
persisted, and the new PutMany() can be used to write a number of
Chunks all at once.
From a command-line tool point of view, -h and -h-auth still work as
expected.
This patch removes the special RemoteDataStore implementation of
CopyReachableChunksP, as this is seldom-used and adds complexity
that stands in the way of Issue 654
This patch is the first step in moving all reading and writing to the
DataStore API, so that we can validate data commited to Noms.
The big change here is that types.ReadValue() no longer exists and is
replaced with a ReadValue() method on DataStore. A similar
WriteValue() method deprecates types.WriteValue(), but fully removing
that is left for a later patch. Since a lot of code in the types
package needs to read and write values, but cannot import the datas
package without creating an import cycle, the types package exports
ValueReader and ValueWriter interfaces, which DataStore implements.
Thus, a DataStore can be passed to anything in the types package which
needs to read or write values (e.g. a collection constructor or
typed-ref)
Relatedly, this patch also introduces the DataSink interface, so that
some public-facing apis no longer need to provide a ChunkSink.
Towards #654
Since we put a type system in Noms, we can now use it
to check that a ref that purports to be the new Head
for a Dataset after a pull actually points to a Commit.
Fixes#133
The generated code for typed structs now uses a Go struct which
implements Value directly. The fields in this struct uses the "user"
type. (The union value still uses types.Value though.)
When a typed struct is created by the decoder, it asks for a struct
builder which returns a channel that the values of the fields of the
struct are sent to.
We now do a recursive call which bottoms out with a ref.Ref for RefKind
Values. This means that we traverse into nested structures consistently.
The effect of this is that we get all the refs that the current chunk
references.
Ref Values now have a TargetRef() method that returns the ref.Ref of
the target the Value is referencing.
Note: This is a breaking change. In old code the Ref() of the Value was
the Ref of the underlying target.
Fixes#464
This makes the new typed serialization the default (the old
serialization is not used but the code has not been cleaned up yet).
Some things are no working in the new world:
Chunking - The compound list is not working correctly any more. The
Chunks method is having issues because it assumed things based on the
old implicit chunking.
Commit - uses a `Set(Commit)` which means that the parent commit is
embedded. We need to change that to be `Set(Ref(Commit))` so that the
parent commit is referenced instead.
We'd wound up in a spot where serialization code used 'TypeRefKind' to
mean one of two very different things...either an actual value that
describes some Noms type, or a reference to a type definition that
lives somewhere else. To get rid of this ambiguity, we introduce
'UnresolvedKind' to take over the latter meaning. Now, TypeRefKind
means _only_ a value that describes a type. If you want to point off
to a type definition elsewhere in the type package, or in another
type package, use UnresolvedKind.
Replace datastore head with a map of datasetID's to commits. Each commit in the map represents that dataset's head. Fixes#402. Fixes#60. Filed #404 about small window of potential conflict with updating root that needs to be resolved at some point. # Please enter the commit message for your changes. Lines starting
The new serialization format use "t " as in typed. The rest of the
message is a JSON array describing the typed data. The type is
described by types.TypeRef
Fixes#384
Issues #281, #304
The TypeRef function for a Noms Struct should be the (Name, PkgRef) and
not the description of the struct fields. This is important because
when serializing we need to write the package ref.
Towards #281#304
Also, switch to using a ref.Ref when getting/setting the package
ref in a TypeRef. Using a types.Ref just led to lots of manual
boxing and unboxing every time I wanted to use the reference.
Toward issue #294
This patch mostly merges parse.Package and types.Package, though it
can't quite go all the way. A types.Package doesn't have 'using'
declarations, while the parsed representation of a .noms file needs to
have that information. Hence, the parse package is moved to the 'pkg'
package, and pkg.Parsed is introduced. This type embeds types.Package
and adds the necessary additional information.
To make inroads on handling imports, I enhanced ParsePackage() (now
called ParseNomDL()) to actually process the 'alias' and 'import'
statements in the input and go replace namespaced type names in the
package with refs of imported packages. For example, the TypeRef for
'Bar' generated in the following package
alias Foo = import "sha1-ffffffff"
struct Bar {
f: Foo.RockinStruct
}
will actually return types.Ref{sha1-ffffffff} when you call PackageRef()
on it.
In addition, I've added a function to the new 'pkg' package,
which allows the caller to get the dependencies of a type package
from a chunk store.
Fixes issue #353, towards issue #294
These were two representations of, essentially, the same information.
They were separate because they provided different APIs to similar
information, but the APIs became more similar once we started using
native types (as opposed to Noms types) for the various Make*TypeRef()
functions.
Unifying these is a big step to unifying parse.Package and types.Package,
which is pretty necessary for dealing with imported packages.
Fixes issue #338
This make Commit a typed struct with a Set(Commit).
This also fixes a case where the recursive detection for determining
if a Def can be created was not working.
