cdb22e131b
Address comments
2015-09-23 11:49:14 -07:00
63c956a5c5
Add types.TypeRef
...
We want to explore encoding type information about Noms data in
the Noms database. So, we need some way to describe types. This
takes the shortest path to making a Noms type called "TypeRef" that
is a peer of Set, Map et al and can describe all the types we currently
use.
2015-09-22 16:19:38 -07:00
591026bcfa
Chunk
2015-09-21 13:24:41 -07:00
1a4bdae624
Make types.Ref explicit
...
No more implicit Get or Equals
2015-09-18 14:03:19 -04:00
bbe396fb94
Merge pull request #308 from arv/set-extras
...
NomDL: Codegen for Set extras
2015-09-17 17:25:59 -04:00
0203a71e9d
Merge pull request #309 from arv/map-extras
...
NomDL: CodeGen for Map extras
2015-09-17 17:23:43 -04:00
f64f6ae81c
NomDL: CodeGen for Map extras
...
This adds Iter, IterAll and Filter to Maps
Issue #303
2015-09-17 17:20:49 -04:00
175cc7ccc1
NomDL: Codegen for Set extras
...
This adds Iter, IterAll and Filter to Sets
Issue #303
2015-09-17 16:49:22 -04:00
8d56b8c968
Reland: ChunkSource.Get() now returns a []byte. chunks.Serialize takes a stream of Chunks
2015-09-17 12:31:25 -07:00
5a83b23f29
NomDL: Codegen for List extras
...
Adds Iter, IterAll and Filter to the List(T) types
Depends on PR #298
Issue #303
2015-09-17 14:24:36 -04:00
e10e6224b0
Codegen for NomDL
...
This adds a new codegen that reads .noms files and generates Go
API for these types
Issue #304
2015-09-17 14:01:49 -04:00
a62cf1bbb8
Revert "ChunkSouce.Get() now returns []byte"
...
This reverts commit 2be269acf4
2015-09-16 20:47:36 -07:00
522df3055b
Merge pull request #302 from aboodman/stable-gen
...
nomgen: stabilize the order types are generated in across runs
2015-09-16 17:41:39 -07:00
91df15a12a
nomgen: stabilize the order types are generated in
2015-09-16 17:26:31 -07:00
2be269acf4
ChunkSouce.Get() now returns []byte
2015-09-16 16:42:36 -07:00
dddf81b095
Add List.Map & List.MapP
2015-09-08 15:12:24 -07:00
7966384aeb
Get rid of alice-short.txt
...
Use a deterministic pseudo random list instead
2015-09-08 11:20:12 -04:00
47953557d8
fix build
2015-09-04 16:18:09 -07:00
dc2ef0274d
remove drone.yml and purposely break build to test
2015-09-04 16:16:21 -07:00
f58670bc83
NewBlob(): Reader.Read() can return both data and error.
...
Fixes #264
2015-09-04 15:02:29 -07:00
58e6666f83
Minor cleanup of compound list append
2015-09-04 14:28:29 -04:00
d06da3ca0a
Chunking: Multi level chunking for blobs
...
After a compound blob is created we try to chunk it again in a similar
way to how we chunk Lists. We use the refs of the sub blob and compute
a rolling hash over these. If the hash matches a pattern then we split
the existing compound blob into a new compound blob with sub blobs
which are slices of the original compound blob.
Issue #17
2015-09-03 19:47:17 -04:00
57c5fd9eeb
Introduce a list iterator to get rid of O(log n) lookups in loops
...
When we are building the chunked lists we had a lot of loops that did
O(log n) Get operations. Since we are just getting consecutive elements
from the list we can make getting the next one O(1) making these loop
go from O(n*log(n)) to O(n)
Issue #215
2015-08-31 18:00:41 -04:00
8d85cc4625
Optimize compound list - tail part
...
When we write the part after the change and we hit a chunk split we
check whether the list also had a split at the same index (adjusted
for adding/removal). If it did then we know that the rest of the sub
list are the same.
Issue #215
2015-08-31 10:50:01 -04:00
6cb4d613b7
Update godeps
2015-08-30 17:11:04 -07:00
c86bc4aee2
Optimize compound lists
...
This reuses the head of a compound list.
Issue #215
2015-08-28 17:12:46 -04:00
b6197aadc4
Add support for compound lists.
...
Lists are now either a leafList or a compoundList. The compoundList
consists of sublists that are the chunks of the whole list.
2015-08-28 15:41:51 -04:00
2d96186090
Add encoding/decoding of compound lists
2015-08-28 15:27:35 -04:00
ab34143ba5
Pin dependencies using godep tool. Rewrite dep urls.
2015-08-26 14:05:40 -07:00
24e0e436e4
Revert change in blob.go
2015-08-26 09:28:04 -07:00
ccd70d7c65
Changed error handling in Marshal and Unmarshal
...
Instead of returning errors, these now use d.Exp to raise catchable
errors.
Also, added commit hash at which code was pulled from encoding/json
Marshal io.Reader into a Blob, unmarshal Blob into io.Writer
2015-08-26 09:28:04 -07:00
5de698b8f1
Add Unmarshal and Marshal
...
Unmarshal and Marshal are tools for moving data from Noms into native Go and
back. The rules are described in the documentation of the two functions, but
the behavior is broadly similar to encoding/json.
Towards issue #160
2015-08-26 09:27:58 -07:00
06c5bc6c1b
Abstract ChunkStoreWriter
2015-08-20 10:58:41 -07:00
0555d7a3c1
Remove errors from read/write/encode/decode
2015-08-18 16:37:04 -07:00
0e7d61efc6
Remove errors from ChunkStore and Ref
2015-08-18 16:24:26 -07:00
1bd5af910a
Add support for int8/uint8
...
It turns out that having these makes marshaling native go to and from noms
cleaner.
Towards issue #160
2015-08-17 13:44:50 -07:00
214b37eccf
Remove global imports of dbg package
...
Fixes #179
2015-08-08 23:57:37 -07:00
f407029526
Merge pull request #181 from aboodman/tagdex
...
Tagdex
2015-08-07 09:52:15 -07:00
ddebdcaefd
Slight modification to compound blob encoding
...
The json serialization now only contains the length of each individual
blob child.
The go representation of this still uses offsets but the offsets are
for the end delimiter.
For "hi" "bye" we get
{"cb", [{"ref": "sha1-hi"}, 2, {"ref": "sha1-bye"}, 3]}
compoundBlob{[2, 5], [sha1-hi, ,sha1-bye]}
Keeping the length in the serialization leads to smaller serializations
Using the end offset leads to simpler binary search and allows us to
use the last entry as the length.
Issue #17
2015-08-07 11:24:27 -04:00
673180c2c9
add values/walk.go
2015-08-07 08:04:42 -07:00
b207d7e7ca
Remove ToItems(), fix error reporting in ReadValue
2015-08-06 17:02:26 -07:00
f4ef0d5cbc
Address comments
...
Ensure reader gets closed in all cases in ReadValue, clean up BUG references,
delete NewCompoundBlob, and switch an io.Copy -> ioutil.ReadAll
2015-08-06 16:46:00 -07:00
bec1b344be
Migrate the types package to JSON decode using the enc package
...
This change removes the json decoding code from the types package and ports
it onto the enc package's encoding API.
Fixes issue #159
2015-08-06 16:09:35 -07:00
07046ce567
Address aa comments
2015-08-06 12:48:39 -07:00
1a3c3e2c41
Migrate the types package to JSON encode using the enc package
...
This change removes the json encoding code from the types package and ports
it onto the enc package's encoding API.
Towards issue #159
2015-08-06 10:30:16 -07:00
ea52c4ac7c
Implement Seek for Blob.Reader()
...
This allows us to only read the relevant chunks
Issue #17 , #155
2015-08-06 12:22:41 -04:00
c9d928f50b
Code review cleanup
2015-08-05 13:37:28 -04:00
d834f7d546
Change JSON serialization format for compound blobs
...
- Put the length last
- Skip the initial 0 since first blob is always at 0
Issue #17
2015-08-05 11:38:56 -04:00
a8db1242a4
Merge pull request #167 from arv/blob-use-offset
...
Swith to use offsets in compoun blobs
2015-08-05 09:52:39 -04:00
26c28e7158
Merge pull request #162 from aboodman/ref
...
Make ref.Ref a type of Value too.
2015-08-04 16:38:12 -07:00
Chris Masone