mirror/dolt
1
0
Fork 0
mirror of https://github.com/dolthub/dolt.git synced 2026-02-11 02:59:34 -06:00
Code Issues Packages Projects Releases 128 Wiki Activity
3,194 Commits 109 Branches 598 Tags
4eca6085cbc14a3d94104e54fc566fbda0d047fb
Commit Graph

3194 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ben Kalman
4eca6085cb Allow skipping invalid fields when marshalling (#2967) 
Currently unexported fields refuse to marshal or unmarshal, even if
they're told to skip. Now, they can be skipped. By default, they are
still errors.
 2016-12-20 16:47:49 -08:00
Rafael Weinstein
cb3390924f Remove unnecessary invariants (#2966) 
Remove unnecessary invariants
 2016-12-20 15:52:26 -08:00
wardn
262f10a14b Update go tour documentation to reflect dataset changes (#2960) 2016-12-20 09:01:17 -05:00
Aaron Boodman
a1c75f04d6 Fix crash in picasa importer with large albums (#2965) 
Fixes https://github.com/attic-labs/attic/issues/317
 2016-12-19 21:42:25 -08:00
Mike Gray
094ec4c94f Allow json-import input to be a file (#2964) 2016-12-19 19:27:05 -05:00
Ben Kalman
fa1ab8a61c Make datas.localFactory public and add NewLocalFactory (#2961) 
I want to adapt datas.NewLocalFactory(chunks.NewMemoryStoreFactory()).
 2016-12-19 10:46:25 -08:00
Dan Willhite
d41a4bc6f7 Add set methods to verbose. (#2962) 2016-12-19 10:26:58 -08:00
cmasone-attic
9998ec0301 NBS: tableSet must exclude empty tables during ToSpecs() (#2959) 
Fixes #2957
 2016-12-15 11:45:39 -08:00
Dan Willhite
150d66010d Use merge policy to avoid dl-cache conflicts in downloader (#2958) 2016-12-15 10:21:12 -08:00
cmasone-attic
35385900d6 Close some NBS tableSets during testing (#2956) 
Leaving these open can leak file handles
 2016-12-14 17:17:58 -08:00
Rafael Weinstein
d2289219c3 dont flush for every batch during VBS.Enqueue (#2955) 2016-12-14 16:36:12 -08:00
Rafael Weinstein
f409db8c74 Add s3 tableIndex cache (#2953) 2016-12-14 15:06:31 -08:00
Rafael Weinstein
e242c9d168 Move concrete impls out of table_persister.go (#2952) 2016-12-14 12:48:05 -08:00
Rafael Weinstein
cc8ffacddf Factor tableIndex out of tableReader (#2950) 
Factor tableIndex out of tableReader
 2016-12-14 12:41:01 -08:00
Rafael Weinstein
d7ee0025d6 Open NBS tables in parallel (#2946) 2016-12-13 14:15:25 -08:00
Rafael Weinstein
373900c790 fix 2016-12-13 10:25:49 -08:00
Rafael Weinstein
c159876992 Make read amplification threshold configurable (#2941) 2016-12-13 09:57:41 -08:00
cmasone-attic
7f36fad716 tablePersister.Compact returns a chunkSource (#2939) 
It turns out the only caller of Compact() immediately
turns around and calls Open, so why don't I just do
that FOR you?

Fixes #2935
 2016-12-13 06:20:33 -08:00
Rafael Weinstein
ef4e6c48d3 AWSStoreFactory (#2938) 2016-12-12 15:55:56 -08:00
Rafael Weinstein
335454b34c ChunkSink.Flush() (#2937) 
Add ChunkSink.Flush() which signals the ChunkSink that any previously Put chunks should be made durable.
 2016-12-12 15:39:13 -08:00
cmasone-attic
de6e49c9e0 compactingChunkStore crash fix (#2936) 
compactingChunkStore.close() must wait for compactions to finish.
 2016-12-12 14:43:46 -08:00
cmasone-attic
7fe3b18a6b Make compaction async (#2934) 
Introduce a 'compactingChunkStore', which knows how to compact itself
in the background. It satisfies get/has requests from an in-memory
table until compaction is complete. Once compaction is done, it
destroys the in-memory table and switches over to using solely the
persistent table.

Fixes #2879
 2016-12-12 14:15:30 -08:00
Rafael Weinstein
0652e0b3e0 Add ChunkSource.GetMany(); RemoteBatchStore getRefs uses GetMany() (#2933) 
Add GetMany(), which most ChunkStores implement by repeated calls to their own Get(), but creates the opportunity for stores to optimize reads of larger blocks of potentially sequential chunks (e.g. NBS).

Add RemoteBatchStore getRefs endpoint support for calling GetMany() rather than Get()

Remove ReadThroughChunkStore which was dead code.
 2016-12-12 11:18:22 -08:00
Erik Arvidsson
80d4894fcc JS: Fix walk 
The promise was resolving too early.
 2016-12-09 18:51:49 -08:00
Aaron Boodman
fe59ae9504 Revert "Fix JS walk handling for values of type Type (#2913)" (#2931) 
This reverts commit 2717908d2b.
 2016-12-09 17:43:26 -08:00
Rafael Weinstein
88691863f4 Add simple read ahead for JS Collections (#2930) 
Adds the ability for SequenceCursors to eagerly load all child sequences when the first child is requested.

The effect is for uses where noms tends to forward scan, it will read in batches of ~150 chunks (e.g. ~512k) rather than one chunk at a time.

On my MBP, this improves the raw blob read perf over http from ~60K/s to ~5MB/s.
 2016-12-09 16:08:38 -08:00
Erik Arvidsson
741563958d Remove again and again 2016-12-09 12:46:54 -08:00
Erik Arvidsson
27267581d5 Again and again (#2929) 2016-12-09 12:16:05 -08:00
Erik Arvidsson
a59463e277 Remove testing again 2016-12-09 12:07:47 -08:00
Erik Arvidsson
e438145f14 Testing again (#2928) 2016-12-09 11:52:46 -08:00
Erik Arvidsson
559393f0b3 Revert "testing" (#2927) 2016-12-09 11:28:03 -08:00
Erik Arvidsson
c58a215415 testing (#2926) 2016-12-09 11:23:36 -08:00
Erik Arvidsson
bde471f690 JS: Do not validate struct type when reading a chunk (#2923) 
When we read a chunk and create structs we were validating that the
struct was of the type it claimed to be.

This now, no longer does that validation, which matches Go.
 2016-12-09 10:38:10 -08:00
Aaron Boodman
7a2c4f4984 Turn off chunk validation in production mode (#2925) 2016-12-09 10:32:13 -08:00
Erik Arvidsson
2717908d2b Fix JS walk handling for values of type Type (#2913) 
Fixes #2911
 2016-12-08 16:38:10 -08:00
cmasone-attic
dff8b67aba Change NBS s3TablePersister to use Multipart upload (#2922) 
Instead of putting an entire table to S3 in a single request, split it
into 5MB parts (the smallest allowable) and send all the parts in
parallel.
 2016-12-08 16:10:37 -08:00
Mike Gray
277f4c6384 Update to jenkins the third (#2920) 2016-12-08 10:33:45 -05:00
Aaron Boodman
914d252748 Move coffeebot back to 3pm (#2919) 2016-12-07 17:41:18 -08:00
Mike Gray
f29032c6cd Revert "Update links to jenkins3.noms.io" (#2918) 2016-12-07 17:36:51 -05:00
Mike Gray
5b9a863471 Update links to jenkins3.noms.io (#2916) 2016-12-07 17:22:43 -05:00
Dan Willhite
e30272abeb Implement poke functionality using diff.Apply function (#2828) 2016-12-07 11:57:48 -08:00
Aaron Boodman
8d4ff605f5 photo-index: output PhotoGroup instead of Photo (#2902) 
photo-index: output PhotoGroup instead of Photo (#2902)
 2016-12-06 16:21:02 -08:00
cmasone-attic
0750459e4e Add AWS backend for NBS (#2914) 
Add a new backend for NBS that stores tables in S3 and
manifests in DynamoDB.

Fixes #2877
 2016-12-06 15:12:28 -08:00
Ben Kalman
dc289245cb Add path @at for positional indexing, and negative indices (#2910) 
This lets you do foo.bar@at(n) to get the nth element of a list, set, or
map (for lists, this is equivalent to foo.bar[n]). This patch also adds
support for negative indices to @at(-n) and [-n] to get the nth element
relative to the back of collections.
 2016-12-05 14:43:41 -08:00
Rafael Weinstein
6edea9665e Verify chunks using suffix index not computing address from data. (#2907) 
Revert to verifying chunks using the suffix index. Replace the inline 4-byte suffix used as integrity check with a more standard and efficient CRC32.
 2016-12-05 11:44:43 -08:00
Dan Willhite
7916e05631 Make blob-get print to stdout if there is no file arg (#2867) 
* Make blob-get print to stdout if there is no file arg

* Review changes
 2016-12-05 11:33:08 -08:00
cmasone-attic
b3eef38fa4 Break NomsBlockStore dependency on disk storage (#2905) 
This patch introduces/expands the 'manifest' and 'tableSet'
abstractions, so that NomsBlockStore is no longer explicitly using any
file system operations

Towards issue #2877
 2016-12-05 09:05:40 -08:00
Ben Kalman
0a10704149 Implement Set.At and Map.At in Go (#2903) 
These get the set/map element at a specific index.
I haven't implemented it in JS yet because the JS code has no method to
create a cursor at an index. This exists in Go because a refactor was
done a few months ago to add it, but it hasn't been ported to JS.
 2016-12-04 11:27:35 -08:00
Aaron Boodman
87958507b0 marshal.Unmarshal(): introduce omitempty and original support (#2900) 
* marshal.Unmarshal(): introduce omitempty and original suppport.

Fixes #2795
Fixes #2796

* review comments
 2016-12-02 15:13:00 -08:00
Rafael Weinstein
a67bb9bf7b Minor rework of hash.Hash API (#2888) 
Define the hash.Hash type to be a 20-byte array, rather than embed one. Hash API Changes: `hash.FromSlice` -> `hash.New`, `hash.FromData` -> `hash.Of`
 2016-12-02 12:11:00 -08:00