Remove validation/normalization of union order and struct field order as we decode a chunk into a type.
Instead the validation happens in ValidatingBatchSink.
We still normalize the union order when a struct type is created directly (not from a chunk) using makeStructType.
The motivation for this change is that computing the OID (order ID) is expensive and it used to be a O(n^2) since we kept recomputing it as we traversed the type hierarchy.
Towards #2836
This patch introduces optional debug logging in util/verbose, and adds
some usage of it to HandleWriteValue and the httpBatchStore
SchedulePut code path. It also modifies chunks.DeserializeToChan() so
that callers can better recover from panics in there.
https://github.com/attic-labs/attic/issues/103
Introduce photo-dedup-by-date
This program deduplicates photos by the date they were taken. It considers two photos a group if they were separated by less than 5 seconds.
* Implement read-ahead in sequence_cursor
For each meta-sequence that contains leaf sequences, start reading ahead in
parallel and deliver in order to a buffered channel. Each advance of the cursor gets
the next sequence in the read-ahead channel.
toward: #2079
-
* Address code review comments:
- Use // for all comments
- Fix label format
- Increase channel read timeout
* Rework read-ahead to use map[int]channel sequence instead of a channel of sequences
* Rework sequence cursor read-ahead for better throughput
- Guts of read-ahead now encapsulted in sequenceReadAhead
- New implemention uses a cursor to iterate across the leaves ahead
of the current cursor
- It reads ahead using short-lived go routines that place each read-ahead
sequence in a channel that is then stored by hash in a map
- When the sequence is needed, the cursor first looks in the map. If found,
it reads the sequence from the channel stored in the map. If not, it reads
it normally.
- This approach allows for reading ahead in parallel without requiring a long
running pool of goroutines
- Introduce sequenceIterator to encapulate read-ahead behind an abstraction that
always reads forward. This is currently used narrowly but could be used more
widely as the the core implementation for all sequence iterators
* Address review comments
If you want to roll just the go/ directory of noms, you can do:
$ roll.py https://github.com/attic-labs/noms --incl go
If you want to roll the AWS SDK without the tests, you can do:
$ roll.py https://github.com/aws/aws-sdk-go --excl awstesting
This works with nested directories too, for example --incl go/util
--excl is evaluated after --incl, so you could exclude the perf/
directory of go/ if you really wanted:
$ roll.py https://github.com/attic-labs/noms --incl go --excl go/perf
This is a side-by-side port, taking inspiration from the old dataspec.go
code. Notably:
- LDB support has been added in Go. It wasn't needed in JS.
- There is an Href() method on Spec now.
- Go now handles IPV6.
- Go no longer treats access_token specially.
- Go now has Pin.
- I found some issues in the JS while doing this, I'll fix later.
I've also updated the config code to use the new API so that basically
all the Go samples use the code, even if they don't really change.
There were several tests in the Database suites that were failing to
close test Databases that had orderedChunkCaches in them (backed by
levelDBs). Close them.
I was ALSO failing to destroy the cache used in LocalDatabase
instances only while testing Pull(). That's cleared up now as well.
ValueStore caches Values that are read out of it, but it doesn't
do the same for Values that are written. This is because we expect
that reading Values shortly after writing them is an uncommon usage
pattern, and because the Chunks that make up novel Values are
generally efficiently retrievable from the BatchStore that backs
a ValueStore. The problem discovered in issue #2802 is that ValueStore
caches non-existence as well as existence of read Values. So, reading
a Value that doesn't exist in the DB would result in the ValueStore
permanently returning nil for that Value -- even if you then go and
write it to the DB.
This patch drops the cache entry for a Value whenever it's written.
Fixes#2802
The httpBatchStore test TestVersionMismatch() expects a panic,
but the test was actually potentially causing MULTIPLE panics.
One due to the version mismatch, and another due to using an
invalid root.
* Add jobs for grouping similar photos in PhotoGroups
Outline:
- The first photo-dhash job adds a dhash field to each photo. The dhash is a 128 bit
downsampled representation of the photo that works well for visual similarity comparisions.
- The second photo-dedup job groups photos that have similar dhash's into PhotoGroups.
fixes: #2787
The big change here is adding a new Spec class in spec.js. This replaces
DatabaseSpec/DatasetSpec/PathSpec in specs.js, but I'm leaving those in
and moving code over in a later patch. For now, only photos UI.
The photos UI change is to plumb through the authorization token through
the Spec code. For now, it's reading it from a URL parameter, but soon
I'll make it session based (probably localStorage).
The demo-server change is to add the Authorization header into CORS.
Private databases begin with "/p/" - for example, "/kalman" is not
private, but "/p/kalman" is private. They are not the same database.
The bulk of this work is the receipt infrastructure.
A receipt is form data that gives access to a database, encrypted using
secretbox. For example, "Database=/p/kalman&Date=12345678" might encrypt
to "SFH5bcIJ3_XgEbtmi_AdCKTItW20fl90czVl5_pF5PAXhNQ366U1yOpYGAjT".
* A new tool receiptkey generates random receipt (secretbox) keys.
* A new tool receipttool generates receipts for databases.
* demo-server has been updated to check for a receipt in the
Authorization header to access private databases.
receipttool and demo-server must be given the same receipt key.
Aaron Boodman