Commit Graph

224 Commits

Author SHA1 Message Date
Aaron Boodman fd7c1cc14d Make dropbox work with the photos sample (#2745)
* dropbox/find-photos: encode auth token in photo URLs so they can
work in UI.

* Remove requirement for datePublished from photo-index

Dropbox doesn't have a publish date

* fix test

* review comments

* npm test
2016-10-21 14:49:03 -07:00
Ben Kalman cf296b48ca Fix photo set intersection bug (#2746) 2016-10-21 14:28:42 -07:00
Aaron Boodman 3afbc18079 picasa/slurp: use the redirect-less flow instead (#2732)
picasa/slurp: use the redirect-less flow instead
2016-10-20 21:32:41 -07:00
Aaron Boodman b9e104bfbd Fix date grouping logic (#2740)
Fix timestamp grouping
2016-10-20 09:49:45 -07:00
Aaron Boodman 7099c89bc9 Introduce the concept of an 'id' to several samples (#2738) 2016-10-20 09:49:22 -07:00
Aaron Boodman 02d0a8e8f7 ['foo'] -> .foo (#2736) 2016-10-20 06:51:40 -07:00
zcstarr d055fb5a37 Refactors face rectangle to face center point (#2737) 2016-10-19 15:37:07 -07:00
Erik Arvidsson e164f8aeec Flow header after copyright (#2734)
This puts the flow header after the copyright header.

It also:
 * fixes the existing files to have valid headers
 * Makes sure the script can handle doctype
2016-10-19 11:36:48 -07:00
Aaron Boodman 3adab7a874 picasa/find-photos (#2714)
* Introduce picasa/find-photos

* npm test
2016-10-19 09:07:49 -07:00
Aaron Boodman 067dc268ae Only dedupe photos by dateTaken, not datePublished (#2730)
Only dedupe photos by dateTaken, not datePublished

It's very common for groups of photos to have same datePublished
2016-10-18 17:39:47 -07:00
Aaron Boodman 4b843b8600 Introduce dropbox/find-photos (#2712)
Introduce dropbox/find-photos
2016-10-16 21:37:45 -07:00
Aaron Boodman 4ae6b16ac4 picasa/slurp: command-line flags should use dash not underscore (#2713) 2016-10-14 21:18:49 -07:00
Aaron Boodman f487358046 dropbox/slurp: path is optional (#2710) 2016-10-14 20:59:55 -07:00
Aaron Boodman 233af18525 dropbox/slurp: add ability to restrict import to a path prefix (#2709) 2016-10-14 14:24:52 -07:00
Ben Kalman 35495aa69a Add Picasa slurp importer (#2701) 2016-10-13 21:51:22 -07:00
Aaron Boodman 8fe85df1a4 Introduce samples/js/dropbox/slurp (#2702)
* Introduce samples/js/dropbox/slurp

Doing the filter step will be a bit different because Dropbox does
not provide secret URLs for every photo by default. Requests have
to be oauth-authenticated so we'll have to figure out how to do
that.

* vendor golang.org/x/oauth2 and golang.org/x/net

* Introduce oauthify and modify dropbox/slurp to use
2016-10-13 17:34:37 -07:00
Erik Arvidsson 5619d2b3ac Update Flow to 0.33 (#2692) 2016-10-12 14:17:49 -07:00
Aaron Boodman 5191b136c2 fb/slurp: fix eslint error (#2697) 2016-10-12 08:58:20 -07:00
Aaron Boodman 71b949489c fb: fix paging logic (#2691)
We didn't notice this before because none of us had enough fb data
to cause it to page.
2016-10-11 17:09:17 -07:00
cmasone-attic 23f94bdf8c demo-server: Remove cachingReadThroughChunkStoreFactory (#2689)
demo-server was using a read-through cache to allow it to serve more
concurrent requests more quickly by reducing disk I/O. As seen in issue
across Databases, leading to incorrect sync results in some instances.

Since we're not worried about demo-server load right now, simply delete
the cache.

Fixes #2688
2016-10-10 14:14:54 -07:00
Ben Kalman 70a24733aa Better error messages when index path is wrong in photo viewer (#2672) 2016-10-10 12:45:31 -07:00
Erik Arvidsson 7783fc5f44 Update style issues found by newer eslint (#2685) 2016-10-10 10:56:02 -07:00
Erik Arvidsson 8bd3d0720c Update the instructions for fb/slurp (#2686) 2016-10-08 10:14:09 -07:00
Aaron Boodman fe922da8ed photo-index: tag commit with run date (#2687)
Would be nice to also tag with pinned path, but that's more work
2016-10-08 01:03:36 -07:00
Aaron Boodman eb00815de4 Flickr: import primarily photo-wise rather than album-wise. (#2682)
Flickr: import primarily photo-wise rather than album-wise.

This way we get photos that aren't in any album.

Fixes: #2675
2016-10-07 17:03:01 -07:00
Erik Arvidsson 8b40a9997d Fix Flickr importer to get an access token (#2681)
We were using the short lived access token when we should have
exchanged that for a long term access token.

Also refactor the code a bit. Sorry!
2016-10-07 15:41:36 -07:00
Mike Gray 67ad849505 Update to Facebook API v2.8 (#2680) 2016-10-07 17:25:12 -04:00
Aaron Boodman 53d5b59017 Track run date and input paths for fb and flickr importers (#2673)
Track run date and input paths for fb and flickr importers
2016-10-06 18:52:30 -07:00
Aaron Boodman 2708e127b1 fb/slurp: add support for getting longer-lived access token (#2670) 2016-10-05 21:16:36 -07:00
Ben Kalman 49e74de710 Add a photo viewing web UI (#2662) 2016-10-05 12:49:56 -07:00
Ben Kalman 4a562d81ef Add tag and face counts to the photo index (#2660) 2016-10-03 11:52:42 -07:00
Eric Halpern 574bdd8483 Make config.Resolve methods public and make photo-index .nomsconfig aware (#2661) 2016-10-03 10:03:11 -07:00
zcstarr 40b28f94e5 Refactor Chunks and ChildValues API to work iteratively (#2599)
* Refactors Chunks and ChildValues API to be iterative change also
exposes WalkValues which replaces SomeP and AllP
2016-09-30 16:53:00 -07:00
Aaron Boodman 596a91d341 Add face indexing to photo-index (#2654) 2016-09-29 13:27:04 -07:00
Dan Willhite 885dd5a21c Add tx-regex, tx-replace, and tx-convert args. (#2651)
Also, make help text more complete.
2016-09-29 13:03:53 -07:00
Ben Kalman 9f5725bd27 Run url-fetch perf tests against a fresh database each time
The subsequent runs of url-fetch on jenkins are way faster, and this
appears to be because commiting is much faster on subsequent runs. The
perf tests now use a new database each time.
2016-09-29 12:49:37 -07:00
Erik Arvidsson a63b588f80 Fix lint errors (#2650) 2016-09-29 10:44:43 -07:00
Aaron Boodman 48e7ed9850 fb/find-photos: add in datePublished, dateUpdated, faces (#2634)
fb/find-photos: add in publishDate, updateDate, along with basic face info.
2016-09-28 17:18:46 -07:00
cmasone-attic dd92a06559 JS: Make Database a mutable API that vends immutable Datasets (#2636)
Noms SDK users frequently shoot themselves in the foot because they're
holding onto an "old" Database object. That is, they have a Database
tucked away in some internal state, they call Commit() on it, and
don't replace the object in their internal state with the new Database
returned from Commit.

This PR fixes #2589 by changing the Database and Dataset JS API to be
in line with the proposal there.
2016-09-28 16:50:57 -07:00
Ben Kalman a738ad2d85 flickr correct nsInSecond/nsInMillisecond (#2645) 2016-09-28 16:45:26 -07:00
Aaron Boodman 3efc6c5f7d You're the computer, you sort the fields (#2641)
Change makeStructType() API for sanity. Update callers.
2016-09-28 15:39:52 -07:00
Dan Willhite 4fd84d5141 Add dummy files to js samples with integration tests (#2643) 2016-09-28 13:59:22 -07:00
Dan Willhite 82b25f370f Remove pitch-index from samples/js (#2642) 2016-09-28 13:42:50 -07:00
Dan Willhite 7d81c9b96b Add integration test for sample: url_fetch (#2639)
Towards #1888
2016-09-28 13:36:19 -07:00
Dan Willhite cd8f995b2c Add integration test for sample: aggregate. (#2633)
Towards #1888
2016-09-28 12:52:55 -07:00
Eric Halpern d9715dba0e Support db aliases and default db for noms cli
This patch implements evolving support for configuring aliases and defaults for the noms cli (started with #2131)

For an introduction, please take a look at the sample code here: https://github.com/attic-labs/noms/blob/master/samples/cli/nomsconfig/README.md

Improvements include: 

 - All go samples now work with .nomsconfig
 - Absolute paths in ldb specs are now properly handled 
 - Add -v|--verbose flag to commands to debug expansion
 - Make default just another alias and change [default] section to [db.default]
 - Introduce the `.` shorthand to refer to a previously mentioned dataset/object
2016-09-27 22:21:32 -07:00
Ben Kalman 81673c2591 Add perf test for url-fetch 2016-09-27 16:52:54 -07:00
Aaron Boodman 577c99ff38 Factor out datas.ReadAbsolutePaths() (#2623) 2016-09-27 14:21:54 -07:00
Aaron Boodman e52775f838 Refactor exit mockery into go/util/exit (#2622) 2016-09-27 13:51:27 -07:00
Ben Kalman 097863ea6f Use parallel NewBlob in the csv perf tests (#2625)
This just involves changing types.NewBlob(io.MultiReader(files...)) to
types.NewBlob(files...). On my laptop it improves
Test01ImportSfCrimeBlobFromTestdata from 21s to 16s - though much of
this is dominated by commit, which wouldn't be affected by this change.
2016-09-27 12:22:25 -07:00