Commit Graph

26 Commits

Author SHA1 Message Date
Rafael Weinstein adabda61a4 Make DataStore an interface 2015-09-22 17:30:14 -07:00
Rafael Weinstein 591026bcfa Chunk 2015-09-21 13:24:41 -07:00
Rafael Weinstein 8d56b8c968 Reland: ChunkSource.Get() now returns a []byte. chunks.Serialize takes a stream of Chunks 2015-09-17 12:31:25 -07:00
Rafael Weinstein a62cf1bbb8 Revert "ChunkSouce.Get() now returns []byte"
This reverts commit 2be269acf4.
2015-09-16 20:47:36 -07:00
Rafael Weinstein 2be269acf4 ChunkSouce.Get() now returns []byte 2015-09-16 16:42:36 -07:00
Rafael Weinstein 9c6effe19f split chunks/http_store into http/http_client & http/http_server 2015-09-16 10:56:43 -07:00
Rafael Weinstein 10350dbd7e switch js to use buffer reads 2015-09-11 14:41:51 -07:00
Rafael Weinstein a877c6fac9 use MemoryStore rather than ChunkBuffer 2015-09-10 12:48:27 -07:00
Rafael Weinstein 9bac46b7f3 better read buffering 2015-09-09 10:58:29 -07:00
Rafael Weinstein 29af6c47a8 Fix chunkstore breakage after removing AWS/FileStore 2015-08-27 13:51:47 -07:00
Rafael Weinstein f6490cbbe5 Add HttpStore implementation 2015-08-21 14:22:45 -07:00
Rafael Weinstein 06c5bc6c1b Abstract ChunkStoreWriter 2015-08-20 10:58:41 -07:00
Rafael Weinstein 0e7d61efc6 Remove errors from ChunkStore and Ref 2015-08-18 16:24:26 -07:00
Rafael Weinstein 0222b95ff2 LevelDBStore 2015-08-18 11:20:17 -07:00
Chris Masone d1e1fb4f01 Revise some comments per feedback 2015-07-29 09:22:56 -07:00
Chris Masone 5a5ba47fb4 Add comments to DataStore and ChunkStore
Towards issue #136
2015-07-28 16:01:18 -07:00
Chris Masone f47d117bf2 Add pull command line tool, test for pulling an initial commit, address comments
In addition to putting in the 'pull' tool that I forgot to add in my initial PR,
I added an extra unit test to cover a case that we found to be buggy, as well
as addressing some comments by aa and arv.

1) Switched to io.Copy in CopyChunks
2) Added NewFlagsWithPrefix()
3) Cleaned up some error reporting
2015-07-28 11:21:45 -07:00
Chris Masone 0d830e3c1d Pull from one ChunkStore to another
This initial implementation requires that both the "remote" and local
ChunkStores be accessible by the machine running the pull command.
I took an initial pass at splitting up the functions so that, e.g.,
calculating which refs are needed could be done on an actual remote
machine, and we can add a chunk copying routine that gets data from
the network or something.

Towards issue #81
2015-07-27 11:41:18 -07:00
Erik Arvidsson f08e7f0f5e Make sure that FileStore returns no error for missing refs 2015-07-21 13:13:46 -07:00
Chris Masone a22b9de144 Changes to xml_importer.go to allow for naive benchmarking
Introduce the 'NopStore', which is a sorta-ChunkStore that throws away
all written data. Get() can't really be called on it, as a result,
but it's useful for taking IO totally out of the equation when testing
our code for performance.

Towards issue #67
2015-07-20 10:08:00 -07:00
Erik Arvidsson e37b50f7a2 Rename s3Store to awsStore
Fixes #38
2015-07-15 18:00:14 -07:00
Aaron Boodman c364850618 Fix some bone-headed-ness in s3 flag handling. 2015-07-14 18:05:58 -07:00
Aaron Boodman cb1fb984b3 Remove distinction between RootTracker and ChunkStore in DataStore 2015-07-05 23:05:51 -07:00
Aaron Boodman fdebc00dc9 Add first app: clients/counter! 2015-06-19 15:23:02 -07:00
Aaron Boodman 4815998aa9 Add AppRootTracker and test 2015-06-18 17:59:48 -07:00
Aaron Boodman b33cab2b34 Rename package 'store' to 'chunks'.
This is in preparation for renaming the 'Commit' abstraction to
DataStore. So we will essentially have a 'chunkstore' abstraction
and a 'datastore' abstraction.
2015-06-16 16:19:16 -07:00