Rafael Weinstein
ee47fb0bf2
Reland: limit write concurrency of leveldb to max open file handles limit
2015-09-23 11:54:21 -07:00
Rafael Weinstein
b661beceae
Revert "limit write concurrency of leveldb to max open file handles limit"
...
This reverts commit 8d2d380676
2015-09-23 11:49:00 -07:00
Rafael Weinstein
8d2d380676
limit write concurrency of leveldb to max open file handles limit
2015-09-23 11:32:02 -07:00
Rafael Weinstein
adabda61a4
Make DataStore an interface
2015-09-22 17:30:14 -07:00
Rafael Weinstein
591026bcfa
Chunk
2015-09-21 13:24:41 -07:00
Rafael Weinstein
8d56b8c968
Reland: ChunkSource.Get() now returns a []byte. chunks.Serialize takes a stream of Chunks
2015-09-17 12:31:25 -07:00
Rafael Weinstein
a62cf1bbb8
Revert "ChunkSouce.Get() now returns []byte"
...
This reverts commit 2be269acf4
2015-09-16 20:47:36 -07:00
Rafael Weinstein
2be269acf4
ChunkSouce.Get() now returns []byte
2015-09-16 16:42:36 -07:00
Rafael Weinstein
0df9f46a2e
Remove http_store (someone got lost in recent patch)
2015-09-16 12:55:55 -07:00
Rafael Weinstein
9c6effe19f
split chunks/http_store into http/http_client & http/http_server
2015-09-16 10:56:43 -07:00
Rafael Weinstein
10350dbd7e
switch js to use buffer reads
2015-09-11 14:41:51 -07:00
Rafael Weinstein
ae4e35d9d1
possible fix for continuous build
2015-09-11 12:18:58 -07:00
Rafael Weinstein
a877c6fac9
use MemoryStore rather than ChunkBuffer
2015-09-10 12:48:27 -07:00
Rafael Weinstein
cdcd60c8ea
Short-term fix for parallel writes
2015-09-09 18:11:22 -07:00
Rafael Weinstein
9bac46b7f3
better read buffering
2015-09-09 10:58:29 -07:00
Rafael Weinstein
89fa402ea1
buffer pending reads
2015-09-08 17:43:32 -07:00
Aaron Boodman
326dfb7953
ldbstore: Reduce the max number of open file handles, but make configurable.
2015-09-03 13:21:46 -07:00
Rafael Weinstein
68ce9c1391
Merge pull request #253 from rafael-atticlabs/fixHttpStoreTest
...
fix http_store test
2015-09-03 05:39:56 -07:00
Rafael Weinstein
b58e20eb8c
fix http_store test
2015-09-02 13:20:34 -07:00
Rafael Weinstein
45c6bc586b
limit number of open files for leveldb store
2015-09-02 13:13:14 -07:00
Rafael Weinstein
d7e73f8ea8
Put back CORS directive (broke in change to ServeMux
2015-08-30 20:48:57 -07:00
Rafael Weinstein
ae7dee8eaa
buffer http puts
2015-08-30 10:48:08 -07:00
Rafael Weinstein
925d268f0b
Use built-in ServeMux
2015-08-28 11:30:06 -07:00
Rafael Weinstein
834445954a
Fully shutdown server
2015-08-28 10:51:59 -07:00
Rafael Weinstein
29af6c47a8
Fix chunkstore breakage after removing AWS/FileStore
2015-08-27 13:51:47 -07:00
Rafael Weinstein
3deeae09df
Remove AWSStore, FileStore, StdoutWriter and deps
2015-08-27 13:44:22 -07:00
Rafael Weinstein
55c9f7a6f2
chunk_writer shouldnt decide to check whether a chunk exists before writing it
2015-08-27 13:07:00 -07:00
Rafael Weinstein
a21d121d71
Dont bind directly to loopback addr; TBR=aa
2015-08-27 10:27:44 -07:00
Dan Willhite
ab34143ba5
Pin dependencies using godep tool. Rewrite dep urls.
2015-08-26 14:05:40 -07:00
Rafael Weinstein
abc7102ad6
Clean following HttpStore implementation
2015-08-24 10:57:50 -07:00
Rafael Weinstein
0b57427496
Remove default hostname for httpstore
2015-08-21 14:34:23 -07:00
Rafael Weinstein
f6490cbbe5
Add HttpStore implementation
2015-08-21 14:22:45 -07:00
Rafael Weinstein
b6ebb67e91
Fix AWS update breakage
2015-08-21 14:21:04 -07:00
Rafael Weinstein
06c5bc6c1b
Abstract ChunkStoreWriter
2015-08-20 10:58:41 -07:00
Rafael Weinstein
94e7c3b73a
Abstract a bunch of (haphazzardly) duplicated tests into chunk_store_test
2015-08-19 18:23:09 -07:00
Rafael Weinstein
0e7d61efc6
Remove errors from ChunkStore and Ref
2015-08-18 16:24:26 -07:00
Rafael Weinstein
0222b95ff2
LevelDBStore
2015-08-18 11:20:17 -07:00
Aaron Boodman
214b37eccf
Remove global imports of dbg package
...
Fixes #179
2015-08-08 23:57:37 -07:00
Rafael Weinstein
ee9a9ae03d
Fix compile breakage as a result of updating aws import
2015-08-04 15:44:40 -07:00
Aaron Boodman
82523740fb
Merge pull request #151 from aboodman/iocopy
...
Use io.Copy in fileChunkWriter rather than Bytes()
2015-07-29 15:26:33 -07:00
Aaron Boodman
3e472d447b
Use io.Copy in fileChunkWriter rather than Bytes()
...
Fixes #145 .
I looked for other occurences via `git grep Bytes()`. Only other
suspicious cases I found were related to types.Blob, which doesn't
deal in io.Reader yet, and memoryChunkWriter. In memoryChunkWriter
we're copying from a buffer, so I don't think there's any win, so
left that one.
2015-07-29 15:21:17 -07:00
Aaron Boodman
4488d6b4c6
Abbreviate chunkstore related flags
...
We type them a lot and the long names are annoying.
2015-07-29 15:01:25 -07:00
Chris Masone
d1e1fb4f01
Revise some comments per feedback
2015-07-29 09:22:56 -07:00
Chris Masone
5a5ba47fb4
Add comments to DataStore and ChunkStore
...
Towards issue #136
2015-07-28 16:01:18 -07:00
Chris Masone
f47d117bf2
Add pull command line tool, test for pulling an initial commit, address comments
...
In addition to putting in the 'pull' tool that I forgot to add in my initial PR,
I added an extra unit test to cover a case that we found to be buggy, as well
as addressing some comments by aa and arv.
1) Switched to io.Copy in CopyChunks
2) Added NewFlagsWithPrefix()
3) Cleaned up some error reporting
2015-07-28 11:21:45 -07:00
Chris Masone
0d830e3c1d
Pull from one ChunkStore to another
...
This initial implementation requires that both the "remote" and local
ChunkStores be accessible by the machine running the pull command.
I took an initial pass at splitting up the functions so that, e.g.,
calculating which refs are needed could be done on an actual remote
machine, and we can add a chunk copying routine that gets data from
the network or something.
Towards issue #81
2015-07-27 11:41:18 -07:00
Erik Arvidsson
1b23a779a1
Merge pull request #94 from arv/read-through-store
...
Add a ReadThroughStore which caches the values in a caching store
2015-07-21 14:18:29 -07:00
Erik Arvidsson
77681f78fe
Add a ReadThroughStore which caches the values in a caching store
2015-07-21 14:15:59 -07:00
Erik Arvidsson
f08e7f0f5e
Make sure that FileStore returns no error for missing refs
2015-07-21 13:13:46 -07:00
Erik Arvidsson
8b566139d0
Fix MemoryStore to get the ref in close
2015-07-21 09:37:07 -07:00
