mirror of
https://github.com/dolthub/dolt.git
synced 2026-01-31 03:18:43 -06:00
338de4e583
Previously the buzhash boundary checker used a single value for the window size, both as the buzhash buffer size when constructing a hash object, and reported as its window size to the boundary checker interface. This was wrong because we don't always pass single byte values to the hasher, for example refs are 20 bytes. The compound list chunking compensated for this by only passing the first byte of each list leaf's ref rather than the full ref. This is bad because there is obviously less entropy in 1 byte vs 20 bytes. The meta sequence chunking compensated for this by multiplying the chunking window size by 20, but this also had the effect of unnecessarily considering 20 times more chunked elements than would fit in the buzhash buffer.
Noms
Noms is a content-addressable, immutable, peer-to-peer datastore for structured data.
In other words, noms is git for data.
This repository will contain the reference implementation of the noms protocol, and will eventually be open sourced.
This includes:
- Go wrappers for all the core noms types
- Support for generating Go types from Nom schema definitions
- Chunking and dechunking
- Serialization and deserialization
- Chunkstore interface as well as several sample implementations
- Search support
- Sample applications
Get the code
git clone https://github.com/attic-labs/noms
Build
go build ./...
go test ./...
Run
cd <noms>/clients/counter
go build
./counter -ldb=/tmp/foo -ds=foo
./counter -ldb=/tmp/foo -ds=foo
./counter -ldb=/tmp/foo -ds=foo
rejoice!
You can see the raw data:
ls /tmp/foo
cat /tmp/foo/*.log | strings
You can also explore the data visually. Follow the instructions in clients/splore.
There are lots of other sample programs in clients/ and they usually have READMEs. Have fun...
TODO: There needs to be more of a big-picture introduction.