#Store All the Things
Noms is a content-addressed, immutable, decentralized, strongly-typed database.
In other words, Noms is Git for data.
This repository contains two reference implementations of the database—one in Go, and one in JavaScript. It also includes a number of tools and sample applications.
Setup
git clone https://github.com/attic-labs/noms $GOPATH/src/github.com/attic-labs/noms
go install github.com/attic-labs/noms/cmd/...
noms log http://demo.noms.io/cli-tour::film-locations
Samples | Command-Line Tour | JavaScript SDK Tour | Intro to Noms
Features
| Versioning Each commit is retained and can be viewed or reverted | Type inference Each dataset has a precise schema that is automatically inferred | Atomic commits Immutability enables atomic commits of any size |
| Diff Compare structured datasets of any size efficiently | Schema versioning Narrow or widen schemas instantly, without rewriting data | Sorted indexes Fast range queries, on a single or a combination of attributes |
| Fork Create your own isolated branch of a dataset to work on | Schema validation (soon) Optionally constrain commit types on a per-dataset basis | Insanely easy import Noms auto-dedupes snapshots and generates a precise changelog |
| Sync Sync disconnected database instances efficiently and correctly | Structural typing Index, search, and match data by structure shape | Awesome export Use dataset history to precisely apply sync changes out of Noms |
Use Cases
We're just getting started, but here are a few use cases we think Noms is especially well-suited for:
Data Collaboration
Work on data together. Track changes, fork, merge, sync, etc. The entire Git workflow, but on large-scale, structured or unstructured data. Useful for teams doing data analysis, cleansing, enrichment, etc.
ETL
ETL based on Noms is naturally:
- Incremental: Noms datasets can be efficiently diffed, so only the changed data needs to be run through the pipeline.
- Versioned: Any transform can be compared to the previous run and trivially undone or re-applied.
- Idempotent: If a transform fails in the middle for any reason, it can simply be re-run. A transform's result will always be the same no matter how many times it is run.
- Auditable: Content-addressing enables precisely tracking inputs to each transform and result.
Data Integration and Enrichment
Use Noms as a central place to collect, integrate, index, and integrate data from disparate sources.
Noms naturally deduplicates all data, so import can be trivially simple - just dump coarse-grained snapshots periodically and only reprocess the changes.
Model metadata non-destructively, as verioned, revertable assertions from source object to metadata.
Decentralized database
Noms is a natural fit for moving structured data around widely distributed or decentralized applications. Rather than moving raw data files, e.g., with rsync, and then rebuilding the database at each node, just move the database itself.
Get Involved
Noms is developed in the open. Come say hi.