diff --git a/README.md b/README.md index 0a45250d88..d7f46d4052 100644 --- a/README.md +++ b/README.md @@ -1,26 +1,20 @@ # Noms -Noms is a content-addressable, immutable, peer-to-peer datastore for structured data. +Noms is a content-addressable, append-only, peer-to-peer, structured data store. In other words, *noms is git for data*. -This repository will contain the reference implementation of the noms protocol, and will eventually be open sourced. - -This includes: - -* Go wrappers for all the core noms types -* Support for generating Go types from schema definitions (aka 'nomdl') -* Chunking and dechunking -* Serialization and deserialization -* Chunkstore interface as well as several sample implementations -* Search support -* Sample applications +This repository contains two reference implementations of the noms protocol - one in Go, and one in JavaScript. It also includes a number of tools and sample applications. # Prerequisites * [Go 1.4+](https://golang.org/dl/) +* [Python 2.7+](https://www.python.org/downloads/) (Note: Python 2.x only, not Python 3.x) +* [Node.js 5.3+](https://nodejs.org/download/) -# Get +# Get the code + +First, ensure `$GOPATH` is [set correctly](https://golang.org/doc/code.html#GOPATH). Then: ``` go get -u -t github.com/attic-labs/noms/... @@ -43,17 +37,10 @@ go build ./counter -ldb=/tmp/foo -ds=foo ``` -# Rejoice! +# What next? -You can see the raw data: - -``` -ls /tmp/foo -cat /tmp/foo/*.log | strings -``` - -You can also explore the data visually. Follow the instructions in `clients/splore`. - -There are lots of other sample programs in `clients/` and they usually have `README`s. Have fun... - -TODO: There needs to be more of a big-picture introduction. +* Learn the core tools: [`server`](clients/server/README.md), [`splore`](clients/splore/README.md), [`shove`](clients/shove/README.md), [`csv_importer`](clients/csv_importer/README.md), [`json_importer`](clients/json_importer), [`xml_importer`](clients/xml_importer) +* Run sample apps: [`sfcrime`](clients/sfcrime/README.md), [`tagshow` photo viewer](clients/tagshow/README.md) +* NomDL reference +* Go SDK +* JavaScript SDK diff --git a/clients/csv_importer/README.md b/clients/csv_importer/README.md index 46c7b98e66..c0332cbf0b 100644 --- a/clients/csv_importer/README.md +++ b/clients/csv_importer/README.md @@ -1,7 +1,6 @@ # CSV Importer -Imports a CSV file as a List of Maps where the first row of the CSV file -describes the keys of the Map. +Imports a CSV file as List where T is generated from the header row of the CSV (this can also be overridden with the `-header` flag). ## Usage diff --git a/clients/server/README.md b/clients/server/README.md index 283cad01f6..f72b0c8bfc 100644 --- a/clients/server/README.md +++ b/clients/server/README.md @@ -1,16 +1,30 @@ # Server -This is an http server for noms. Currently it is read-only. +Server implements a noms datastore over HTTP. -## Build +## Example ``` -cd /clients/server +cd $GOPATH/src/github.com/attic-labs/noms/clients/counter go build +./counter -ldb="/tmp/servertest" -ds="counter" +./counter -ldb="/tmp/servertest" -ds="counter" +./counter -ldb="/tmp/servertest" -ds="counter" + +cd ../server +go build +./server -ldb="/tmp/servertest" & ``` -## Run +Then navigate a web browser to [http://localhost:8000/root]. You should see a string starting with `sha1-...`. This _ref_ is the unique identifier for the current state of the datastore. You an explore it further by fetching URLs like [http://localhost:8000/ref/sha1-...]. + +## About + +Server is not commonly used directly by users, but is a building block used by other tools. For example, you can connect the counter application to your running server like so: + ``` -# See -h for more options -./server -ldb="/tmp/foo" +./counter -h="http://localhost:8000" -ds="counter" +./counter -h="http://localhost:8000" -ds="counter" ``` + +Most noms clients accept this `-h` flag to connect to an http datastore. diff --git a/clients/shove/README.md b/clients/shove/README.md new file mode 100644 index 0000000000..0275db545f --- /dev/null +++ b/clients/shove/README.md @@ -0,0 +1,28 @@ +# Shove + +Shove syncs between datastores and datasets. It is the noms equivalent of Git's `push` and `pull` commands. + +## Howto + +``` +cd $GOPATH/src/github.com/attic-labs/noms/clients/counter +go build +./counter -ldb="/tmp/shovetest1" -ds="counter" +./counter -ldb="/tmp/shovetest1" -ds="counter" +./counter -ldb="/tmp/shovetest1" -ds="counter" + +cd ../shove +go build +./shove -source-ldb="/tmp/shovetest1" -source="counter" -sink-ldb="/tmp/shovetest2" -sink-ds="counter2" +../counter/counter -ldb="/tmp/shovetest2" -ds="counter2" + +# Shove can also connect to http datastores +cd ../server +go build +./server -ldb="/tmp/shovetest2" & + +../shove/shove -source-h="http://localhost:8000" -source="counter2" -sink-ldb="/tmp/shovetest3" -sink-ds="counter3" +../counter/counter -ldb="/tmp/shovetest3" -ds="counter3" +``` + +There are currently a small collection of datasets you can sync available at `-h="ds.noms.io"`. You can browse them at [http://apps.noms.io/splore](http://apps.noms.io/splore) (username: attic, password: labs). \ No newline at end of file diff --git a/clients/splore/README.md b/clients/splore/README.md index 91b1bdedc9..0b5dddde0c 100644 --- a/clients/splore/README.md +++ b/clients/splore/README.md @@ -1,20 +1,29 @@ # Splore -This is a generic noms data explorer. +Splore is a general-purpose debug UI for exploring noms data. -## Requirements +![splore and counter](screenshot.png) -* Node.js: https://nodejs.org/download/ -* You probably want to configure npm to [use a global module path that your user owns](https://docs.npmjs.com/getting-started/fixing-npm-permissions) +## Howto -## Build +``` +cd $GOPATH/src/github.com/attic-labs/noms/clients/counter +go build +./counter -ldb="/tmp/sploretest" -ds="counter" +./counter -ldb="/tmp/sploretest" -ds="counter" -* `NOMS_SERVER=http://localhost:8000 python build.py` +# Splore requires server to be running +cd ../server +go build +./server -ldb="/tmp/sploretest" & -## Run +cd ../splore +PYTHONPATH=$GOPATH/src/github.com/attic-labs/noms/tools NOMS_SERVER=http://localhost:8000 python build.py +./node_modules/.bin/http-server +``` + +Then, navigate to [http://localhost:8080]. -* `python -m SimpleHTTPServer 8082` (expects ../server to run on same host, port 8000) -* navigate to http://localhost:8082/ ## Develop diff --git a/clients/splore/package.json b/clients/splore/package.json index c1b428c8ef..f109216868 100644 --- a/clients/splore/package.json +++ b/clients/splore/package.json @@ -28,6 +28,7 @@ "eslint-plugin-react": "^3.8.0", "flow-bin": "^0.19.1", "grunt": "^0.4.5", + "http-server": "^0.8.5", "mocha": "^2.3.0", "uglify-js": "^2.6.1", "watchify": "^3.6" diff --git a/clients/splore/screenshot.png b/clients/splore/screenshot.png new file mode 100644 index 0000000000..59c1f20d1a Binary files /dev/null and b/clients/splore/screenshot.png differ