Commit Graph

3385 Commits

Author SHA1 Message Date
zcstarr b35137b282 Add panic on invalid namespace for aws chunkstore (#3243) 2017-03-02 12:14:13 -08:00
cmasone-attic c89c09c5bb Close output channels in streaming collection builders (#3242)
Since these functions return a receive-only channel, the
caller _can't_ close the channel it pulls the constructed
collection off of. Since the builder function can easily
know when it's safe to close the channel, it seems fine
to just have NewStreamingWhatever() do the close.
2017-03-02 11:36:27 -08:00
Erik Arvidsson 25f3824f6e GraphQL: Add support for keys to Maps (#3237)
This adds a `keys` argument to Map types. This allows you to get
multiple entries in one line:

```graphql
{
  root {
    elements(keys: ["a", "c"]) {
      key
      value
    }
  }
}
```

Outputs:

```json
{
  "data": {
    "root": {
      "elements": [
        {"key": "a", "value": 1},
        {"key": "c", "value": 3}
      ]
    }
  }
}
```

If `keys` is present all other args are ignored.

Issue #3227
2017-03-01 18:12:56 -08:00
Dan Willhite 1d4a7a5921 Add function that makes json_to_noms create named structs (#3240) 2017-03-01 14:46:36 -08:00
Jesse Ditson c42056b15d Remove implicit directory creation when creating a local db (#3235)
Remove implicit directory creation when creating a local db

- remove Mkdir from NewLocalStore and NewLocalStoreFactory
- add specific error messages for directory does not exist and path is not a directory
- add tests for missing directory and path not directory

fixes: #3222
2017-03-01 13:29:09 -08:00
Erik Arvidsson 9d3a972973 GraphQL: Add support for key and through (#3233)
If key is provided as an argument to a Map/Set elements that defines
where the collection should start iterating from.

If through is provided the iteration will end after visiting through
(or if the key is larger than through)

If both key and at are present, at is ignored.

If key is present but count and through are missing then we use a count
of 1.

Closes #3227
2017-02-28 13:48:22 -08:00
cmasone-attic a89adf3276 Add additional logging in s3_table_reader.go (#3231)
This is to help create a support case for the S3
connection-reset-by-peer issue we're seeing on prod.
2017-02-27 15:12:06 -08:00
Erik Arvidsson 68de09f8db GraphQL map set at (#3229)
Add support for at arguments to Map and Set elements

Towards #3227
2017-02-24 17:34:43 -08:00
Erik Arvidsson cc77359e65 Add Set.IteratorAt and Set.IteratorFrom (#3228)
These two were missing for Set
2017-02-24 16:57:54 -08:00
Erik Arvidsson 046d842181 Update to Jest 19 (#3226) 2017-02-24 12:46:17 -08:00
Eric Halpern a355d619fc Implement TypesIntersect(a, b *types.Type) bool (#3223)
* Implement `TypesIntersect(a, b *types.Type) bool`

fixes: #3203

* Respond to review:
- Require at least one key type to match for map intersction
- Do parallel scan of field when comparing struct
- Add more tests for unions and nested collections

* Rename to ContainCommonSupertype
2017-02-24 11:23:47 -08:00
Aaron Boodman 1f25ceced8 Introduce Map.Iterator() and friends (#3225)
Introduce Map.Iterator() and friends.
2017-02-24 11:10:12 -08:00
Aaron Boodman dc41306cf2 Update README.md 2017-02-23 22:45:22 -08:00
Aaron Boodman ace7527396 Update README.md 2017-02-23 22:44:51 -08:00
Aaron Boodman e8b93dea34 Update nbs/README.md (#3224) 2017-02-23 22:37:37 -08:00
Aaron Boodman d1244c16d2 Update CONTRIBUTING.md 2017-02-23 14:15:37 -08:00
Dan Willhite 2dadf814d2 Add marshal.MustUnmarshal() function (#3220) 2017-02-23 14:11:00 -08:00
Eric Halpern 7ae6264782 Implement unionTypes(a, b *types.Type) *types.Type (#3204) (#3214)
* Implement `unionTypes(a, b *types.Type) *types.Type`

Implements type merging  building off of makeSimplifiedUnion.
The difference between siplified unions and merged types  are
in how structs are handled. In the former, we produce the intersection
of input structs to ensure the simplified type is a supertype of the input
types. In the later, we produce the union of the input structs which leads
to a type that is a subtype of all inputs.

Response to review and discussion
- Rename to MakeFullUnion to MakeMergedType
- Revert to old file naming
- Rename makeSimplifiedType to makeSupertype
- MakeUnion delegates to makeSupertype. Should prabably be be renamed, but
  this touches a lot of files, so keeping same for now.

fixes: #3204

* Rename to makeSimplifiedType and makeSimplifiedType2
2017-02-23 13:17:59 -08:00
Erik Arvidsson 310742211c Marshal to type (#3221)
This takes a Go value and reads the Go type from that and returns a Noms
type for that.
2017-02-23 13:09:37 -08:00
Ben Kalman c1afbb9013 Fix ngql test (#3217) 2017-02-22 15:34:57 -08:00
Ben Kalman 449b6d6b8f Use attic-labs/graphql (#3216) 2017-02-22 15:04:33 -08:00
cmasone-attic b1e918d1d4 Share s3, dynamodb clients (#3212)
These objects manage their own pools of HTTP connections and
other resources, so it's generally best to share them
process-wide if you can.

Fixes #3027
2017-02-22 13:41:23 -08:00
Eric Halpern 5b63cf6873 Ignore .nomsconfig 2017-02-22 12:40:56 -08:00
Aaron Boodman 4c55f2d64e noms log: add ability to log any path, not just dataset (#3202) 2017-02-21 23:12:11 -08:00
Ben Kalman 88c24324e4 Upgrade all extra JS and samples to Flow 0.39 (#3211) 2017-02-21 17:14:22 -08:00
Aaron Boodman 2a61225de3 Update README.md 2017-02-21 15:28:46 -08:00
Aaron Boodman 06bbabd29d Update README.md 2017-02-21 15:28:15 -08:00
Aaron Boodman 8a353cda95 Update README.md 2017-02-21 15:27:38 -08:00
Aaron Boodman 0f2291477c Update README.md 2017-02-21 15:27:07 -08:00
Aaron Boodman 3ef9dc2a75 Update README.md 2017-02-21 15:26:26 -08:00
Aaron Boodman a496c53897 Update README.md 2017-02-21 15:25:24 -08:00
Ben Kalman 9617c5c12a Make AsyncIterator a type, upgrade flow to 0.39 (#3209) 2017-02-21 14:03:26 -08:00
Aaron Boodman 4ea3981ba6 Update README.md 2017-02-21 13:13:44 -08:00
Aaron Boodman 784ff847ff Update README.md 2017-02-21 13:12:02 -08:00
Aaron Boodman 6a5dc86714 Update README.md 2017-02-21 13:08:47 -08:00
Aaron Boodman ba90efec1b Update README.md 2017-02-21 13:04:20 -08:00
Aaron Boodman c1487cca25 Refresh of the Noms homepage (#3208) 2017-02-21 13:01:06 -08:00
Aaron Boodman 4eaa0ba353 Remove Spec.DatasetName and Spec.Spec (#3102)
Remove Spec.DatasetName and Spec.Spec.

They were duplicating state that already existed elsewhere and easily
got out of sync.
2017-02-21 11:26:12 -08:00
Erik Arvidsson edb81e4d15 Fix NPE in GraphQL (#3201)
When we have a cyclic type we ended up getting the non complete type out
of the map.

Use graphql.FieldsThunk which is designed to allow recursive types.
2017-02-17 14:01:41 -08:00
Ben Kalman a020555eaf Propagate not found as GQL errors (#3200) 2017-02-17 09:32:31 -08:00
cmasone-attic 8b284a7c1e Revert "Log when s3TableReader hits the rate limiter" (#3199) 2017-02-16 15:48:45 -08:00
cmasone-attic 3391a5549c Roll aws-sdk-go (#3198)
Pull in https://github.com/aws/aws-sdk-go at 2d3b3bc3aae6a09a9b194aa6eb71326fcbe2e918
2017-02-16 15:00:27 -08:00
cmasone-attic 0cbd4b3809 Revert "Drop s3 read rate limit to 768" (#3197) 2017-02-16 11:51:16 -08:00
cmasone-attic 4069dc2109 Drop s3 read rate limit to 768 (#3196) 2017-02-16 11:44:55 -08:00
cmasone-attic 9798d6aab5 Log when s3TableReader hits the rate limiter (#3195)
We're getting some TCP hangups when trying to talk to S3.
We have an S3 reading rate limiter that's supposed to
prevent issues like this, so the question is whether that's
set too high. Rather than just turning the knob and seeing
if things are OK, this patch adds some logging to verify
that we're actually hitting the rate limiter.
2017-02-16 10:32:13 -08:00
Erik Arvidsson cb520e76fa Fix GraphQL to use unique type names (#3194)
The names of GraphQL types needs to be globally unique. We therefore
name struct types as NAME_HASH (where hash is the first 6 chars of
the noms hash).

Also, make sure that we always map the noms type to the same GraphQL
type, even if the boxedIfScalar is true.

Fixes #3161
2017-02-15 17:24:38 -08:00
Rafael Weinstein 83b657fe62 Remove LevelDBStore (#3193)
Remove LevelDBStore
2017-02-14 21:52:25 -08:00
Rafael Weinstein 9527907674 Nbs local store factory (#3191)
Add NBS LocalStoreFactory
2017-02-14 20:52:30 -08:00
cmasone-attic 5025a45b0b Remove usages of LevelDBStore (#3190)
NBS is stable enough that we've made it the default store for command
line tools, and the go-to store for tests that require temporary, but
persistent, storage.

We intend to remove support for LevelDB-backed chunk storage
completely ASAP. This patch removes all usage of LevelDBStore from
noms.git, but doesn't remove LevelDBStore _just_ yet as there are
still some dependencies on it elsewhere.

Toward #3127
2017-02-14 19:49:23 -08:00
cmasone-attic 53d343a546 NBS: tableSet.Flatten() must exclude empty tables (#3189)
I missed this in the compaction patch :-/ I caught it in another
test when the code panic'd while trying to write a manifest with
an empty table in it. So at least it got caught there?
2017-02-14 10:46:43 -08:00