Introduce clients/js/flickr
This version of the Flickr importer does its work in two stages:
1. Import the raw data
2. Parse out photo information
This is the type of thing I think will be common in archival use
cases for Noms: just get everything now. Worry about parsing it
later.
A common case for io.Writer errors is during noms-log or noms-show when
piped through head, i.e. "noms log ldb:/path/to:noms | head".
noms-log handles this itself by recovering from the panic, but rather
than porting this logic to noms-show, just make it so that you don't
need to recover in the first place.
This is similar to how errors from fmt.Println etc don't panic.
Non-write errors (like crashes) will continue to panic.
* Go: In httpBatchStore, order outgoing Chunks by ref-height
Rather than sending chunks to the server in the order that callers put
them into an httpBatchStore, instead order them by ref-height at the
time that Flush() is called. Making this change requires that
ref-height for a given chunk be passed in through the SchedulePut()
API, because Chunks (by design) don't carry much metadata about
themselves.
Toward #1510
The old timeout mechanism started counting when the client
began trying to connect to the server. This meant that sending
a large batch of chunks could cause the request to fail before
all the data even made it to the server. What we actually want
is to timeout when the connection has been idle for some amount
of time. This means that long server processing times can still
cause clients to time out, but that's probably as it should be.
Towards #1414
* JS: back BatchStore's put-cache with tingodb
tingodb is a pure-JS, file-backed mongodb-api-compatible database.
Before this patch, BatchStore was caching all pending chunks in memory,
which seems unlikely to scale. Putting them in a tingodb spills them to
disk as needed, though we can still iterate them in put-order when it's
time to send them.
Fixes#1349
makeUnionType now flattens and removes duplicate types. With this
change makeUnionType might no longer actually create a union
type in the case there is only one type in the union.
Towards #1491
This allows having a set/map of heterogenous elements/keys. To support
this we have to be able to have a total ordering for all noms values.
The ordering is false < true < -999 < 0 < 999 < "" < "a" < "z" < Hash
In other words, Bool < Number < String < * and for comparing non
primitives we compare the hash of the object.
Fixes#1104Fixes#1312
This is because:
* All type.Ref are now typed, so Typed was a tautology.
* The only way to construct a type.Ref is with a Value, so FromValue was
a tautology (with a small amount of work to remove callers of NewRef).
Rafael Weinstein