# Dataset pulling algorithm
The approach is to explore the chunk graph of both sink and source in order of decreasing ref-height. As the code walks, it uses the knowledge gained about which chunks are present in the sink to both prune the source-graph-walk and build up a set of `hints` that can be sent to a remote Database to aid in chunk validation.

## Basic algorithm

- let `sink` be the *sink* database
- let `source` be the *source* database
- let `snkQ` and `srcQ` be priority queues of `Ref` prioritized by highest `Ref.height`
- let `hints` be a map of `hash => hash`
- let `reachableChunks` be a set of hashes
- let `snkHdRef` be the ref (of `Commit`) of the head of the *sink* dataset
- let `srcHdRef` be the ref of the *source* `Commit`, which must descend from the `Commit` indicated by `snkHdRef`

- let `traverseSource(srcRef, srcQ, sink, source, reachableChunks)` be
  - pop `srcRef` from `srcQ`
    - if `!sink.has(srcRef)`
      - let `c` = `source.batchStore().Get(srcRef.targetHash)`
      - let `v` = `types.DecodeValue(c, source)`
      - insert all child refs, `cr`, from `v` into `srcQ` and into reachableRefs
      - `sink.batchStore().Put(c, srcRef.height, no hints)`
        - (hints will all be gathered and handed to sink.batchStore at the end)


- let `traverseSink(sinkRef, snkQ, sink, hints)` be
  - pop `snkRef` from `snkQ`
  - if `snkRef.height` > 1
    - let `v` = `sink.readValue(snkRef.targetHash)`
    - insert all child refs, `cr`, from `v` into `snkQ` and `hints[cr] = snkRef`


- let `traverseCommon(comRef, snkHdRef, snkQ, srcQ, sink, hints)` be
  - pop `comRef` from both `snkQ` and `srcQ`
  - if `comRef.height` > 1
    - if `comRef` is a `Ref` of `Commit`
      - let `v` = `sink.readValue(comRef.targetHash)`
      - if `comRef` == snkHdRef
        - *ignore all parent refs*
        - insert each other child ref `cr` from `v` into `snkQ` *only*, set `hints[cr] = comRef`
      - else
        - insert each child ref `cr` from `v` into both `snkQ` and `srcQ`, set `hints[cr] = comRef`


- let `pull(source, sink, srcHdRef, sinkHdRef)
  - insert `snkHdRef` into `snkQ` and `srcHdRef` into `srcQ`
  - create empty `hints` and `reachableChunks`
  - while `srcQ` is non-empty
    - let `srcHt` and `snkHt` be the respective heights of the *top* `Ref` in each of `srcQ` and `snkQ`
    - if `srcHt` > `snkHt`, for every `srcHdRef` in `srcQ` which is of greater height than `snkHt`
      - `traverseSource(srcHdRef, srcQ, sink, source)`
    - else if `snkHt` > `srcHt`, for every `snkHdRef` in `snkQ` which is of greater height than `srcHt`
      - `traverseSink(snkHdRef, snkQ, sink)`
    - else
      - for every `comRef` in which is common to `snkQ` and `srcQ` which is of height `srcHt` (and `snkHt`)
        - `traverseCommon(comRef, snkHdRef, snkQ, srcQ, sink, hints)`
      - for every `ref` in `srcQ` which is of height `srcHt`
        - `traverseSource(ref, srcQ, sink, source, reachableChunks)`
      - for every `ref` in `snkQ` which is of height `snkHt`
        - `traverseSink(ref, snkQ, sink, hints)`
  - for all `hash` in `reachableChunks`
    - sink.batchStore().addHint(hints[hash])


## Isomorphic, but less clear, algorithm

- let all identifiers be as above
- let `traverseSource`, `traverseSink`, and `traverseCommon` be as above

- let `higherThan(refA, refB)` be
  - if refA.height == refB.height
    - return refA.targetHash < refB.targetHash
  - return refA.height > refB.height

- let `pull(source, sink, srcHdRef, sinkHdRef)
  - insert `snkHdRef` into `snkQ` and `srcHdRef` into `srcQ`
  - create empty `hints` and `reachableChunks`
  - while `srcQ` is non-empty
    - if `sinkQ` is empty
      - pop `ref` from `srcQ`
      - `traverseSource(ref, srcQ, sink, source, reachableChunks))
    - else if `higherThan(head of srcQ, head of snkQ)`
      - pop `ref` from `srcQ`
      - `traverseSource(ref, srcQ, sink, source, reachableChunks))
    - else if `higherThan(head of snkQ, head of srcQ)`
      - pop `ref` from `snkQ`
      - `traverseSink(ref, snkQ, sink, hints)`
    - else, heads of both queues are the same
      - pop `comRef` from `snkQ` and `srcQ`
      - `traverseCommon(comRef, snkHdRef, snkQ, srcQ, sink, hints)`
  - for all `hash` in `reachableChunks`
    - sink.batchStore().addHint(hints[hash])