From 5126a8885633d17d1c19f2131db2e510d2e9ac76 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rn=20Friedrich=20Dreyer?= Date: Wed, 6 Jul 2022 13:32:39 +0000 Subject: [PATCH] add upload postprocessing sequence diagrams MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Jörn Friedrich Dreyer --- docs/architecture/upload-processing.md | 121 +++++++++++++++++++++++++ 1 file changed, 121 insertions(+) create mode 100644 docs/architecture/upload-processing.md diff --git a/docs/architecture/upload-processing.md b/docs/architecture/upload-processing.md new file mode 100644 index 0000000000..fbc3c5e9d5 --- /dev/null +++ b/docs/architecture/upload-processing.md @@ -0,0 +1,121 @@ +--- +title: "Upload processing" +date: 2022-07-06T12:47:00+01:00 +weight: 30 +geekdocRepo: https://github.com/owncloud/ocis +geekdocEditPath: edit/master/docs/architecture +geekdocFilePath: upload-processing.md +--- + +Uploads are handled by a dedicated service that uses TUS.io for rusumable uploads. When all bytes have been transferred the upload is finalized by making the file available in file listings and for download. + +The finalization may be asynchronous when mandatory workflow steps are involved. + +## Legacy PUT upload + +{{}} + +%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%% +%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976 +%% edit this diagram by pasting it into eg. https://mermaid.live +sequenceDiagram + participant Client + participant ocdav + participant storageprovider + participant dataprovider + + Client->>+ocdav: PUT /dav/spaces/{spaceid}/newfile.bin + ocdav->>+storageprovider: InitiateFileUpload + storageprovider-->>-ocdav: OK, Protocol simple, UploadEndpoint: /data, Token: {jwt} + Note right of ocdav: The {jwt} contains the internal actual target, eg.:
http://localhost:9158/data/simple/91cc9882-db71-4b37-b694-a522850fcee1 + ocdav->>+dataprovider: PUT /data
X-Reva-Transfer: {jwt} + dataprovider-->>-ocdav: 201 Created + ocdav-->>-Client: 201 Created + +{{}} + +## TUS upload + +{{}} + +%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%% +%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976 +%% edit this diagram by pasting it into eg. https://mermaid.live +sequenceDiagram + participant Client + participant ocdav + participant storageprovider + participant datagateway + participant dataprovider + + + Client->>+ocdav: POST /dav/spaces/{spaceid}
Upload-Metadata: {base64 encoded filename etc}
TUS-Resumable: 1.0.0 + ocdav->>+storageprovider: InitiateFileUpload + storageprovider-->>-ocdav: OK, Protocol tus, UploadEndpoint: /data, Token: {jwt} + Note right of ocdav: The {jwt} contains the internal actual target, eg.:
http://localhost:9158/data/tus/24d893f5-b942-4bc7-9fb0-28f49f980160 + ocdav-->>-Client: 201 Created
Location: /data/{jwt}
TUS-Resumable: 1.0.0 + + Client->>+datagateway: PATCH /data/{jwt}
TUS-Resumable: 1.0.0
Upload-Offset: 0 + Note over datagateway: unwrap the {jwt} target + datagateway->>+dataprovider: PATCH /data/tus/24d893f5-b942-4bc7-9fb0-28f49f980160
X-Reva-Transfer: {jwt} + Note over dataprovider: storage driver
handles request + dataprovider-->>-datagateway: 204 No Content
TUS-Resumable: 1.0.0
Upload-Offset: 363976 + datagateway-->>-Client: 204 No Content
TUS-Resumable: 1.0.0
Upload-Offset: 363976 + +{{}} + + +## TUS upload with async postprocessing + + + +{{}} + +%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%% +%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976 +%% edit this diagram by pasting it into eg. https://mermaid.live +sequenceDiagram + participant Client + participant ocdav + participant storageprovider + participant datagateway + participant dataprovider + participant nats + + + Client->>+ocdav: POST /dav/spaces/{spaceid}
Upload-Metadata: {base64 encoded filename etc}
TUS-Resumable: 1.0.0 + ocdav->>+storageprovider: InitiateFileUpload + storageprovider-->>-ocdav: OK, Protocol tus, UploadEndpoint: /data, Token: {jwt} + Note right of ocdav: The {jwt} contains the internal actual target, eg.:
http://localhost:9158/data/tus/24d893f5-b942-4bc7-9fb0-28f49f980160 + ocdav-->>-Client: 201 Created
Location: /data/{jwt}
TUS-Resumable: 1.0.0 + + Client->>+datagateway: PATCH /data/{jwt}
TUS-Resumable: 1.0.0
Upload-Offset: 0 + + Note over datagateway: unwrap the {jwt} target + datagateway->>+dataprovider: PATCH /data/tus/24d893f5-b942-4bc7-9fb0-28f49f980160
X-Reva-Transfer: {jwt} + Note over dataprovider: storage driver
handles request + dataprovider-)nats: emit all-bytes-received event + nats-)processing: all-bytes-received({uploadid}) event + Note over dataprovider: TODO: A lot of time may pass here, we could use
the `Prefer: respond-async` header to return early
with a 202 Accepted status and a Location header
to a websocket endpoint + alt success + processing-)nats: emit processing-finished({uploadid}) event + nats-)dataprovider: processing-finished({uploadid}) event + dataprovider-->>-datagateway: 204 No Content
TUS-Resumable: 1.0.0
Upload-Offset: 363976 + datagateway-->>-Client: 204 No Content
TUS-Resumable: 1.0.0
Upload-Offset: 363976 + else failure + activate dataprovider + activate datagateway + processing-)nats: emit processing-aborted({uploadid}) event + nats-)dataprovider: processing-aborted({uploadid}) event + Note over dataprovider: FIXME: What HTTP status code should we report?
422 Unprocessable Content is just a proposal, see
https://httpwg.org/specs/rfc9110.html#status.422 + dataprovider-->>-datagateway: 422 Unprocessable Content
TUS-Resumable: 1.0.0
Upload-Offset: 363976 + datagateway-->>-Client: 422 Unprocessable Content
TUS-Resumable: 1.0.0
Upload-Offset: 363976 + end + +{{}} + + +## Async TUS upload with postprocessing +This might be a TUS extension or a misunderstanding on our side of what tus can do for us. Clients should send a `Prefer: respond-async` header to allow the server to return early when postprocessing might take longer. The PATCH requests can then return status `202 Accepted` and a `Location` header to a websocket that clients can use to track the processing / upload progress. + +TODO there is a conflict with the TUS.io POST request with the creation extension, as that also returns a `Location` header which carries the upload URL. We would need another header to transport the websocket location. Maybe `Websocket-Location` or `Progress-Location`? \ No newline at end of file