mirror/opencloud
1
0
Fork 0
mirror of https://github.com/opencloud-eu/opencloud.git synced 2026-04-22 11:09:02 -05:00
Code Issues Packages Projects Releases Wiki Activity

add a store to postprocessing

Browse Source 
Signed-off-by: jkoberg <jkoberg@owncloud.com>
This commit is contained in:
jkoberg
2023-05-11 13:05:49 +02:00
parent 7d8a245d37
commit a324ad48a0
6 changed files with 131 additions and 48 deletions
Download Patch File Download Diff File Expand all files Collapse all files
services/postprocessing/README.md
+19 -1
View File
@@ -1,6 +1,6 @@
# Postprocessing
The `postprocessing` service handles the coordination of asynchronous postprocessing steps.
The `postprocessing` service handles the coordination of asynchronous postprocessing steps.
## General Prerequisites
@@ -14,6 +14,24 @@ The `postprocessing` service will then coordinate configured postprocessing step
When all postprocessing steps have completed successfully, the file will be made accessible for users.
## Storing Postprocessing Data
The `postprocessing` service needs to store some metadata about uploads to be able to orchestrate post-processing. When running in single binary mode, the default in-memory implementation will be just fine. In distributed deployments it is recommended to use a persistent store, see below for more details.
The `postprocessing` service stores its metadata via the configured store in `POSTPROCESSING_STORE_TYPE`. Possible stores are:
- `memory`: Basic in-memory store and the default.
- `ocmem`: Advanced in-memory store allowing max size.
- `redis`: Stores data in a configured Redis cluster.
- `redis-sentinel`: Stores data in a configured Redis Sentinel cluster.
- `etcd`: Stores data in a configured etcd cluster.
- `nats-js`: Stores data using key-value-store feature of [nats jetstream](https://docs.nats.io/nats-concepts/jetstream/key-value-store)
- `noop`: Stores nothing. Useful for testing. Not recommended in production environments.
1. Note that in-memory stores are by nature not reboot-persistent.
2. Though usually not necessary, a database name and a database table can be configured for event stores if the event store supports this. Generally not applicable for stores of type `in-memory`. These settings are blank by default which means that the standard settings of the configured store apply.
3. The postprocessing service can be scaled if not using `in-memory` stores and the stores are configured identically over all instances.
4. When using `redis-sentinel`, the Redis master to use is configured via `POSTPROCESSING_STORE_NODES` in the form of `<sentinel-host>:<sentinel-port>/<redis-master>` like `10.10.0.200:26379/mymaster`.
## Additional Prerequisites for the Postprocessing Service
When postprocessing has been enabled, configuring any postprocessing step will require the requested services to be enabled and pre-configured. For example, to use the `virusscan` step, one needs to have an enabled and configured `antivirus` service.