From 23ee8615c6d63c20de2f3dfe6782920f996eaddc Mon Sep 17 00:00:00 2001 From: mmattel Date: Thu, 16 Mar 2023 15:03:29 +0100 Subject: [PATCH] [docs-only] Adds missing services and fixes content for owncloud.dev --- docs/services/eventhistory/_index.md | 43 ++++++ docs/services/eventhistory/configuration.md | 15 +++ docs/services/policies/_index.md | 97 ++++++++++---- docs/services/port-ranges.md | 5 +- docs/services/postprocessing/_index.md | 57 ++++++++ docs/services/postprocessing/configuration.md | 15 +++ docs/services/userlog/_index.md | 63 +++++++++ docs/services/userlog/configuration.md | 15 +++ docs/services/webfinger/_index.md | 123 +++++++++++++++++- services/policies/README.md | 96 ++++++++++---- services/postprocessing/README.md | 4 +- services/userlog/README.md | 10 +- services/webfinger/README.md | 7 +- 13 files changed, 489 insertions(+), 61 deletions(-) create mode 100644 docs/services/eventhistory/_index.md create mode 100644 docs/services/eventhistory/configuration.md create mode 100644 docs/services/postprocessing/_index.md create mode 100644 docs/services/postprocessing/configuration.md create mode 100644 docs/services/userlog/_index.md create mode 100644 docs/services/userlog/configuration.md diff --git a/docs/services/eventhistory/_index.md b/docs/services/eventhistory/_index.md new file mode 100644 index 000000000..d0c7b34df --- /dev/null +++ b/docs/services/eventhistory/_index.md @@ -0,0 +1,43 @@ +--- +title: Eventhistory +weight: 20 +geekdocRepo: https://github.com/owncloud/ocis +geekdocEditPath: edit/master/docs/services/eventhistory +geekdocFilePath: _index.md +geekdocCollapseSection: true +--- + +## Abstract + +The `eventhistory` consumes all events from the configured event system like NATS, stores them and allows other services to retrieve them via an eventid. + +## Table of Contents + +{{< toc-tree >}} + +## Prerequisites + +Running the eventhistory service without an event sytem like NATS is not possible. + +## Consuming + +The `eventhistory` services consumes all events from the configured event sytem. + +## Storing + +The `eventhistory` service stores each consumed event via the configured store in `EVENTHISTORY_STORE_TYPE`. Possible stores are: + - `mem`: Basic in-memory store and the default. + - `ocmem`: Advanced in-memory store allowing max size. + - `redis`: Stores data in a configured redis cluster. + - `etcd`: Stores data in a configured etcd cluster. + - `nats-js`: Stores data using key-value-store feature of [nats jetstream](https://docs.nats.io/nats-concepts/jetstream/key-value-store) + - `noop`: Stores nothing. Useful for testing. Not recommended in productive enviroments. + +1. Note that in-memory stores are by nature not reboot persistent. +2. Though usually not necessary, a database name and a database table can be configured for event stores if the event store supports this. Generally not applicapable for stores of type `in-memory`. These settings are blank by default which means that the standard settings of the configured store applies. +3. Events stay in the store for 2 weeks by default. Use `EVENTHISTORY_RECORD_EXPIRY` to adjust this value. +4. The eventhistory service can be scaled if not using `in-memory` stores and the stores are configured identically over all instances. + +## Retrieving + +Other services can call the `eventhistory` service via a grpc call to retrieve events. The request must contain the eventid that should be retrieved. diff --git a/docs/services/eventhistory/configuration.md b/docs/services/eventhistory/configuration.md new file mode 100644 index 000000000..752ebd654 --- /dev/null +++ b/docs/services/eventhistory/configuration.md @@ -0,0 +1,15 @@ +--- +title: Service Configuration +date: 2018-05-02T00:00:00+00:00 +weight: 20 +geekdocRepo: https://github.com/owncloud/ocis +geekdocEditPath: edit/master/docs/services/eventhistory +geekdocFilePath: configuration.md +geekdocCollapseSection: true +--- + +## Example YAML Config + +{{< include file="services/_includes/eventhistory-config-example.yaml" language="yaml" >}} + +{{< include file="services/_includes/eventhistory_configvars.md" >}} diff --git a/docs/services/policies/_index.md b/docs/services/policies/_index.md index cfb962e41..08ec7ebee 100644 --- a/docs/services/policies/_index.md +++ b/docs/services/policies/_index.md @@ -11,21 +11,21 @@ geekdocCollapseSection: true The policies service provides a new grpc api which can be used to return whether a requested operation is allowed or not. To do so, Open Policy Agent (OPA) is used to determine the set of rules of what is permitted and what is not. +Policies are written in the [rego query language](https://www.openpolicyagent.org/docs/latest/policy-language/). The location of the rego files can be configured via yaml, a configuration via environment variables is not possible. + ## Table of Contents {{< toc-tree >}} -## Rego +## General Information -Policies are written in the [rego query language](https://www.openpolicyagent.org/docs/latest/policy-language/). The location of the rego files can be configured via yaml, a configuration via environment variables is not possible. +The policies service consists of the following modules: -The Policies Service consists of the following modules: +* Proxy authorization (middleware) +* Event authorization (async post-processing) +* gRPC API (can be used by other services) -* Proxy Authorization (middleware) -* Event Authorization (async post-processing) -* GRPC API (can be used from other services) - -To configure the Policies Service, three environment variables need to be defined: +To configure the policies service, three environment variables need to be defined: * `POLICIES_ENGINE_TIMEOUT` * `POLICIES_POSTPROCESSING_QUERY` @@ -37,48 +37,75 @@ To activate a the policies service for a module, it must be started with a yaml When using async post-processing which is done via the postprocessing service, the value `policies` must be added to the `POSTPROCESSING_STEPS` configuration in postprocessing service in the order where the evaluation should take place. +variable defined in the Rego rule set the corresponding step uses for the evaluation. If the variable is mistyped or not found, the evaluation defaults to deny. Individual query definitions can be defined for each module. + +To activate the policies service for a module, it must be started with a yaml configuration that points to at least one Rego file that contains the complete rule variable to be queried. Note that if the service is scaled horizontally, each instance should have access to the same Rego files to avoid unpredictable results. If a file path has been configured but the file it is not present or accessible, the evaluation defaults to deny. + +When using async post-processing via the postprocessing service, the value `policies` must be added to the `POSTPROCESSING_STEPS` configuration in the order in which the evaluation should take place. Example: First check if a file contains questionable content via policies. If it looks okay, continue to check for viruses. + +For configuration examples, the [Example Policies](#example-policies) from below are used. + ## Modules -### GRPC Service +### gRPC API -This service can be used from any other internal service. It can also be used for example by third parties to find out if an action is allowed or not. This layer is already used by the proxy middleware. - -### Event Service - -This layer is event-based and part of the postprocessing service. Since processing at this point is asynchronous, the operations can also take longer and be more expensive, like evaluating the bytes of a file. +The gRPC API can be used by any other internal service. It can also be used for example by third parties to find out if an action is allowed or not. This layer is already used by the proxy middleware. There is no configuration necessary, because the query setting (complete rule variable) must be part of the request. ### Proxy Middleware -The [ocis proxy](../proxy) already includes such a middleware which uses the [GRPC service](#grpc-service) to evaluate the policies by using a configurable query. Since the Proxy is in heavy use and every request is processed here, only simple and quick decisions should be evaluated. More complex queries such as file evaluation are strongly discouraged. +The proxy service already includes a middleware which uses the internal [gRPC API](#grpc-api) to evaluate the policies. Since the proxy is in heavy use and every HTTP request is processed here, only simple and quick decisions should be evaluated. More complex queries such as file content evaluation are _strongly_ discouraged. -## Example Policies +### Event Service (Postprocessing) -The policies service contains a set of pre-configured example policies. Those policies can be found in the [examples directory](https://github.com/owncloud/ocis/tree/master/deployments/examples/service_policies/policies). The contained policies disallows ocis to create certain filetypes, both for the proxy middleware and the events service. +This layer is event-based and part of the postprocessing service. Since processing at this point is asynchronous, the operations can also take longer and be more expensive, like evaluating the contents of a file. -To use the example policies, it's required to configure ocis to use these files which can be done by adding: +## Defining Policies to Evaluate + +Each module can have as many policy files as needed for evaluation. Files can also include other files if necessary. To use policies, they have to be saved to a location that is accessible to the policies service. As a good starting point, take the config directory and use a subdirectory collecting all the `.rego` files, though any other directory can be defined. The config directory is already accessible by all services and usually is included in a xref:maintenance/b-r/backup.adoc[backup] plan. + +If this is done, it's required to configure the policies service to use these files: + +NOTE: It is important that *all* necessary files are added to the list of files the policies service uses. ```yaml policies: engine: policies: - - YOUR_PATH/examples/policies/proxy.rego - - YOUR_PATH/examples/policies/postprocessing.rego - - YOUR_PATH/examples/policies/utils.rego + - your_path_to_policies/proxy.rego + - your_path_to_policies/postprocessing.rego + - your_path_to_policies/util.rego ``` -Once the policies are configured correctly, the _QUERY configuration needs to be defined for the proxy middleware and for the events service. +Once the references to policy files are configured correctly, the _QUERY configuration needs to be defined for the proxy middleware and for the events service. + +## Setting the Query Configuration + +To define a value for the query evaluation, the following scheme is necessary: + +`data..` + +* The keyword `data` is mandatory and must be present. +* The `package-name` is defined in one .rego file like `package postprocessing`. It is not related to the filename. For more details, see the [packages](https://www.openpolicyagent.org/docs/latest/policy-language/#packages) documentation. +* The `complete-rule-variable-name` is the variable providing the result of the evaluation. +* Exact one of the defined files, which is responsible for returning the evaluation result, must contain the combination of `` and ``. ### Proxy +Note that this setting has to be part of the proxy configuration. + ```yaml proxy: policies_middleware: query: data.proxy.granted ``` -The same can be achieved by setting the `PROXY_POLICIES_QUERY=data.proxy.granted` environment variable. +The same can be achieved by setting the following evironment variable: -### ASYNC Postprocessing +```yaml +PROXY_POLICIES_QUERY=data.proxy.granted +``` + +### Postprocessing ```yaml policies: @@ -86,4 +113,24 @@ policies: query: data.postprocessing.granted ``` -The same can be achieved by setting the `POLICIES_POSTPROCESSING_QUERY=data.postprocessing.granted` environment variable. As soon as that query is configured correctly, postprocessing must be informed to use the policies step by setting the environment variable `POSTPROCESSING_STEPS=policies`. Note that additional steps can be configured and their appearance defines the order of processing. For details see the postprocessing service documentation. +The same can be achieved by setting the following evironment variable: + +```yaml +POLICIES_POSTPROCESSING_QUERY=data.postprocessing.granted +``` + +As soon as that query is configured, the postprocessing service must be informed to use the policies step by setting the environment variable: + +```yaml +POSTPROCESSING_STEPS=policies +``` + +Note that additional steps can be configured and their position in the list defines the order of processing. For details see the postprocessing service documentation. + +## Rego Key Match + +To identify available keys for OPA, you need to look at [engine.go](https://github.com/owncloud/ocis/blob/master/services/policies/pkg/engine/engine.go) and the [policies.swagger.json](https://github.com/owncloud/ocis/blob/master/protogen/gen/ocis/services/policies/v0/policies.swagger.json) file. Note that which keys are avaialble depends from which module it is used. + +## Example Policies + +The policies service contains a set of preconfigured example policies. See the [deployment examples](https://github.com/owncloud/ocis/tree/master/deployments/examples) directory for details. The contained policies disallow Infinite Scale to create certain file types, both via the proxy middleware and the events service via postprocessing. diff --git a/docs/services/port-ranges.md b/docs/services/port-ranges.md index 0a75793c4..c260930d1 100644 --- a/docs/services/port-ranges.md +++ b/docs/services/port-ranges.md @@ -4,12 +4,11 @@ date: 2018-05-02T00:00:00+00:00 weight: 0 geekdocRepo: https://github.com/owncloud/ocis geekdocEditPath: edit/master/docs/extensions -geekdocFilePath: port-ranges.go +geekdocFilePath: port-ranges.md geekdocCollapseSection: true --- -oCIS services often need a port to expose their services to other services or the outside world. -As users may have many different extensions running on the same machine, we should track port usage in the oCIS ecosystem. In the best case we ensure that each extension uses a non colliding port range, to make life of users easier. +oCIS services often need a port to expose their services to other services or the outside world. As users may have many different extensions running on the same machine, we should track port usage in the oCIS ecosystem. In the best case we ensure that each extension uses a non colliding port range, to make life of users easier. This page tracks the knowingly used port ranges. diff --git a/docs/services/postprocessing/_index.md b/docs/services/postprocessing/_index.md new file mode 100644 index 000000000..9daa97e96 --- /dev/null +++ b/docs/services/postprocessing/_index.md @@ -0,0 +1,57 @@ +--- +title: Postprocessing +weight: 20 +geekdocRepo: https://github.com/owncloud/ocis +geekdocEditPath: edit/master/docs/services/postprocessing +geekdocFilePath: _index.md +geekdocCollapseSection: true +--- + +## Abstract + +The `postprocessing` service handles the coordination of asynchronous postprocessing steps. + +## Table of Contents + +{{< toc-tree >}} + +## General Prerequisites + +To use the postprocessing service, an event system needs to be configured for all services. By default, `ocis` ships with a preconfigured `nats` service. + +## Postprocessing Functionality + +The storageprovider service (`storage-users`) can be configured to initiate asynchronous postprocessing by setting the `STORAGE_USERS_OCIS_ASYNC_UPLOADS` environment variable to `true`. If this is the case, postprocessing will get initiated *after* uploading a file and all bytes have been received. + +The `postprocessing` service will then coordinate configured postprocessing steps like scanning the file for viruses. During postprocessing, the file will be in a `processing state` where only a limited set of actions are available. Note that this processing state excludes file accessability by users. + +When all postprocessing steps have completed successfully, the file will be made accessible for users. + +## Additional Prerequisites for the Postprocessing Service + +When postprocessing has been enabled, configuring any postprocessing step will require the requested services to be enabled and pre-configured. For example, to use the `virusscan` step, one needs to have an enabled and configured `antivirus` service. + +## Postprocessing Steps + +The postporcessing service is individually configurable. This is achieved by allowing a list of postprocessing steps that are processed in order of their appearance in the `POSTPROCESSING_STEPS` envvar. This envvar expects a comma separated list of steps that will be executed. Currently known steps to the system are `virusscan` and `delay`. Custom steps can be added but need an existing target for processing. + +### Virus Scanning + +To enable virus scanning as a postprocessing step after uploading a file, the environment variable `POSTPROCESSING_STEPS` needs to contain the word `virusscan` at one location in the list of steps. As a result, each uploaded file gets virus scanned as part of the postprocessing steps. Note that the `antivirus` service is required to be enabled and configured for this to work. + +### Delay + +Though this is for development purposes only and NOT RECOMMENDED on production systems, setting the environment variable `POSTPROCESSING_DELAY` to a duration not equal to zero will add a delay step with the configured amount of time. ocis will continue postprocessing the file after the configured delay. Use the enviroment variable `POSTPROCESSING_STEPS` and the keyword `delay` if you have multiple postprocessing steps and want to define their order. If `POSTPROCESSING_DELAY` is set but the keyword `delay` is not contained in `POSTPROCESSING_STEPS`, it will be processed as last postprocessing step without being listed there. In this case, a log entry will be written on service startup to notify the admin about that situation. That log entry can be avoided by adding the keyword `delay` to `POSTPROCESSING_STEPS`. + +### Custom Postprocessing Steps +By using the envvar `POSTPROCESSING_STEPS`, custom postprocessing steps can be added. Any word can be used as step name but be careful not to conflict with exising keywords like `virusscan` and `delay`. In addition, if a keyword is misspelled or the corresponding service does either not exist or does not follow the necessary event communication, the postprocessing service will wait forever getting the required response to proceed and does not continue any other processing. + +#### Prerequisites +For using custom postprocessing steps you need a custom service listening to the configured event system (see `General Prerequisites`) + +#### Workflow +When setting a custom postprocessing step (eg. `"customstep"`) the postprocessing service will eventually sent an event during postprocessing. The event will be of type `StartPostprocessingStep` with its field `StepToStart` set to `"customstep"`. When the custom service receives this event it can savely execute its actions, postprocessing service will wait until it has finished its work. The event contains further information (filename, executing user, size, ...) and also required tokens and urls to download the file in case byte inspection is necessary. + +Once the custom service has finished its work, it should sent an event of type `PostprocessingFinished` via the configured events system. This event needs to contain a `FinishedStep` field set to `"customstep"`. It also must contain the outcome of the step, which can be one of "delete" (abort postprocessing, delete the file), "abort" (abort postprocessing, keep the file) and "continue" (continue postprocessing, this is the success case). + +See the [cs3 org](https://github.com/cs3org/reva/blob/edge/pkg/events/postprocessing.go) for up-to-date information of reserved step names and event definitons. diff --git a/docs/services/postprocessing/configuration.md b/docs/services/postprocessing/configuration.md new file mode 100644 index 000000000..2e827aaa7 --- /dev/null +++ b/docs/services/postprocessing/configuration.md @@ -0,0 +1,15 @@ +--- +title: Service Configuration +date: 2018-05-02T00:00:00+00:00 +weight: 20 +geekdocRepo: https://github.com/owncloud/ocis +geekdocEditPath: edit/master/docs/services/postprocessing +geekdocFilePath: configuration.md +geekdocCollapseSection: true +--- + +## Example YAML Config + +{{< include file="services/_includes/postprocessing-config-example.yaml" language="yaml" >}} + +{{< include file="services/_includes/postprocessing_configvars.md" >}} diff --git a/docs/services/userlog/_index.md b/docs/services/userlog/_index.md new file mode 100644 index 000000000..21a942d03 --- /dev/null +++ b/docs/services/userlog/_index.md @@ -0,0 +1,63 @@ +--- +title: Userlog +weight: 20 +geekdocRepo: https://github.com/owncloud/ocis +geekdocEditPath: edit/master/docs/services/userlog +geekdocFilePath: _index.md +geekdocCollapseSection: true +--- + +## Abstract + +The `userlog` service is a mediator between the `eventhistory` service and clients who want to be informed about user related events. It provides an API to retrieve those. + +## Table of Contents + +{{< toc-tree >}} + +## Prerequisites + +Running the `userlog` service without running the `eventhistory` service is not possible. + +## Storing + +The `userlog` service persists information via the configured store in `USERLOG_STORE_TYPE`. Possible stores are: + - `mem`: Basic in-memory store and the default. + - `ocmem`: Advanced in-memory store allowing max size. + - `redis`: Stores data in a configured redis cluster. + - `etcd`: Stores data in a configured etcd cluster. + - `nats-js`: Stores data using key-value-store feature of [nats jetstream](https://docs.nats.io/nats-concepts/jetstream/key-value-store) + - `noop`: Stores nothing. Useful for testing. Not recommended in productive enviroments. + +1. Note that in-memory stores are by nature not reboot persistent. +2. Though usually not necessary, a database name and a database table can be configured for event stores if the event store supports this. Generally not applicapable for stores of type `in-memory`. These settings are blank by default which means that the standard settings of the configured store applies. +3. The userlog service can be scaled if not using `in-memory` stores and the stores are configured identically over all instances. + +## Configuring + +For the time being, the configuration which user related events are of interest is hardcoded and cannot be changed. + +## Retrieving + +The `userlog` service provides an API to retrieve configured events. For now, this API is mostly following the [oc10 notification GET API](https://doc.owncloud.com/server/next/developer_manual/core/apis/ocs-notification-endpoint-v1.html#get-user-notifications). + +## Deleting + +To delete events for an user, use a `DELETE` request to `ocs/v2.php/apps/notifications/api/v1/notifications` containing the IDs to delete. + +## Translations + +The `userlog` service has embedded translations sourced via transifex to provide a basic set of translated languages. These embedded translations are available for all deployment scenarios. In addition, the service supports custom translations, though it is currently not possible to just add custom translations to embedded ones. If custom translations are configured, the embedded ones are not used. To configure custom translations, the `USERLOG_TRANSLATION_PATH` environment variable needs to point to a base folder that will contain the translation files. This path must be available from all instances of the userlog service, a shared storage is recommended. Translation files must be of type [.po](https://www.gnu.org/software/gettext/manual/html_node/PO-Files.html#PO-Files) or [.mo](https://www.gnu.org/software/gettext/manual/html_node/Binaries.html). For each language, the filename needs to be `userlog.po` (or `userlog.mo`) and stored in a folder structure defining the language code. In general the path/name pattern for a translation file needs to be: + +```text +{USERLOG_TRANSLATION_PATH}/{language-code}/LC_MESSAGES/userlog.po +``` + +The language code pattern is composed of `language[_territory]` where `language` is the base language and `_territory` is optional and defines a country. + +For example, for the language `de_DE`, one needs to place the corresponding translation files to `{USERLOG_TRANSLATION_PATH}/de_DE/LC_MESSAGES/userlog.po`. + +### Translation Rules + +* If a requested language code is not available, the service tries to fall back to the base language if available. For example, if `de_DE` is not available, the service tries to fall back to translations in the `de` folder. +* If the base language `de` is also not available, the service falls back to the system's default English (`en`), which is the source of the texts provided by the code. diff --git a/docs/services/userlog/configuration.md b/docs/services/userlog/configuration.md new file mode 100644 index 000000000..555962003 --- /dev/null +++ b/docs/services/userlog/configuration.md @@ -0,0 +1,15 @@ +--- +title: Service Configuration +date: 2018-05-02T00:00:00+00:00 +weight: 20 +geekdocRepo: https://github.com/owncloud/ocis +geekdocEditPath: edit/master/docs/services/userlog +geekdocFilePath: configuration.md +geekdocCollapseSection: true +--- + +## Example YAML Config + +{{< include file="services/_includes/userlog-config-example.yaml" language="yaml" >}} + +{{< include file="services/_includes/userlog_configvars.md" >}} diff --git a/docs/services/webfinger/_index.md b/docs/services/webfinger/_index.md index 178f0c84d..eac7d4b22 100644 --- a/docs/services/webfinger/_index.md +++ b/docs/services/webfinger/_index.md @@ -10,9 +10,130 @@ geekdocCollapseSection: true ## Abstract -This service provides endpoints a the /.well-known/webfinger implementation. +The webfinger service provides an RFC7033 WebFinger lookup of ownCloud instances relevant for a given user account via endpoints a the /.well-known/webfinger implementation. + +It is based on https://github.com/owncloud/lookup-webfinger-sciebo but also returns localized `titles` in addition to the `href` property. ## Table of Contents {{< toc-tree >}} +## OpenID Connect Discovery + +Clients can make an unauthenticated `GET https://drive.ocis.test/.well-known/webfinger?resource=https%3A%2F%2Fcloud.ocis.test` request to discover the OpenID Connect Issuer in the `http://openid.net/specs/connect/1.0/issuer` relation: + +```json +{ + "subject": "acct:einstein@drive.ocis.test", + "links": [ + { + "rel": "http://openid.net/specs/connect/1.0/issuer", + "href": "https://sso.example.org/cas/oidc/" + } + ] +} +``` + +Here, the `resource` takes the instance domain URI, but an `acct:` URI works as well. + +## Authenticated Instance Discovery + +When using OpenID connect to authenticate requests, clients can look up the owncloud instances a user has access to. + +* Authentication is necessary to prevent leaking information about existing users. +* Basic auth is not supported. + +The default configuration will simply return the `OCIS_URL` and direct clients to that domain: + +```json +{ + "subject": "acct:einstein@drive.ocis.test", + "links": [ + { + "rel": "http://openid.net/specs/connect/1.0/issuer", + "href": "https://sso.example.org/cas/oidc/" + }, + { + "rel": "http://webfinger.owncloud/rel/server-instance", + "href": "https://abc.drive.example.org", + "titles": { + "en": "oCIS Instance" + } + } + ] +} +``` + +## Configure Different Instances Based on OpenidConnect UserInfo Claims + +A more complex example for configuring different instances could look like this: + +```yaml +webfinger: + instances: + - claim: email + regex: einstein@example\.org + href: "https://{{.preferred_username}}.cloud.ocis.test" + title: + "en": "oCIS Instance for Einstein" + "de": "oCIS Instanz für Einstein" + break: true + - claim: "email" + regex: marie@example\.org + href: "https://{{.preferred_username}}.cloud.ocis.test" + title: + "en": "oCIS Instance for Marie" + "de": "oCIS Instanz für Marie" + break: false + - claim: "email" + regex: .+@example\.org + href: "https://example-org.cloud.ocis.test" + title: + "en": "oCIS Instance for example.org" + "de": "oCIS Instanz für example.org" + break: true + - claim: "email" + regex: .+@example\.com + href: "https://example-com.cloud.ocis.test" + title: + "en": "oCIS Instance for example.com" + "de": "oCIS Instanz für example.com" + break: true + - claim: "email" + regex: .+@.+\..+ + href: "https://cloud.ocis.test" + title: + "en": "oCIS Instance" + "de": "oCIS Instanz" + break: true +``` + +Now, an authenticated webfinger request for `acct:me@example.org` (when logged in as marie) would return two instances, based on her `email` claim, the regex matches and break flags: + +```json +{ + "subject": "acct:marie@example.org", + "links": [ + { + "rel": "http://openid.net/specs/connect/1.0/issuer", + "href": "https://sso.example.org/cas/oidc/" + }, + { + "rel": "http://webfinger.owncloud/rel/server-instance", + "href": "https://marie.cloud.ocis.test", + "titles": { + "en": "oCIS Instance for Marie", + "de": "oCIS Instanz für Marie" + } + }, + { + "rel": "http://webfinger.owncloud/rel/server-instance", + "href": "https://xyz.drive.example.org", + "titles": { + "en": "oCIS Instance for example.org", + "de": "oCIS Instanz für example.org" + } + } + ] +} +``` diff --git a/services/policies/README.md b/services/policies/README.md index 82a67e18b..dd93f6e5d 100644 --- a/services/policies/README.md +++ b/services/policies/README.md @@ -1,16 +1,18 @@ # Policies Service -The policies service provides a new grpc api which can be used to return whether a requested operation is allowed or not. To do so, Open Policy Agent (OPA) is used to determine the set of rules of what is permitted and what is not. +The policies service provides a new gRPC API which can be used to check whether a requested operation is allowed or not. To do so, Open Policy Agent (OPA) is used to define the set of rules of what is permitted and what is not. Policies are written in the [rego query language](https://www.openpolicyagent.org/docs/latest/policy-language/). The location of the rego files can be configured via yaml, a configuration via environment variables is not possible. -The Policies Service consists of the following modules: +## General Information -* Proxy Authorization (middleware) -* Event Authorization (async post-processing) -* GRPC API (can be used from other services) +The policies service consists of the following modules: -To configure the Policies Service, three environment variables need to be defined: +* Proxy authorization (middleware) +* Event authorization (async post-processing) +* gRPC API (can be used by other services) + +To configure the policies service, three environment variables need to be defined: * `POLICIES_ENGINE_TIMEOUT` * `POLICIES_POSTPROCESSING_QUERY` @@ -22,47 +24,75 @@ To activate a the policies service for a module, it must be started with a yaml When using async post-processing which is done via the postprocessing service, the value `policies` must be added to the `POSTPROCESSING_STEPS` configuration in postprocessing service in the order where the evaluation should take place. +variable defined in the Rego rule set the corresponding step uses for the evaluation. If the variable is mistyped or not found, the evaluation defaults to deny. Individual query definitions can be defined for each module. + +To activate the policies service for a module, it must be started with a yaml configuration that points to at least one Rego file that contains the complete rule variable to be queried. Note that if the service is scaled horizontally, each instance should have access to the same Rego files to avoid unpredictable results. If a file path has been configured but the file it is not present or accessible, the evaluation defaults to deny. + +When using async post-processing via the postprocessing service, the value `policies` must be added to the `POSTPROCESSING_STEPS` configuration in the order in which the evaluation should take place. Example: First check if a file contains questionable content via policies. If it looks okay, continue to check for viruses. + +For configuration examples, the [Example Policies](#example-policies) from below are used. + ## Modules -### GRPC Service +### gRPC API -This service can be used from any other internal service. It can also be used for example by third parties to find out if an action is allowed or not. This layer is already used by the proxy middleware. - -### Event Service - -This layer is event-based and part of the postprocessing service. Since processing at this point is asynchronous, the operations can also take longer and be more expensive, like evaluating the bytes of a file. +The gRPC API can be used by any other internal service. It can also be used for example by third parties to find out if an action is allowed or not. This layer is already used by the proxy middleware. There is no configuration necessary, because the query setting (complete rule variable) must be part of the request. ### Proxy Middleware -The [ocis proxy](../proxy) already includes such a middleware which uses the [GRPC service](#grpc-service) to evaluate the policies by using a configurable query. Since the Proxy is in heavy use and every request is processed here, only simple and quick decisions should be evaluated. More complex queries such as file evaluation are strongly discouraged. +The proxy service already includes a middleware which uses the internal [gRPC API](#grpc-api) to evaluate the policies. Since the proxy is in heavy use and every HTTP request is processed here, only simple and quick decisions should be evaluated. More complex queries such as file content evaluation are _strongly_ discouraged. -## Example Policies +### Event Service (Postprocessing) -The policies service contains a set of pre-configured example policies. Those policies can be found in the [examples directory](../../deployments/examples/service_policies/policies). The contained policies disallows ocis to create certain filetypes, both for the proxy middleware and the events service. +This layer is event-based and part of the postprocessing service. Since processing at this point is asynchronous, the operations can also take longer and be more expensive, like evaluating the contents of a file. -To use the example policies, it's required to configure ocis to use these files which can be done by adding: +## Defining Policies to Evaluate + +Each module can have as many policy files as needed for evaluation. Files can also include other files if necessary. To use policies, they have to be saved to a location that is accessible to the policies service. As a good starting point, take the config directory and use a subdirectory collecting all the `.rego` files, though any other directory can be defined. The config directory is already accessible by all services and usually is included in a xref:maintenance/b-r/backup.adoc[backup] plan. + +If this is done, it's required to configure the policies service to use these files: + +NOTE: It is important that *all* necessary files are added to the list of files the policies service uses. ```yaml policies: engine: policies: - - YOUR_PATH/examples/policies/proxy.rego - - YOUR_PATH/examples/policies/postprocessing.rego - - YOUR_PATH/examples/policies/utils.rego + - your_path_to_policies/proxy.rego + - your_path_to_policies/postprocessing.rego + - your_path_to_policies/util.rego ``` -Once the policies are configured correctly, the _QUERY configuration needs to be defined for the proxy middleware and for the events service. + +Once the references to policy files are configured correctly, the _QUERY configuration needs to be defined for the proxy middleware and for the events service. + +## Setting the Query Configuration + +To define a value for the query evaluation, the following scheme is necessary: + +`data..` + +* The keyword `data` is mandatory and must be present. +* The `package-name` is defined in one .rego file like `package postprocessing`. It is not related to the filename. For more details, see the [packages](https://www.openpolicyagent.org/docs/latest/policy-language/#packages) documentation. +* The `complete-rule-variable-name` is the variable providing the result of the evaluation. +* Exact one of the defined files, which is responsible for returning the evaluation result, must contain the combination of `` and ``. ### Proxy +Note that this setting has to be part of the proxy configuration. + ```yaml proxy: policies_middleware: query: data.proxy.granted ``` -The same can be achieved by setting the `PROXY_POLICIES_QUERY=data.proxy.granted` environment variable. +The same can be achieved by setting the following evironment variable: -### ASYNC Postprocessing +```yaml +PROXY_POLICIES_QUERY=data.proxy.granted +``` + +### Postprocessing ```yaml policies: @@ -70,4 +100,24 @@ policies: query: data.postprocessing.granted ``` -The same can be achieved by setting the `POLICIES_POSTPROCESSING_QUERY=data.postprocessing.granted` environment variable. As soon as that query is configured correctly, postprocessing must be informed to use the policies step by setting the environment variable `POSTPROCESSING_STEPS=policies`. Note that additional steps can be configured and their appearance defines the order of processing. For details see the postprocessing service documentation. +The same can be achieved by setting the following evironment variable: + +```yaml +POLICIES_POSTPROCESSING_QUERY=data.postprocessing.granted +``` + +As soon as that query is configured, the postprocessing service must be informed to use the policies step by setting the environment variable: + +```yaml +POSTPROCESSING_STEPS=policies +``` + +Note that additional steps can be configured and their position in the list defines the order of processing. For details see the postprocessing service documentation. + +## Rego Key Match + +To identify available keys for OPA, you need to look at [engine.go](https://github.com/owncloud/ocis/blob/master/services/policies/pkg/engine/engine.go) and the [policies.swagger.json](https://github.com/owncloud/ocis/blob/master/protogen/gen/ocis/services/policies/v0/policies.swagger.json) file. Note that which keys are avaialble depends from which module it is used. + +## Example Policies + +The policies service contains a set of preconfigured example policies. See the [deployment examples](https://github.com/owncloud/ocis/tree/master/deployments/examples) directory for details. The contained policies disallow Infinite Scale to create certain file types, both via the proxy middleware and the events service via postprocessing. diff --git a/services/postprocessing/README.md b/services/postprocessing/README.md index 2e8329cfb..d5717b343 100644 --- a/services/postprocessing/README.md +++ b/services/postprocessing/README.md @@ -14,7 +14,7 @@ The `postprocessing` service will then coordinate configured postprocessing step When all postprocessing steps have completed successfully, the file will be made accessible for users. -## Additional Prerequisites for the `postprocessing` Service +## Additional Prerequisites for the Postprocessing Service When postprocessing has been enabled, configuring any postprocessing step will require the requested services to be enabled and pre-configured. For example, to use the `virusscan` step, one needs to have an enabled and configured `antivirus` service. @@ -41,4 +41,4 @@ When setting a custom postprocessing step (eg. `"customstep"`) the postprocessin Once the custom service has finished its work, it should sent an event of type `PostprocessingFinished` via the configured events system. This event needs to contain a `FinishedStep` field set to `"customstep"`. It also must contain the outcome of the step, which can be one of "delete" (abort postprocessing, delete the file), "abort" (abort postprocessing, keep the file) and "continue" (continue postprocessing, this is the success case). -See https://github.com/cs3org/reva/blob/edge/pkg/events/postprocessing.go for up-to-date information of reserved step names and event definitons. +See the [cs3 org](https://github.com/cs3org/reva/blob/edge/pkg/events/postprocessing.go) for up-to-date information of reserved step names and event definitons. diff --git a/services/userlog/README.md b/services/userlog/README.md index a9c3476f3..b44fc931a 100644 --- a/services/userlog/README.md +++ b/services/userlog/README.md @@ -34,17 +34,17 @@ To delete events for an user, use a `DELETE` request to `ocs/v2.php/apps/notific ## Translations -The `userlog` service has embedded translations sourced via transifex to provide a basic set of translated languages. These embedded translations are available for all deployment scenarios. In addition, the service supports custom translations, though it is currently not possible to just add custom translations to embedded ones. If custom translations are configured, the embedded ones are not used. To configure custom translations, the `USERLOG_TRANSLATION_PATH` environment variable needs to point to a base folder that will further contain the translation files. This path must be available from all instances of the userlog service, a shared storage is recommended. Translation files must be of type [.po](https://www.gnu.org/software/gettext/manual/html_node/PO-Files.html#PO-Files) or [.mo](https://www.gnu.org/software/gettext/manual/html_node/Binaries.html). For each language, the filename needs to be `userlog.po` (or `userlog.mo`) and stored in a folder structure defining the language code. In general the path/name pattern for a translation file needs to be: +The `userlog` service has embedded translations sourced via transifex to provide a basic set of translated languages. These embedded translations are available for all deployment scenarios. In addition, the service supports custom translations, though it is currently not possible to just add custom translations to embedded ones. If custom translations are configured, the embedded ones are not used. To configure custom translations, the `USERLOG_TRANSLATION_PATH` environment variable needs to point to a base folder that will contain the translation files. This path must be available from all instances of the userlog service, a shared storage is recommended. Translation files must be of type [.po](https://www.gnu.org/software/gettext/manual/html_node/PO-Files.html#PO-Files) or [.mo](https://www.gnu.org/software/gettext/manual/html_node/Binaries.html). For each language, the filename needs to be `userlog.po` (or `userlog.mo`) and stored in a folder structure defining the language code. In general the path/name pattern for a translation file needs to be: ```text {USERLOG_TRANSLATION_PATH}/{language-code}/LC_MESSAGES/userlog.po ``` -The language-code pattern is composed as `language[_territory]` where `language` is the base language and `_territory` is optional defining a country. +The language code pattern is composed of `language[_territory]` where `language` is the base language and `_territory` is optional and defines a country. -As example, for the language `de_DE`, one needs to place the corresponding translation files to `{USERLOG_TRANSLATION_PATH}/de_DE/LC_MESSAGES/userlog.po`. +For example, for the language `de_DE`, one needs to place the corresponding translation files to `{USERLOG_TRANSLATION_PATH}/de_DE/LC_MESSAGES/userlog.po`. ### Translation Rules -* If a requested language-code is not available, the service tries to fallback to the base language if available. As example, if `de_DE` is not available, the service tries to fall back to translations in `de` folder. -* If the base language is also not available like when the language code is `de_DE` and neither `de_DE` nor the `de` folder is available, the service falls back to the systems default `en`, which is the source of the texts provided by the code. +* If a requested language code is not available, the service tries to fall back to the base language if available. For example, if `de_DE` is not available, the service tries to fall back to translations in the `de` folder. +* If the base language `de` is also not available, the service falls back to the system's default English (`en`), which is the source of the texts provided by the code. diff --git a/services/webfinger/README.md b/services/webfinger/README.md index 1ba4d95fc..f306e97d9 100644 --- a/services/webfinger/README.md +++ b/services/webfinger/README.md @@ -1,12 +1,13 @@ # Webfinger Service -The webfinger service provides an RFC7033 WebFinger lookup of ownCloud instances relevant for a given user account. +The webfinger service provides an RFC7033 WebFinger lookup of ownCloud instances relevant for a given user account via endpoints a the /.well-known/webfinger implementation. It is based on https://github.com/owncloud/lookup-webfinger-sciebo but also returns localized `titles` in addition to the `href` property. ## OpenID Connect Discovery Clients can make an unauthenticated `GET https://drive.ocis.test/.well-known/webfinger?resource=https%3A%2F%2Fcloud.ocis.test` request to discover the OpenID Connect Issuer in the `http://openid.net/specs/connect/1.0/issuer` relation: + ```json { "subject": "acct:einstein@drive.ocis.test", @@ -24,6 +25,7 @@ Here, the `resource` takes the instance domain URI, but an `acct:` URI works as ## Authenticated Instance Discovery When using OpenID connect to authenticate requests, clients can look up the owncloud instances a user has access to. + * Authentication is necessary to prevent leaking information about existing users. * Basic auth is not supported. @@ -51,6 +53,7 @@ The default configuration will simply return the `OCIS_URL` and direct clients t ## Configure Different Instances Based on OpenidConnect UserInfo Claims A more complex example for configuring different instances could look like this: + ```yaml webfinger: instances: @@ -119,4 +122,4 @@ Now, an authenticated webfinger request for `acct:me@example.org` (when logged i } ] } -``` \ No newline at end of file +```