mirror/opencloud
1
0
Fork 0
mirror of https://github.com/opencloud-eu/opencloud.git synced 2026-01-03 02:39:52 -06:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into nats-check-for-error

Browse Source
This commit is contained in:
Willy Kloucek
2022-03-24 12:27:15 +01:00
parent b2b11c4de8 b0745e115f
commit 90d5ef3c9d
17 changed files with 136 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
.drone.env
View File

@@ -1,7 +1,7 @@
# The test runner source for API tests
CORE_COMMITID=5b738dfa70b5493cb305123ad1d1a2c3055863c0
CORE_COMMITID=87d2f9a69a11838b9b591a09da4d13f860dd4a31
CORE_BRANCH=master
# The test runner source for UI tests
WEB_COMMITID=bb612cfc1c63316a159c7e29b81438595cef8fdb
WEB_COMMITID=77faf9890974083c0c555fd83586c2448845b11d
WEB_BRANCH=master

2
.drone.star
View File

@@ -1753,7 +1753,7 @@ def deploy(ctx, config, rebuild):
},
{
"name": "deploy",
"image": SELENIUM_STANDALONE_CHROME_DEBUG,
"image": OC_CI_DRONE_ANSIBLE,
"failure": "ignore",
"environment": {
"CONTINUOUS_DEPLOY_SERVERS_CONFIG": "../%s" % (config),

1
.github/settings.yml vendored
View File

@@ -65,3 +65,4 @@ branches:
- employees
...

20
CHANGELOG.md
View File

@@ -9,6 +9,7 @@ The following sections list the changes for unreleased.
* Bugfix - Network configuration in individiual_services example: [#3238](https://github.com/owncloud/ocis/pull/3238)
* Bugfix - Improve gif thumbnails: [#3305](https://github.com/owncloud/ocis/pull/3305)
* Bugfix - Fix error handling in GraphAPI GetUsers call: [#3357](https://github.com/owncloud/ocis/pull/3357)
* Bugfix - Replace public mountpoint fileid with grant fileid: [#3349](https://github.com/owncloud/ocis/pull/3349)
* Change - Settings service now stores its data via metadata service: [#3232](https://github.com/owncloud/ocis/pull/3232)
* Enhancement - Audit logger will now log file events: [#3332](https://github.com/owncloud/ocis/pull/3332)
* Enhancement - Add password reset link to login page: [#3329](https://github.com/owncloud/ocis/pull/3329)
@@ -17,7 +18,7 @@ The following sections list the changes for unreleased.
* Enhancement - Include etags in drives listing: [#3267](https://github.com/owncloud/ocis/pull/3267)
* Enhancement - Improve thumbnails API: [#3272](https://github.com/owncloud/ocis/pull/3272)
* Enhancement - Update reva to vXXXX: [#3330](https://github.com/owncloud/ocis/pull/3330)
* Enhancement - Update ownCloud Web to v5.3.0-rc.1: [#6561](https://github.com/owncloud/web/pull/6561)
* Enhancement - Update ownCloud Web to v5.3.0: [#6561](https://github.com/owncloud/web/pull/6561)
## Details
@@ -41,6 +42,14 @@ The following sections list the changes for unreleased.
https://github.com/owncloud/ocis/pull/3357
* Bugfix - Replace public mountpoint fileid with grant fileid: [#3349](https://github.com/owncloud/ocis/pull/3349)
We now show the same resoucre id for resources when accessing them via a public links as when
using a logged in user. This allows the web ui to start a WOPI session with the correct resource
id.
https://github.com/owncloud/ocis/pull/3349
* Change - Settings service now stores its data via metadata service: [#3232](https://github.com/owncloud/ocis/pull/3232)
Instead of writing files to disk it will use metadata service to do so
@@ -96,16 +105,17 @@ The following sections list the changes for unreleased.
https://github.com/owncloud/ocis/pull/3330
* Enhancement - Update ownCloud Web to v5.3.0-rc.1: [#6561](https://github.com/owncloud/web/pull/6561)
* Enhancement - Update ownCloud Web to v5.3.0: [#6561](https://github.com/owncloud/web/pull/6561)
Tags: web
We updated ownCloud Web to v5.3.0-rc.1. Please refer to the changelog (linked) for details on
the web release.
We updated ownCloud Web to v5.3.0. Please refer to the changelog (linked) for details on the web
release.
https://github.com/owncloud/web/pull/6561
https://github.com/owncloud/ocis/pull/3291
https://github.com/owncloud/web/releases/tag/v5.3.0-rc.1
https://github.com/owncloud/ocis/pull/3375
https://github.com/owncloud/web/releases/tag/v5.3.0
# Changelog for [1.18.0] (2022-03-03)
The following sections list the changes for 1.18.0.

5
changelog/unreleased/publicstorageprovider-rewrite.md Normal file
View File

@@ -0,0 +1,5 @@
Bugfix: replace public mountpoint fileid with grant fileid
We now show the same resoucre id for resources when accessing them via a public links as when using a logged in user. This allows the web ui to start a WOPI session with the correct resource id.
https://github.com/owncloud/ocis/pull/3349

9
changelog/unreleased/update-web-5.3.0-rc.1.md
View File

@@ -1,9 +0,0 @@
Enhancement: Update ownCloud Web to v5.3.0-rc.1
Tags: web
We updated ownCloud Web to v5.3.0-rc.1. Please refer to the changelog (linked) for details on the web release.
https://github.com/owncloud/web/pull/6561
https://github.com/owncloud/ocis/pull/3291
https://github.com/owncloud/web/releases/tag/v5.3.0-rc.1

10
changelog/unreleased/update-web-5.3.0.md Normal file
View File

@@ -0,0 +1,10 @@
Enhancement: Update ownCloud Web to v5.3.0
Tags: web
We updated ownCloud Web to v5.3.0. Please refer to the changelog (linked) for details on the web release.
https://github.com/owncloud/web/pull/6561
https://github.com/owncloud/ocis/pull/3291
https://github.com/owncloud/ocis/pull/3375
https://github.com/owncloud/web/releases/tag/v5.3.0

4
docs/clients/rclone/webdav-sync-basic-auth.md
View File

@@ -12,7 +12,7 @@ geekdocCollapseSection: true
## WebDAV with Basic Authentication
{{< hint danger >}}
Basic Authentication is disabled by default in oCIS because of security considerations. In order to make the following Rclone commands work the oCIS administrator needs to enable Basic Authentication eg. by setting the the environment variable `PROXY_ENABLE_BASIC_AUTH` to `true`.
Basic Authentication is disabled by default in oCIS because of security considerations. In order to make the following Rclone commands work the oCIS administrator needs to enable Basic Authentication e.g. by setting the environment variable `PROXY_ENABLE_BASIC_AUTH` to `true`.
Please consider to use [Rclone with OpenID Connect]({{< ref "webdav-sync-oidc.md" >}}) instead.
{{< /hint >}}
@@ -43,4 +43,4 @@ We now can use Rclone to sync the local folder `/tmp/test` to `/test` in your oC
rclone sync :local:/tmp :webdav:/test
```
If your oCIS doesn't use valid SSL certificates, you may need to use `rclone --no-check-certificate sync ...`.
If your oCIS doesn't use valid SSL certificates, you may need to use `rclone --no-check-certificate sync ...`.

57
docs/extensions/idm/setup.md Normal file
View File

@@ -0,0 +1,57 @@
---
title: Service Setup
date: 2022-03-22T00:00:00+00:00
weight: 20
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/extensions/idm
geekdocFilePath: setup.md
geekdocCollapseSection: true
---
{{< toc >}}
## Using ocis with libregraph/idm
Currently, oCIS still runs the accounts and glauth services to manage users. Until the default is switched
to libregraph/idm, oCIS has to be started with a custom configuration in order to use libregraph/idm as
the users and groups backend (this setup also disables the glauth and accounts service):
```
export GRAPH_IDENTITY_BACKEND=ldap
export GRAPH_LDAP_URI=ldaps://localhost:9235
export GRAPH_LDAP_BIND_DN="uid=libregraph,ou=sysusers,o=libregraph-idm"
export GRAPH_LDAP_BIND_PASSWORD=idm
export GRAPH_LDAP_USER_EMAIL_ATTRIBUTE=mail
export GRAPH_LDAP_USER_NAME_ATTRIBUTE=uid
export GRAPH_LDAP_USER_BASE_DN="ou=users,o=libregraph-idm"
export GRAPH_LDAP_GROUP_BASE_DN="ou=groups,o=libregraph-idm"
export GRAPH_LDAP_SERVER_WRITE_ENABLED="true"
export IDP_INSECURE="true"
export IDP_LDAP_FILTER="(&(objectclass=inetOrgPerson)(objectClass=owncloud))"
export IDP_LDAP_URI=ldaps://localhost:9235
export IDP_LDAP_BIND_DN="uid=idp,ou=sysusers,o=libregraph-idm"
export IDP_LDAP_BIND_PASSWORD="idp"
export IDP_LDAP_BASE_DN="ou=users,o=libregraph-idm"
export IDP_LDAP_LOGIN_ATTRIBUTE=uid
export IDP_LDAP_UUID_ATTRIBUTE="ownclouduuid"
export IDP_LDAP_UUID_ATTRIBUTE_TYPE=binary
export PROXY_ACCOUNT_BACKEND_TYPE=cs3
export OCS_ACCOUNT_BACKEND_TYPE=cs3
export STORAGE_LDAP_HOSTNAME=localhost
export STORAGE_LDAP_PORT=9235
export STORAGE_LDAP_INSECURE="true"
export STORAGE_LDAP_BASE_DN="o=libregraph-idm"
export STORAGE_LDAP_BIND_DN="uid=reva,ou=sysusers,o=libregraph-idm"
export STORAGE_LDAP_BIND_PASSWORD=reva
export STORAGE_LDAP_LOGINFILTER='(&(objectclass=inetOrgPerson)(objectclass=owncloud)(|(uid={{login}})(mail={{login}})))'
export STORAGE_LDAP_USERFILTER='(&(objectclass=inetOrgPerson)(objectclass=owncloud)(|(ownclouduuid={{.OpaqueId}})(uid={{.OpaqueId}})))'
export STORAGE_LDAP_USERATTRIBUTEFILTER='(&(objectclass=owncloud)({{attr}}={{value}}))'
export STORAGE_LDAP_USERFINDFILTER='(&(objectclass=owncloud)(|(uid={{query}}*)(cn={{query}}*)(displayname={{query}}*)(mail={{query}}*)(description={{query}}*)))'
export STORAGE_LDAP_USERGROUPFILER='(&(objectclass=groupOfNames)(member={{query}}*))'
export STORAGE_LDAP_GROUPFILTER='(&(objectclass=groupOfNames)(objectclass=owncloud)(ownclouduuid={{.OpaqueId}}*))'
export OCIS_RUN_EXTENSIONS=settings,storage-metadata,graph,graph-explorer,ocs,store,thumbnails,web,webdav,storage-frontend,storage-gateway,storage-userprovider,storage-groupprovider,storage-authbasic,storage-authbearer,storage-authmachine,storage-users,storage-shares,storage-public-link,storage-appprovider,storage-sharing,proxy,idp,nats,idm
export OCIS_INSECURE=true
bin/ocis server
```

6
docs/ocis/development/testing.md
View File

@@ -28,7 +28,7 @@ Basically we have two sources for feature tests and test suites:
At the moment both can be applied to oCIS since the api of oCIS is designed to be compatible to ownCloud.
Since we have to offer an migration path to existing users of ownCloud, you can use your existing ownCloud as storage backend for oCIS. As another storage backend we offer oCIS native storage, also called "oCIS". This stores files directly on disk. Which storage backend is used is also reflected in the tests, there are always different tests for oCIS storage and ownCloud storage.
Since we have to offer a migration path to existing users of ownCloud, you can use your existing ownCloud as storage backend for oCIS. As another storage backend we offer oCIS native storage, also called "oCIS". This stores files directly on disk. Which storage backend is used is also reflected in the tests, there are always different tests for oCIS storage and ownCloud storage.
You can invoke two types of test suite runs:
@@ -55,7 +55,7 @@ This must be pointing to a valid feature definition.
### oCIS image to be tested (or: skip build and take existing image)
By default the tests will be run against docker image built from your current working state of the oCIS repository. For some purposes it might also be handy to use a oCIS image from Docker Hub. Therefore you can provide the optional flag `OCIS_IMAGE_TAG=...` which must contain an available docker tag of the [owncloud/ocis registry on Docker Hub](https://hub.docker.com/r/owncloud/ocis) (eg. 'latest').
By default, the tests will be run against the docker image built from your current working state of the oCIS repository. For some purposes it might also be handy to use a oCIS image from Docker Hub. Therefore you can provide the optional flag `OCIS_IMAGE_TAG=...` which must contain an available docker tag of the [owncloud/ocis registry on Docker Hub](https://hub.docker.com/r/owncloud/ocis) (eg. 'latest').
```
make -C tests/acceptance/docker localApiTests-apiAccountsHashDifficulty-ocis OCIS_IMAGE_TAG=latest
@@ -75,7 +75,7 @@ The log output is opened in `less`. You can navigate up and down with your curso
### Cleanup
During testing we start an redis and oCIS docker container. These will not be stopped automatically. You can stop them with:
During testing we start a redis and oCIS docker container. These will not be stopped automatically. You can stop them with:
```
make -C tests/acceptance/docker clean

4
docs/ocis/getting-started/_index.md
View File

@@ -22,7 +22,7 @@ We are distributing oCIS as binaries and Docker images.
{{< hint warning >}}
The examples in this document assume that oCIS is accessed from the same host as it is running on (`localhost`). If you would like
to access oCIS remotely please refer to the [Basic Remote Setup]({{< ref "../deployment/basic-remote-setup" >}}) section. Especially
to the notes about setting the `PROXY_HTTP_ADDR` and `OCIS_URL` enviroment variables.
to the notes about setting the `PROXY_HTTP_ADDR` and `OCIS_URL` environment variables.
{{< /hint >}}
You can find more deployment examples in the [deployment section]({{< ref "../deployment" >}}).
@@ -113,7 +113,7 @@ The version command prints the version of your installed oCIS.
ocis --version
{{< / highlight >}}
The health command is used to execute a health check, if the exit code equals zero the service should be up and running, if the exist code is greater than zero the service is not in a healthy state. Generally this command is used within our Docker containers, it could also be used within Kubernetes.
The health command is used to execute a health check, if the exit code equals zero the service should be up and running, if the exit code is greater than zero the service is not in a healthy state. Generally this command is used within our Docker containers, it could also be used within Kubernetes.
{{< highlight txt >}}
ocis health --help

16
docs/ocis/migration.md
View File

@@ -45,7 +45,7 @@ _TODO allow limiting the web ui switch to an 'early adopters' group_
</div>
#### Validation
Ensure switching back an forth between the classic ownCloud 10 web UI and ownCloud web works as at our https://demo.owncloud.com.
Ensure switching back and forth between the classic ownCloud 10 web UI and ownCloud web works as at our https://demo.owncloud.com.
#### Rollback
Should there be problems with ownCloud web at this point it can simply be removed from the menu and be undeployed.
@@ -82,7 +82,7 @@ When introducing OpenID Connect, the clients will detect the new authentication
reauthorize at the OpenID Connect IdP, which again, may be configured to skip the consent step for trusted clients.
#### Steps
1. There are multiple products that can be used as an OpenID Connect IdP. We test with [LibreGraph Connect](https://github.com/libregraph/lico), which is also [embedded in oCIS](https://github.com/owncloud/web/). Other alternatives include [Keycloak](https://www.keycloak.org/) or [Ping](https://www.pingidentity.com/). Please refer to the corresponding setup instructions for the product you intent to use.
1. There are multiple products that can be used as an OpenID Connect IdP. We test with [LibreGraph Connect](https://github.com/libregraph/lico), which is also [embedded in oCIS](https://github.com/owncloud/web/). Other alternatives include [Keycloak](https://www.keycloak.org/) or [Ping](https://www.pingidentity.com/). Please refer to the corresponding setup instructions for the product you intend to use.
<div class="editpage">
@@ -308,7 +308,7 @@ _Feel free to add your question as a PR to this document using the link at the t
<div style="break-after: page"></div>
### Stage-6: parallel deployment
Running ownCloud 10 and oCIS in parallel is a crucial stage for the migration: it allows users access to group shares regardless of the system that is being used to to access the data. A user by user migration with multiple domains would technically break group shares when users vanish because they (and their data) are no longer available in the old system.
Running ownCloud 10 and oCIS in parallel is a crucial stage for the migration: it allows users access to group shares regardless of the system that is being used to access the data. A user by user migration with multiple domains would technically break group shares when users vanish because they (and their data) are no longer available in the old system.
Depending on the amount of power users on an instance, the admin may want to allow users to voluntarily migrate to the oCIS backend. A monitoring system can be used to visualize the behavior for the two systems and gain trust in the overall stability and performance.
@@ -324,7 +324,7 @@ _TODO @butonic update performance comparisons nightly_
#### Steps
There are several options to move users to the oCIS backend:
- Use a canary app to let users decide themselves
- Use an early adopters group with an opt in
- Use an early adopters group with an opt-in
- Force migrate users in batch or one by one at the administrators will
#### Verification
@@ -469,8 +469,8 @@ Depending on chosen the share manager provider some sharing requests should be f
_TODO for HA implement share manager with redis / nats / ... key value store backend: use the micro store interface please ..._
_TODO for batch migration implement share data migration cli with progress that reads all shares via the cs3 api from one provider and writes them into another provider_
_TODO for seamless migration implement tiered/chained share provider that reads share data from the old provider and writes newc shares to the new one_
_TODO for storage provider as source of truth persist ALL share data in the storage provider. Currently, part is stored in the share manager, part is in the storage provider. We can keep both, but the the share manager should directly persist its metadata to the storage system used by the storage provider so metadata is kept in sync_
_TODO for seamless migration implement tiered/chained share provider that reads share data from the old provider and writes new shares to the new one_
_TODO for storage provider as source of truth persist ALL share data in the storage provider. Currently, part is stored in the share manager, part is in the storage provider. We can keep both, but the share manager should directly persist its metadata to the storage system used by the storage provider so metadata is kept in sync_
</div>
@@ -588,7 +588,7 @@ The `filecache` table itself has more metadata:
| `checksum` | varchar(255) | YES | | NULL | | *same as blob checksum* | SHOULD become the checksum in the storage provider. eos calculates it itself, `ocis` driver stores it in extended attributes |
> Note: for EOS a hot migration only works seamlessly if file ids in oc10 are already read from eos. otherwise either a mapping from the oc10 filecache file id to the new eos file id has to be created under the assumption that these id sets do not intersect or files and corresponding shares need to be exported and imported offline to generate a new set of ids. While this will preserve public links, user, group and even federated shares, old internal links may still point to different files because they contain the oc10 fileid
> Note: for EOS a hot migration only works seamlessly if file ids in oc10 are already read from eos. Otherwise, either a mapping from the oc10 filecache file id to the new eos file id has to be created under the assumption that these id sets do not intersect or files and corresponding shares need to be exported and imported offline to generate a new set of ids. While this will preserve public links, user, group and even federated shares, old internal links may still point to different files because they contain the oc10 fileid
<div style="break-after: page"></div>
@@ -682,7 +682,7 @@ _TODO clarify how OCM handles this and where we store / configure this. It seems
Users are migrated in two steps:
1. They should all be authenticated using OpenID Connect, which already moves them to a common identity management system.
2. To search share recipients, both, ownCloud 10 and oCIS need access to the same user directory using eg. LDAP.
2. To search share recipients, both, ownCloud 10 and oCIS need access to the same user directory using e.g. LDAP.
<div class="editpage">

15
idm/ldif/base.ldif.tmpl
View File

@@ -22,3 +22,18 @@ uid: {{ .Name }}
userPassword:: {{ .Password }}
{{ end -}}
## Service user for the settings service
dn: uid=95cb8724-03b2-11eb-a0a6-c33ef8ef53ad,ou=users,o=libregraph-idm
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: ownCloud
objectClass: person
objectClass: top
uid: 95cb8724-03b2-11eb-a0a6-c33ef8ef53ad
givenName: 95cb8724-03b2-11eb-a0a6-c33ef8ef53ad
sn: 95cb8724-03b2-11eb-a0a6-c33ef8ef53ad
cn: 95cb8724-03b2-11eb-a0a6-c33ef8ef53ad
displayName: 95cb8724-03b2-11eb-a0a6-c33ef8ef53ad
ownCloudUUID: 95cb8724-03b2-11eb-a0a6-c33ef8ef53ad

8
settings/ui/tests/acceptance/helpers/language.js
View File

@@ -6,8 +6,8 @@ const filesMenu = {
'Deleted files'
],
Deutsch: [
'Persöhnlich',
'Shares',
'Persönlich',
'Geteilt',
'Spaces\nbeta',
'Gelöschte Dateien'
],
@@ -55,24 +55,28 @@ const accountMenu = {
const filesListHeaderMenu = {
English: [
'Name',
'Shares',
'Size',
'Modified',
'Actions'
],
Deutsch: [
'Name',
'Geteilt',
'Größe',
'Bearbeitet',
'Aktionen'
],
Español: [
'Nombre',
'Shares',
'Tamaño',
'Modificado',
'Acciones'
],
Français: [
'Nom',
'Shares',
'Taille',
'Modifié',
'Actions'

8
storage/pkg/command/gateway.go
View File

@@ -253,8 +253,12 @@ func spacesProviders(cfg *config.Config, logger log.Logger) map[string]map[strin
// public link storage returns the mount id of the actual storage
cfg.Reva.StoragePublicLink.Endpoint: {
"spaces": map[string]interface{}{
"public": map[string]interface{}{
"mount_point": "/public",
"grant": map[string]interface{}{
"mount_point": ".",
},
"mountpoint": map[string]interface{}{
"mount_point": "/public",
"path_template": "/public/{{.Space.Root.OpaqueId}}",
},
},
},

4
tests/acceptance/expected-failures-webUI-on-OCIS-storage.md
View File

@@ -308,7 +308,7 @@ Other free text and markdown formatting can be used elsewhere in the document if
- [webUISharingPublicDifferentRoles/shareByPublicLinkDifferentRoles.feature:60](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingPublicDifferentRoles/shareByPublicLinkDifferentRoles.feature#L60)
### [Listing shares via ocs API does not show path for parent folders](https://github.com/owncloud/ocis/issues/1231)
- [webUISharingPublicManagement/shareByPublicLink.feature:134](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingPublicManagement/shareByPublicLink.feature#L134)
- [webUISharingPublicManagement/shareByPublicLink.feature:133](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingPublicManagement/shareByPublicLink.feature#L133)
### [Propfind response to trashbin endpoint is different in ocis](https://github.com/owncloud/product/issues/186)
### [restoring a file from "Deleted files" (trashbin) is not possible if the original folder does not exist any-more](https://github.com/owncloud/web/issues/1753)
@@ -469,7 +469,7 @@ Other free text and markdown formatting can be used elsewhere in the document if
- [webUIUpload/upload.feature:129](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIUpload/upload.feature#L129)
- [webUIUpload/upload.feature:142](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIUpload/upload.feature#L142)
- [webUIUpload/upload.feature:159](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIUpload/upload.feature#L159)
- [webUIUpload/uploadEdgecases.feature:67](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIUpload/uploadEdgecases.feature#L67)
- [webUIUpload/uploadEdgecases.feature:69](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIUpload/uploadEdgecases.feature#L69)
### [browsing directly to a details 'tab' is not possible](https://github.com/owncloud/web/issues/5464)
- [webUIFiles/browseDirectlyToDetailsTab.feature:21](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIFiles/browseDirectlyToDetailsTab.feature#L21)

2
web/Makefile
View File

@@ -1,6 +1,6 @@
SHELL := bash
NAME := web
WEB_ASSETS_VERSION = v5.3.0-rc.1
WEB_ASSETS_VERSION = v5.3.0
include ../.make/recursion.mk