From 09da9305de8ca8b75df21b280dfd084c8ae77ad2 Mon Sep 17 00:00:00 2001 From: "A.Unger" Date: Tue, 24 Aug 2021 13:54:05 +0200 Subject: [PATCH 01/56] add profiling docs --- docs/ocis/development/profiling.md | 121 +++++++++++++++++++++++++++++ 1 file changed, 121 insertions(+) create mode 100644 docs/ocis/development/profiling.md diff --git a/docs/ocis/development/profiling.md b/docs/ocis/development/profiling.md new file mode 100644 index 0000000000..c43b506313 --- /dev/null +++ b/docs/ocis/development/profiling.md @@ -0,0 +1,121 @@ +--- +title: "Profiling" +date: 2021-08-24T12:32:20+01:00 +weight: 56 +geekdocRepo: https://github.com/owncloud/ocis +geekdocEditPath: edit/master/docs/ocis/development +geekdocFilePath: profiling.md +--- + +{{< toc >}} + +The only way to enable the profiler currently is to explicitly select which areas to collect samples for. In order to do this, the following steps have to be followed. + +## 1. Clone Reva + +`git clone github.com/cs3org/reva` + +## 2. Patch reva with the area that you want sampled. + +For the purposes of this docs let's use the WebDAV `PROPFIND` path. + +```diff +diff --git a/internal/http/services/owncloud/ocdav/propfind.go b/internal/http/services/owncloud/ocdav/propfind.go +index 0e9c99be..f271572f 100644 +--- a/internal/http/services/owncloud/ocdav/propfind.go ++++ b/internal/http/services/owncloud/ocdav/propfind.go +@@ -32,6 +32,8 @@ import ( + "strings" + "time" + ++ _ "net/http/pprof" ++ + userv1beta1 "github.com/cs3org/go-cs3apis/cs3/identity/user/v1beta1" + rpc "github.com/cs3org/go-cs3apis/cs3/rpc/v1beta1" + link "github.com/cs3org/go-cs3apis/cs3/sharing/link/v1beta1" +@@ -311,6 +313,12 @@ func requiresExplicitFetching(n *xml.Name) bool { + return true + } + ++func init() { ++ go func() { ++ http.ListenAndServe(":1234", nil) ++ }() ++} ++ + // from https://github.com/golang/net/blob/e514e69ffb8bc3c76a71ae40de0118d794855992/webdav/xml.go#L178-L205 + func readPropfind(r io.Reader) (pf propfindXML, status int, err error) { + c := countingReader{r: r} +``` + +The previous patch will: + +1. import `net/http/pprof`, which will register debug handlers in `DefaultServeMux`. +2. define a `init()` function that starts an http server with the previously registered handlers. + +With everything running one should have access to http://localhost:1234/debug/pprof/ + +## 3. Replace ocis go.mod to local reva and build a new binary + +```diff +diff --git a/go.mod b/go.mod +index 131d14d7b..9668c38e4 100644 +--- a/go.mod ++++ b/go.mod +@@ -78,6 +78,7 @@ require ( + + replace ( + github.com/crewjam/saml => github.com/crewjam/saml v0.4.5 ++ github.com/cs3org/reva => path/to/your/reva + go.etcd.io/etcd/api/v3 => go.etcd.io/etcd/api/v3 v3.0.0-20210204162551-dae29bb719dd + go.etcd.io/etcd/pkg/v3 => go.etcd.io/etcd/pkg/v3 v3.0.0-20210204162551-dae29bb719dd + ) +``` + +Make sure to replace `github.com/cs3org/reva => path/to/your/reva` with the correct location of your reva. + +## 4. Build a new ocis binary + +From owncloud/ocis root: + +```console +$ cd ocis +$ make clean build +``` + +## 5. Start oCIS server + +From owncloud/ocis root: + +```console +$ OCIS_LOG_PRETTY=true OCIS_LOG_COLOR=true ocis/bin/ocis server +``` + +## 6. Run `pprof` + +### Install pprof + +If `pprof` is not installed make sure to get it; one way of installing it is using the Go tools: + +```console +$ go get -u github.com/google/pprof +``` + +### Collecting samples + +Collect 30 seconds of samples: + +```console +$ prof -web http://:1234/debug/pprof/profile\?seconds\=30 +``` + +Once the collection is done a browser tab will open with the result `svg`, looking similar to this: + +![img](https://i.imgur.com/vo0EbcX.jpg) + +For references on how to interpret this graph, [continue reading here](https://github.com/google/pprof/blob/master/doc/README.md#interpreting-the-callgraph). + +## References + +- https://medium.com/swlh/go-profile-your-code-like-a-master-1505be38fdba +- https://dave.cheney.net/ From 147a8f8004082908d8ea7150efcbdc8d1f750f4c Mon Sep 17 00:00:00 2001 From: Alex Unger <6905948+refs@users.noreply.github.com> Date: Tue, 24 Aug 2021 15:48:54 +0200 Subject: [PATCH 02/56] Update docs/ocis/development/profiling.md Co-authored-by: Pascal Wengerter --- docs/ocis/development/profiling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ocis/development/profiling.md b/docs/ocis/development/profiling.md index c43b506313..51ddf89db8 100644 --- a/docs/ocis/development/profiling.md +++ b/docs/ocis/development/profiling.md @@ -55,7 +55,7 @@ The previous patch will: With everything running one should have access to http://localhost:1234/debug/pprof/ -## 3. Replace ocis go.mod to local reva and build a new binary +## 3. Replace reva in oCIS go.mod with local version and build a new binary ```diff diff --git a/go.mod b/go.mod From 06a4a8ef0e55528b596ce4980a0f6c767757118c Mon Sep 17 00:00:00 2001 From: Alex Unger <6905948+refs@users.noreply.github.com> Date: Tue, 24 Aug 2021 15:50:06 +0200 Subject: [PATCH 03/56] Apply suggestions from code review Co-authored-by: Pascal Wengerter --- docs/ocis/development/profiling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ocis/development/profiling.md b/docs/ocis/development/profiling.md index 51ddf89db8..be7091390d 100644 --- a/docs/ocis/development/profiling.md +++ b/docs/ocis/development/profiling.md @@ -106,7 +106,7 @@ $ go get -u github.com/google/pprof Collect 30 seconds of samples: ```console -$ prof -web http://:1234/debug/pprof/profile\?seconds\=30 +$ pprof -web http://:1234/debug/pprof/profile\?seconds\=30 ``` Once the collection is done a browser tab will open with the result `svg`, looking similar to this: From f5797866c5c1c5f278bd4edfb03052312efc4998 Mon Sep 17 00:00:00 2001 From: "A.Unger" Date: Tue, 24 Aug 2021 15:57:17 +0200 Subject: [PATCH 04/56] docs: apply review --- docs/ocis/development/profiling.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/ocis/development/profiling.md b/docs/ocis/development/profiling.md index be7091390d..22807c00a5 100644 --- a/docs/ocis/development/profiling.md +++ b/docs/ocis/development/profiling.md @@ -9,6 +9,15 @@ geekdocFilePath: profiling.md {{< toc >}} +# 0. Prerequisites + +- Go development kit of a [supported version](https://golang.org/doc/devel/release.html#policy). + Follow [these instructions](http://golang.org/doc/code.html) to install the + go tool and set up GOPATH. + +- Graphviz: http://www.graphviz.org/ + Optional, used to generate graphic visualizations of profiles + The only way to enable the profiler currently is to explicitly select which areas to collect samples for. In order to do this, the following steps have to be followed. ## 1. Clone Reva @@ -118,4 +127,4 @@ For references on how to interpret this graph, [continue reading here](https://g ## References - https://medium.com/swlh/go-profile-your-code-like-a-master-1505be38fdba -- https://dave.cheney.net/ +- https://dave.cheney.net/2013/07/07/introducing-profile-super-simple-profiling-for-go-programs From 93c833348e149f3f3db3531f12c0c585c6ff2c92 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rn=20Friedrich=20Dreyer?= Date: Thu, 22 Jul 2021 15:36:48 +0000 Subject: [PATCH 05/56] separate proposed changes from terminology MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Jörn Friedrich Dreyer --- docs/extensions/storage/proposedchanges.md | 114 +++++ .../storage/static/spacesregistry.drawio.svg | 434 ++++++++++++++++++ .../storage/static/storageprovider.drawio.svg | 390 ++++++++++++---- .../static/storageregistry-spaces.drawio.svg | 327 ------------- docs/extensions/storage/storages.md | 10 +- docs/extensions/storage/terminology.md | 158 ++----- docs/extensions/storage/updating.md | 4 +- docs/ocis/_index.md | 14 +- docs/ocis/migration.md | 49 +- 9 files changed, 924 insertions(+), 576 deletions(-) create mode 100644 docs/extensions/storage/proposedchanges.md create mode 100644 docs/extensions/storage/static/spacesregistry.drawio.svg delete mode 100644 docs/extensions/storage/static/storageregistry-spaces.drawio.svg diff --git a/docs/extensions/storage/proposedchanges.md b/docs/extensions/storage/proposedchanges.md new file mode 100644 index 0000000000..e8c1a14cec --- /dev/null +++ b/docs/extensions/storage/proposedchanges.md @@ -0,0 +1,114 @@ +--- +title: "Proposed Changes" +date: 2018-05-02T00:00:00+00:00 +weight: 18 +geekdocRepo: https://github.com/owncloud/ocis +geekdocEditPath: edit/master/docs/extensions/storage +geekdocFilePath: proposedchanges.md +--- + +Some architectural changes still need to be clarified or changed. Maybe an ADR is in order for all of the below. + +## Reva Gateway changes + +## A dedicated shares storage provider + +Currently, the *gateway* treats `/home/shares` different than any other path: it will stat all children and calculate an etag to allow clients to discover changes in accepted shares. This requires the storage provider to cooperate and provide this special `/shares` folder in the root of a users home when it is accessed as a home storage, which is a config flag that needs to be set for every storage driver. + +The `enable_home` flag will cause drivers to jail path based requests into a `` subfolder. In effect it divides a storage provider into multiple [*storage spaces*]({{< ref "#storage-spaces" >}}): when calling `CreateHome` a subfolder following the `` is created and market as the root of a users home. Both, the eos and ocis storage drivers use extended attributes to mark the folder as the end of the size aggregation and tree mtime propagation mechanism. Even setting the quota is possible like that. All this literally is a [*storage space*]({{< ref "#storage-spaces" >}}). + +We can implement [ListStorageSpaces](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ListStorageSpacesRequest) by either +- iterating over the root of the storage and treating every folder following the `` as a `home` *storage space*, +- iterating over the root of the storage and treating every folder following a new `` as a `project` *storage space*, or +- iterating over the root of the storage and treating every folder following a generic `` as a *storage space* for a configurable space type, or +- we allow configuring a map of `space type` to `layout` (based on the [CreateStorageSpaceRequest](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.CreateStorageSpaceRequest)) which would allow things like +``` +home=/var/lib/ocis/storage/home/{{substr 0 1 .Owner.Username}}/{{.Owner.Username}} +spaces=/spaces/var/lib/ocis/storage/projects/{{.Name}} +``` + +This would make the `GetHome()` call return the path to the *storage provider* including the relative path to the *storage space*. No need for a *storage provider* mounted at `/home`. This is just a UI alias for `/users/`. Just like a normal `/home/` on a linux machine. + +But if we have no `/home` where do we find the shares, and how can clients discover changes in accepted shares? + +The `/shares` namespace should be provided by a *shares storage provider* that lists all accepted shares for the current user... but what about copy pasting links from the browser? Well this storage is only really needed to have a path to ocm shares that actually reside on other instances. In the UI the shares would be listed by querying a *share manager*. It returns ResourceIds, which can be stated to fetch a path that is then accessible in the CS3 global namespace. Two caveats: +- This only works for resources that are actually hosted by the current instance. For those it would leak the parent path segments to a shared resource. +- For accepted OCM shares there must be a path in the [*CS3 global namespace*]({{< ref "./namespaces.md#cs3-global-namespaces" >}}) that has to be the same for all users, otherwise they cannot copy and share those URLs. + +Work on this is done in https://github.com/cs3org/reva/pull/1846 + +### The gateway should be responsible for path transformations + +Currently, storage providers are aware af their mount point, coupling them tightly with the gateway. + +Tracked in https://github.com/cs3org/reva/issues/578 + +Work is done in https://github.com/cs3org/reva/pull/1866 + +## URL escaped string representation of a CS3 reference + +For the `/dav/spaces/` endpoint we need to encode the *reference* in a url compatible way. +1. We can separate the path using a `/`: `/dav/spaces//` +2. The `spaceid` currently is a cs3 resourceid, consisting of `` and ``. Since the nodeid might contain `/` eg. for the local driver we have to urlencode the spaceid. + +To access resources by id we need to make the `/dav/meta/` able to list directories... Otherwise id based navigation first has to look up the path. Or we use the libregraph api for id based navigation. + +A *reference* is a logical concept. It identifies a [*resource*]({{< ref "#resources" >}}) and consists of a `` and a ``. A `` consists of a `` and a ``. They can be concatenated using the separators `!` and `:`: +``` +!: +``` +While all components are optional, only three cases are used: +| format | example | description | +|-|-|-| +| `!:` | `!:/absolute/path/to/file.ext` | absolute path | +| `!:` | `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!:path/to/file.ext` | path relative to the root of the storage space | +| `!:` | `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!c3cf23bb-8f47-4719-a150-1d25a1f6fb56:to/file.ext` | path relative to the specified node in the storage space, used to reference resources without disclosing parent paths | + +`` should be a UUID to prevent references from breaking when a *user* or [*storage space*]({{< ref "#storage-spaces" >}}) gets renamed. But it can also be derived from a migration of an oc10 instance by concatenating an instance identifier and the numeric storage id from oc10, e.g. `oc10-instance-a$1234`. + +A reference will often start as an absolute/global path, e.g. `!:/home/Projects/Foo`. The gateway will look up the storage provider that is responsible for the path + +| Name | Description | Who resolves it? | +|------|-------------|-| +| `!:/home/Projects/Foo` | the absolute path a client like davfs will use. | The gateway uses the storage registry to look up the responsible storage provider | +| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!:/Projects/Foo` | the `storage_space` is the same as the `root`, the path becomes relative to the root | the storage provider can use this reference to identify this resource | + +Now, the same file is accessed as a share +| Name | Description | +|------|-------------| +| `!:/users/Einstein/Projects/Foo` | `Foo` is the shared folder | +| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:` | `56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a` is the id of `Foo`, the path is empty | + + +The `:`, `!` and `$` are chosen from the set of [RFC3986 sub delimiters](https://tools.ietf.org/html/rfc3986#section-2.2) on purpose. They can be used in URLs without having to be encoded. In some cases, a delimiter can be left out if a component is not set: +| reference | interpretation | +|-|-| +| `/absolute/path/to/file.ext` | absolute path, all delimiters omitted | +| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!path/to/file.ext` | relative path in the given storage space, root delimiter `:` omitted | +| `56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:to/file.ext` | relative path in the given root node, storage space delimiter `!` omitted | +| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:` | node id in the given storage space, `:` must be present | +| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62` | root of the storage space, all delimiters omitted, can be distinguished by the `/` | + +## space providers +When looking up an id based resource the reference must use a logical space id, not a CS3 resource id. Otherwise id based requests, which only have a resourceid consisting of a storage id and a node id cannot be routed to the correct storage provider if the storage has moved from one storage provider to another. + +if the registry routes based on the storageid AND the nodeid it has to keep a cache of all nodeids in order to route all requests for a storage space (which consists of storage it + nodeid) to the correct storage provider. the correct resourceid for a node in a storage space would be `$!`. The `$` part allow the storage registry to route all id based requests to the correct storage provider. This becomes relevant when the storage space was moved from one storage provider to another. The storage space id remains the same, but the internal address and port change. + +TODO discuss to clarify further + +## Storage drivers + +### allow clients to send a uuid on upload +iOS clients can only queue single requests to be executed in the background. They queue an upload and need to be able to identify the uploaded file after it has been uploaded to the server. The disconnected nature of the connection might cause workflows or manual user interaction with the file on the server to move the file to a different place or changing the content while the device is offline. However, on the device users might have marked the file as favorite or added it to other iOS specific collections. To be able to reliably identify the file the client can generate a `uuid` and attach it to the file metadata during the upload. While it is not necessary to look up files by this `uuid` having a second file id that serves exactly the same purpose as the `file id` is redundant. + +Another aspect for the `file id` / `uuid` is that it must be a logical identifier that can be set, at least by internal systems. Without a writeable fileid we cannot restore backups or migrate storage spaces from one storage provider to another storage provider. + +Technically, this means that every storage driver needs to have a map of a `uuid` to an internal resource identifier. This internal resource identifier can be +- an eos fileid, because eos can look up files by id +- an inode if the filesystem and the storage driver support looking up by inode +- a path if the storage driver has no way of looking up files by id. + - In this case other mechanisms like inotify, kernel audit or a fuse overlay might be used to keep the paths up to date. + - to prevent excessive writes when deep folders are renamed a reverse map might be used: it will map the `uuid` to `:`, in order to trade writes for reads + - as a fallback a sync job can read the file id from the metadata of the resources and populate the uuid to internal id map. + +The TUS upload can take metadata, for PUT we might need a header. \ No newline at end of file diff --git a/docs/extensions/storage/static/spacesregistry.drawio.svg b/docs/extensions/storage/static/spacesregistry.drawio.svg new file mode 100644 index 0000000000..e3a771b07e --- /dev/null +++ b/docs/extensions/storage/static/spacesregistry.drawio.svg @@ -0,0 +1,434 @@ + + + + + + + +
+
+
+ oCIS System +
+ [Software System] +
+
+
+ + + oCIS System... + + + + + + + + + +
+
+
+ + Einstein + +
+ [Person] +
+
+
+ End user +
+
+
+
+ + + Einstein... + + + + + + + +
+
+
+ + Client + +
+ [Container: C++, Kotlin, Swift or Vue] +
+
+
+ A desktop, mobile or web Client +
+
+
+
+ + + Client... + + + + + + + +
+
+
+ + Storage Space Registry + +
+ [Container: golang, HTTP, libregraph] +
+
+
+ Manages spaces for users +
+
+
+
+ + + Storage Space Registry... + + + + + + + +
+
+
+ + Storage Provider + +
+ [Container: golang] +
+
+
+ Persists storage spaces using reva +
+
+
+
+ + + Storage Provider... + + + + + + + +
+
+
+ + Storage System + +
+ [Software System] +
+
+
+ provides persistent storage +
+
+
+
+ + + Storage System... + + + + + + + + + +
+
+
+ + Moss + +
+ [Person] +
+
+
+ Administrator +
+
+
+
+ + + Moss... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [POSIX, S3] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [WebDAV, libregraph, CS3, tus] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + + +
+
+
+
+
+ + Manages the users Storage Spaces in + +
+
+ [libregraph] +
+
+
+
+
+ + + Manages the users Storage Spac... + + + + + + + + +
+
+
+
+
+ + Manages resources with + +
+
+ [Web UI or native clients] +
+
+
+
+
+ + + Manages resources with... + + + + + + + + +
+
+
+
+
+ + Registers itself at and +
+ sends space root etag changes to + +
+
+ [CS3, libregraph?, PUSH] +
+
+
+
+
+ + + Registers itself at and... + + + + + + + + +
+
+
+
+
+ + Manages organizational Storage Spaces in + +
+
+ [WebDAV, libregraph, CS3, CLI] +
+
+
+
+
+ + + Manages organizational Storage... + + + + + + + +
+
+
+ + Identity Management System + +
+ [Software System] +
+
+
+ provides users and groups +
+
+
+
+ + + Identity Management System... + + + + + + + + +
+
+
+
+
+ + Authenticates users and searches recipients with + +
+
+ [OpenID Connect, LDAP, REST] +
+
+
+
+
+ + + Authenticates users and search... + + + + + + + +
+
+
+

+ C4 Container diagram for the oCIS System +

+

+ As a platform, the oCIS system may not only includes web, mobile and desktop clients but also the underlying storage system or an identity management system +

+

+ Date: 2021-07-22T16:43 +

+
+
+
+ + + C4 Container diagram for the oCIS System... + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/docs/extensions/storage/static/storageprovider.drawio.svg b/docs/extensions/storage/static/storageprovider.drawio.svg index 4b88a71c17..e7ba5ea770 100644 --- a/docs/extensions/storage/static/storageprovider.drawio.svg +++ b/docs/extensions/storage/static/storageprovider.drawio.svg @@ -1,119 +1,345 @@ - + - - + -
-
-
- - CS3 -
- storage provider -
- API (GRPC) -
- +
+
+
+ oCIS storage provider +
+ [Software System]
- - CS3... + + oCIS storage provider... - - - - - - - - - - + -
+
-
- storage provider +
+ + reva storage provider + +
+ [Component: golang] +
+
+
+ hosts multiple storage spaces using a storage driver +
- - storage provider + + reva storage provider... - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + -
+
-
- - / - +
+ + reva gateway + +
+ [Component: golang] +
+
+
+ API facade for internal reva services +
- - / + + reva gateway... + + + + + + + +
+
+
+ + Storage System + +
+ [Software System] +
+
+
+ provides persistent storage +
+
+
+
+ + + Storage System... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [POSIX, S3] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + +
+
+
+ + reva frontend + +
+ [Component: golang] +
+
+
+ handles protocol translation +
+
+
+
+ + + reva frontend... + + + + + + + +
+
+
+ + oCIS proxy + +
+ [Component: golang] +
+
+
+ Routes requests to oc10 or ecis +
+
+
+
+ + + oCIS proxy... + + + + + + + + +
+
+
+
+
+ + Mints an internal JWT +
+ and torwards requests to + +
+
+ [WebDAV, OCS, OCM, tus] +
+
+
+
+
+ + + Mints an internal JWT... + + + + + + + +
+
+
+ + Client + +
+ [Container: C++, Kotlin, +
+ Swift or Vue] +
+
+
+ A desktop, mobile or web Client +
+
+
+
+ + + Client... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [WebDAV, libregraph, CS3] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [CS3, tus] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + + +
+
+
+
+
+ + Forwards to + +
+
+ [CS3, storage registry] +
+
+
+
+
+ + + Forwards to... + + + + + + + +
+
+
+

+ C4 Component diagram for an oCIS storage provider +

+

+ An oCIS storage provider manages resources in storage spaces by persisting them with a specific storage driver in a storage system. +

+

+ Date: 2021-07-22T12:40 +

+
+
+
+ + + C4 Component diagram for an oCIS storage provider... - - - - - - - - diff --git a/docs/extensions/storage/static/storageregistry-spaces.drawio.svg b/docs/extensions/storage/static/storageregistry-spaces.drawio.svg deleted file mode 100644 index 3c2d49717f..0000000000 --- a/docs/extensions/storage/static/storageregistry-spaces.drawio.svg +++ /dev/null @@ -1,327 +0,0 @@ - - - - - - - - -
-
-
- The storage registry currently maps paths and storageids to the -
- - address:port - - of the corresponding storage provider -
-
-
- - - The storage registry currently maps... - - - - - - - - - - -
-
-
- storage registry -
-
-
- - - storage registry - - - - - - - - - - -
-
-
- storage providers -
-
-
- - - storage providers - - - - - - - - - - - - - - - -
-
-
- The gateway uses the storage registry to look up the storage provider that is responsible for path and id based references in incoming requests. -
-
-
- - - The gateway uses the storage regist... - - - - - - - - - - -
-
-
- gateway -
-
-
- - - gateway - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Viewer does not support full SVG 1.1 - - - - \ No newline at end of file diff --git a/docs/extensions/storage/storages.md b/docs/extensions/storage/storages.md index 6172dc10ef..59927fd933 100644 --- a/docs/extensions/storage/storages.md +++ b/docs/extensions/storage/storages.md @@ -78,7 +78,15 @@ The storage keeps an activity history, tracking the different actions that have ## Storage drivers -Reva currently has four storage driver implementations that can be used for *storage providers* an well as *data providers*. +Reva currently has several storage driver implementations that can be used for *storage providers* an well as *data providers*. + +### OCIS and S3NG Storage Driver + +The oCIS storage driver is the default storage driver. It decomposes the metadata and persists it in a POSIX filesystem. Blobs are stored on the filesystem as well. The layout makes extensive use of symlinks and extended attributes. A filesystem like xfs or zfs without inode size limitations is recommended. We will evolve this to further integrate with file systems like cephfs or gpfs. + +The S3NG storage driver uses the same metadata layout on a POSIX storage as the oCIS driver, but it uses S3 as the blob storage. + +TODO add list of capabilities / tradeoffs ### Local Storage Driver diff --git a/docs/extensions/storage/terminology.md b/docs/extensions/storage/terminology.md index 0b3ce00a4e..2b883b5705 100644 --- a/docs/extensions/storage/terminology.md +++ b/docs/extensions/storage/terminology.md @@ -9,66 +9,50 @@ geekdocFilePath: terminology.md Communication is hard. And clear communication is even harder. You may encounter the following terms throughout the documentation, in the code or when talking to other developers. Just keep in mind that whenever you hear or read *storage*, that term needs to be clarified, because on its own it is too vague. PR welcome. -## Resources -A *resource* is a logical concept. Resources can be of [different types](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceType): +## Logical concepts + +### Resources +A *resource* is the basic building block that oCIS manages. It can be of [different types](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceType): - an actual *file* - a *container*, e.g. a folder or bucket - a *symlink*, or - a [*reference*]({{< ref "#references" >}}) which can point to a resource in another [*storage provider*]({{< ref "#storage-providers" >}}) -## References +### References -A *reference* is a logical concept that identifies a [*resource*]({{< ref "#resources" >}}). A [*CS3 reference*](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.Reference) consists of either -- a *path* based reference, used to identify a [*resource*]({{< ref "#resources" >}}) in the [*namespace*]({{< ref "./namespaces.md" >}}) of a [*storage provider*]({{< ref "#storage-providers" >}}). It must start with a `/`. -- a [CS3 *id* based reference](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId), uniquely identifying a [*resource*]({{< ref "#resources" >}}) in the [*namespace*]({{< ref "./namespaces.md" >}}) of a [*storage provider*]({{< ref "#storage-providers" >}}). It consists of a `storage provider id` and an `opaque id`. The `storage provider id` must NOT start with a `/`. - -{{< hint info >}} -The `/` is important because currently the static [*storage registry*]({{< ref "#storage-space-registries" >}}) uses a map to look up which [*storage provider*]({{< ref "#storage-providers" >}}) is responsible for the resource. Paths must be prefixed with `/` so there can be no collisions between paths and storage provider ids in the same map. -{{< /hint >}} +A *reference* identifies a [*resource*]({{< ref "#resources" >}}). A [*CS3 reference*](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.Reference) can carry a *path* and a [CS3 *resource id*](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId). The references come in two flavors: absolute and combined. +Absolute references have either the *path* or the *resource id* set: +- An absolute *path* MUST start with a `/`. The *resource id* MUST be empty. +- An absolute *resource id* uniquely identifies a [*resource*]({{< ref "#resources" >}}) and is used as a stable identifier for sharing. The *path* MUST be empty. +Combined references have both, *path* and *resource id* set: +- the *resource id* identifies the root [*resource*]({{< ref "#resources" >}}) +- the *path* is relative to that root. It MUST start with `.` -{{< hint warning >}} -### Alternative: reference triple #### -A *reference* is a logical concept. It identifies a [*resource*]({{< ref "#resources" >}}) and consists of -a `storage_space`, a `` and a `` -``` -!: -``` -While all components are optional, only three cases are used: -| format | example | description | -|-|-|-| -| `!:` | `!:/absolute/path/to/file.ext` | absolute path | -| `!:` | `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!:path/to/file.ext` | path relative to the root of the storage space | -| `!:` | `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!c3cf23bb-8f47-4719-a150-1d25a1f6fb56:to/file.ext` | path relative to the specified node in the storage space, used to reference resources without disclosing parent paths | +### Storage Spaces +A *storage space* organizes a set of [*resources*]({{< ref "#resources" >}}) in a hierarchical tree. It has a single *owner* (*user* or *group*), +a *quota*, *permissions* and is identified by a `storage space id`. -`` should be a UUID to prevent references from breaking when a *user* or [*storage space*]({{< ref "#storage-spaces" >}}) gets renamed. But it can also be derived from a migration of an oc10 instance by concatenating an instance identifier and the numeric storage id from oc10, e.g. `oc10-instance-a$1234`. +{{< svg src="extensions/storage/static/storagespace.drawio.svg" >}} -A reference will often start as an absolute/global path, e.g. `!:/home/Projects/Foo`. The gateway will look up the storage provider that is responsible for the path +Examples would be every user's personal storage space, project storage spaces or group storage spaces. While they all serve different purposes and may or may not have workflows like anti virus scanning enabled, we need a way to identify and manage these subtrees in a generic way. By creating a dedicated concept for them this becomes easier and literally makes the codebase cleaner. A [*storage space registry*]({{< ref "#storage-space-registries" >}}) then allows listing the capabilities of [*storage spaces*]({{< ref "#storage-spaces" >}}), e.g. free space, quota, owner, syncable, root etag, upload workflow steps, ... -| Name | Description | Who resolves it? | -|------|-------------|-| -| `!:/home/Projects/Foo` | the absolute path a client like davfs will use. | The gateway uses the storage registry to look up the responsible storage provider | -| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!:/Projects/Foo` | the `storage_space` is the same as the `root`, the path becomes relative to the root | the storage provider can use this reference to identify this resource | +Finally, a logical `storage space id` is not tied to a specific [*storage provider*]({{< ref "#storage-providers" >}}). If the [*storage driver*]({{< ref "#storage-drivers" >}}) supports it, we can import existing files including their `file id`, which makes it possible to move [*storage spaces*]({{< ref "#storage-spaces" >}}) between [*storage providers*]({{< ref "#storage-providers" >}}) to implement storage classes, e.g. with or without archival, workflows, on SSDs or HDDs. -Now, the same file is accessed as a share -| Name | Description | -|------|-------------| -| `!:/users/Einstein/Projects/Foo` | `Foo` is the shared folder | -| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:` | `56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a` is the id of `Foo`, the path is empty | +### Shares +*To be clarified: we are aware that [*storage spaces*]({{< ref "#storage-spaces" >}}) may be too 'heavywheight' for ad hoc sharing with groups. That being said, there is no technical reason why group shares should not be treated like [*storage spaces*]({{< ref "#storage-spaces" >}}) that users can provision themselves. They would share the quota with the users home [*storage space*]({{< ref "#storage-spaces" >}}) and the share initiator would be the sole owner. Technically, the mechanism of treating a share like a new [*storage space*]({{< ref "#storage-spaces" >}}) would be the same. This obviously also extends to user shares and even file indvidual shares that would be wrapped in a virtual collection. It would also become possible to share collections of arbitrary files in a single storage space, e.g. the ten best pictures from a large album.* -The `:`, `!` and `$` are chosen from the set of [RFC3986 sub delimiters](https://tools.ietf.org/html/rfc3986#section-2.2) on purpose. They can be used in URLs without having to be encoded. In some cases, a delimiter can be left out if a component is not set: -| reference | interpretation | -|-|-| -| `/absolute/path/to/file.ext` | absolute path, all delimiters omitted | -| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!path/to/file.ext` | relative path in the given storage space, root delimiter `:` omitted | -| `56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:to/file.ext` | relative path in the given root node, storage space delimiter `!` omitted | -| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:` | node id in the given storage space, `:` must be present | -| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62` | root of the storage space, all delimiters omitted, can be distinguished by the `/` | +### Storage Space Registries -{{< /hint >}} +A *storage space registry* manages the [*namespace*]({{< ref "./namespaces.md" >}}) for a *user*: it is used by *clients* to look up storage spaces a user has access to, the `/dav/spaces` endpoint to access it via WabDAV, and where the client should mount it in the users personal namespace. -## Storage Drivers +{{< svg src="extensions/storage/static/spacesregistry.drawio.svg" >}} + + +## Technical concepts + +### Storage Drivers A *storage driver* implements access to a [*storage system*]({{< ref "#storage-systems" >}}): @@ -77,38 +61,14 @@ It maps the *path* and *id* based CS3 *references* to an appropriate [*storage s - posix inodes or paths - deconstructed filesystem nodes -{{< hint warning >}} -**Proposed Change** -iOS clients can only queue single requests to be executed in the background. The queue an upload and need to be able to identify the uploaded file after it has been uploaded to the server. The disconnected nature of the connection might cause worksflows or manual user interaction with the file on the server to move the file to a different place or changing the content while the device is offline. However, on the device users might have marked the file as favorite or added it to other iOS specific collections. To be able to reliably identify the file the client can generate a `uuid` and attach it to the file metadata during the upload. While it is not necessary to look up files by this `uuid` having a second file id that serves exactly the same purpose as the `file id` is redundant. - -Another aspect for the `file id` / `uuid` is that it must be a logical identifier that can be set, at least by internal systems. Without a writeable fileid we cannot restore backups or migrate storage spaces from one storage provider to another storage provider. - -Technically, this means that every storage driler needs to have a map of a `uuid` to in internal resource identifier. This internal resource identifier can be -- an eos fileid, because eos can look up files by id -- an inode if the filesystem and the storage driver support lookung up by inode -- a path if the storage driver has no way of looking up files by id. - - In this case other mechanisms like inotify, kernel audit or a fuse overlay might be used to keep the paths up to date. - - to prevent excessive writes when deep folders are renamed a reverse map might be used: it will map the `uuid` to `:`, allowing to trade writes for reads - -{{< /hint >}} -## Storage Providers +### Storage Providers A *storage provider* manages [*resources*]({{< ref "#resources" >}}) identified by a [*reference*]({{< ref "#references" >}}) by accessing a [*storage system*]({{< ref "#storage-systems" >}}) with a [*storage driver*]({{< ref "#storage-drivers" >}}). {{< svg src="extensions/storage/static/storageprovider.drawio.svg" >}} -{{< hint warning >}} -**Proposed Change** -A *storage provider* manages multiple [*storage spaces*]({{< ref "#storage-space" >}}) -by accessing a [*storage system*]({{< ref "#storage-systems" >}}) with a [*storage driver*]({{< ref "#storage-drivers" >}}). - -{{< svg src="extensions/storage/static/storageprovider-spaces.drawio.svg" >}} - -By making [*storage providers*]({{< ref "#storage-providers" >}}) aware of [*storage spaces*]({{< ref "#storage-spaces" >}}) we can get rid of the current `enablehome` flag / hack in reva, which lead to the [spawn of `*home` drivers](https://github.com/cs3org/reva/tree/master/pkg/storage/fs). Furthermore, provisioning a new [*storage space*]({{< ref "#storage-space" >}}) becomes a generic operation, regardless of the need of provisioning a new user home or a new project space. -{{< /hint >}} - -## Storage Space Registries +### Storage Registry A *storage registry* manages the [*CS3 global namespace*]({{< ref "./namespaces.md#cs3-global-namespaces" >}}): It is used by the *gateway* @@ -117,65 +77,11 @@ that should handle a [*reference*]({{< ref "#references" >}}). {{< svg src="extensions/storage/static/storageregistry.drawio.svg" >}} -{{< hint warning >}} -**Proposed Change** -A *storage space registry* manages the [*namespace*]({{< ref "./namespaces.md" >}}) for a *user*: -It is used by the *gateway* -to look up `address` and `port` of the [*storage provider*]({{< ref "#storage-providers" >}}) -that is currently serving a [*storage space*]({{< ref "#storage-space" >}}). - -{{< svg src="extensions/storage/static/storageregistry-spaces.drawio.svg" >}} - -By making *storage registries* aware of [*storage spaces*]({{< ref "#storage-spaces" >}}) we can query them for a listing of all [*storage spaces*]({{< ref "#storage-spaces" >}}) a user has access to. Including his home, received shares, project folders or group drives. See [a WIP PR for spaces in the oCIS repo (#1827)](https://github.com/owncloud/ocis/pull/1827) for more info. -{{< /hint >}} - -## Storage Spaces -A *storage space* is a logical concept: -It is a tree of [*resources*]({{< ref "#resources" >}})*resources* -with a single *owner* (*user* or *group*), -a *quota* and *permissions*, identified by a `storage space id`. - -{{< svg src="extensions/storage/static/storagespace.drawio.svg" >}} - -Examples would be every user's home storage space, project storage spaces or group storage spaces. While they all serve different purposes and may or may not have workflows like anti virus scanning enabled, we need a way to identify and manage these subtrees in a generic way. By creating a dedicated concept for them this becomes easier and literally makes the codebase cleaner. A [*storage space registry*]({{< ref "#storage-space-registries" >}}) then allows listing the capabilities of [*storage spaces*]({{< ref "#storage-spaces" >}}), e.g. free space, quota, owner, syncable, root etag, upload workflow steps, ... - -Finally, a logical `storage space id` is not tied to a specific [*storage provider*]({{< ref "#storage-providers" >}}). If the [*storage driver*]({{< ref "#storage-drivers" >}}) supports it, we can import existing files including their `file id`, which makes it possible to move [*storage spaces*]({{< ref "#storage-spaces" >}}) between [*storage providers*]({{< ref "#storage-providers" >}}) to implement storage classes, e.g. with or without archival, workflows, on SSDs or HDDs. - -## Shares -*To be clarified: we are aware that [*storage spaces*]({{< ref "#storage-spaces" >}}) may be too 'heavywheight' for ad hoc sharing with groups. That being said, there is no technical reason why group shares should not be treated like [*storage spaces*]({{< ref "#storage-spaces" >}}) that users can provision themselves. They would share the quota with the users home [*storage space*]({{< ref "#storage-spaces" >}}) and the share initiator would be the sole owner. Technically, the mechanism of treating a share like a new [*storage space*]({{< ref "#storage-spaces" >}}) would be the same. This obviously also extends to user shares and even file indvidual shares that would be wrapped in a virtual collection. It would also become possible to share collections of arbitrary files in a single storage space, e.g. the ten best pictures from a large album.* - - -## Storage Systems +### Storage Systems Every *storage system* has different native capabilities like id and path based lookups, recursive change time propagation, permissions, trash, versions, archival and more. A [*storage provider*]({{< ref "#storage-providers" >}}) makes the storage system available in the CS3 API by wrapping the capabilities as good as possible using a [*storage driver*]({{< ref "#storage-drivers" >}}). There migt be multiple [*storage drivers*]({{< ref "#storage-drivers" >}}) for a *storage system*, implementing different tradeoffs to match varying requirements. -## Gateways +### Gateways A *gateway* acts as a facade to the storage related services. It authenticates and forwards API calls that are publicly accessible. - -{{< hint warning >}} -**Proposed Change** -Currently, the *gateway* treats `/home/shares` different than any other path: it will stat all children and calculate an etag to allow clients to discover changes in accepted shares. This requires the storage provider to cooperate and provide this special `/shares` folder in the root of a users home when it is accessed as a home storage, which is a config flag that needs to be set for every storage driver. - -The `enable_home` flag will cause drivers to jail path based requests into a `` subfolder. In effect it divides a storage provider into multiple [*storage spaces*]({{< ref "#storage-spaces" >}}): when calling `CreateHome` a subfolder following the `` is created and market as the root of a users home. Both, the eos and ocis storage drivers use extended attributes to mark the folder as the end of the size aggregation and tree mtime propagation mechanism. Even setting the quota is possible like that. All this literally is a [*storage space*]({{< ref "#storage-spaces" >}}). - -We can implement [ListStorageSpaces](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ListStorageSpacesRequest) by either -- iterating over the root of the storage and treating every folder following the `` as a `home` *storage space*, -- iterating over the root of the storage and treating every folder following a new `` as a `project` *storage space*, or -- iterating over the root of the storage and treating every folder following a generic `` as a *storage space* for a configurable space type, or -- we allow configuring a map of `space type` to `layout` (based on the [CreateStorageSpaceRequest](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.CreateStorageSpaceRequest)) which would allow things like -``` -home=/var/lib/ocis/storage/home/{{substr 0 1 .Owner.Username}}/{{.Owner.Username}} -spaces=/spaces/var/lib/ocis/storage/projects/{{.Name}} -``` - -This would make the `GetHome()` call return the path to the *storage provider* including the relative path to the *storage space*. No need for a *storage provider* mounted at `/home`. This is just a UI alias for `/users/`. Just like a normal `/home/` on a linux machine. - -But if we have no `/home` where do we find the shares, and how can clients discover changes in accepted shares? - -The `/shares` namespace should be provided by a *storage provider* that lists all accepted shares for the current user... but what about copy pasting links from the browser? Well this storage is only really needed to have a path to ocm shares that actually reside on other instances. In the UI the shares would be listed by querying a *share manager*. It returns ResourceIds, which can be stated to fetch a path that is then accessible in the CS3 global namespace. Two caveats: -- This only works for resources that are actually hosted by the current instance. For those it would leak the parent path segments to a shared resource. -- For accepted OCM shares there must be a path in the [*CS3 global namespace*]({{< ref "./namespaces.md#cs3-global-namespaces" >}}) that has to be the same for all users, otherwise they cannot copy and share those URLs. - -{{< /hint >}} \ No newline at end of file diff --git a/docs/extensions/storage/updating.md b/docs/extensions/storage/updating.md index 2a5133618c..dd19203766 100644 --- a/docs/extensions/storage/updating.md +++ b/docs/extensions/storage/updating.md @@ -11,9 +11,9 @@ geekdocFilePath: updating.md ## Updating reva -1. Run `go get github.com/cs3org/reva@master` +1. Run `go get github.com/cs3org/reva@master` in all repos that depend on reva 2. Create a changelog entry containing changes that were done in [reva](https://github.com/cs3org/reva/commits/master) -3. Create a Pull Request to ocis-reva master with those changes +3. Create a Pull Request to ocis master with those changes 4. If test issues appear, you might need to adjust the tests 5. After the PR is merged, consider doing a [release of the storage submodule]({{< ref "releasing" >}}) diff --git a/docs/ocis/_index.md b/docs/ocis/_index.md index ed233878b1..46f0b599a5 100644 --- a/docs/ocis/_index.md +++ b/docs/ocis/_index.md @@ -10,18 +10,16 @@ geekdocFilePath: _index.md {{< figure class="floatright" src="/media/is.png" width="70%" height="auto" >}} ## ownCloud Infinite Scale - Welcome to oCIS, the modern file-sync and share platform, which is based on our knowledge and experience with the PHP based [ownCloud server](https://owncloud.com/#server). ### The idea of federated storage - To creata a truly federated storage architecture oCIS breaks down the old ownCloud 10 user specific namespace, which is assembled on the server side, and makes the individual parts accessible to clients as storage spaces and storage space registries. The below diagram shows the core conceps that are the foundation for the new architecture: - End user devices can fetch the list of *storage spaces* a user has access to, by querying one or multiple *storage space registries*. The list contains a unique endpoint for every *storage space*. -- [*Storage space registries*]({{< ref "../extensions/storage/terminology#storage-space-registries" >}}) manage the list of storage spaces a user has access to. They may subscrible to *storage spaces* in order to receive notifications about changes on behalf of an end users mobile or desktop client. -- [*Storage spaces*]({{< ref "../extensions/storage/terminology#storage-spaces" >}}) represent a collection of files and folders. A users personal files are a *storage space*, a group or project drive is a *storage space*, and even incoming shares are treated and implemented as *storage spaces*. Each with properties like owners, permissions, quota and type. -- [*Storage providers*]({{< ref "../extensions/storage/terminology#storage-providers" >}}) can hold multiple *storage spaces*. At an oCIS instance, there might be a dedicated *storage provider* responsible for users personal storage spaces. There might be multiple, sharing the load or there might be just one, hosting all types of *storage spaces*. +- [*Storage space registries*]({{< ref "../extensions/storage/terminology#storage-space-registries" >}}) manage the list of storage spaces a user has access to. They may subscribe to *storage spaces* in order to receive notifications about changes on behalf of an end users mobile or desktop client. +- [*Storage spaces*]({{< ref "../extensions/storage/terminology#storage-spaces" >}}) represent a collection of files and folders. A users personal files are contained in a *storage space*, a group or project drive is a *storage space*, and even incoming shares are treated and implemented as *storage spaces*. Each with properties like owners, permissions, quota and type. +- [*Storage providers*]({{< ref "../extensions/storage/terminology#storage-providers" >}}) can hold multiple *storage spaces*. At an oCIS instance, there might be a dedicated *storage provider* responsible for users personal storage spaces. There might be multiple, either to shard the load, provide different levels of redundancy or support custom workflows. Or there might be just one, hosting all types of *storage spaces*. {{< svg src="ocis/static/idea.drawio.svg" >}} @@ -35,19 +33,18 @@ Einstein copies the URL in the browser (or an email with the same URL is sent au When Marie enters that URL she will be presented with a login form on the `https://cloud.zurich.test` instance, because the share was created on that domain. If `https://cloud.zurich.test` trusts her OpenID Connect identity provider `https://idp.paris.test` she can log in. This time, the *storage space registry* discovery will come up with `https://cloud.paris.test` though. Since that registry is different than the registry tied to `https://cloud.zurich.test` oCIS web can look up the *storage space* `716199a6-00c0-4fec-93d2-7e00150b1c84` and register the WebDAV URL `https://cloud.zurich.test/dav/spaces/716199a6-00c0-4fec-93d2-7e00150b1c84/a/rel/path` in Maries *storage space registry* at `https://cloud.paris.test`. When she accepts that share her clients will be able to sync the new *storage space* at `https://cloud.zurich.test`. -### oCIS microservice runtime +Or in other words: _total world federation!_ +### oCIS microservice runtime The oCIS runtime allows us to dynamically manage services running in a single process. We use [suture](https://github.com/thejerf/suture) to create a supervisor tree that starts each service in a dedicated goroutine. By default oCIS will start all built-in oCIS extensions in a single process. Individual services can be moved to other nodes to scale-out and meet specific performance requirements. A [go-micro](https://github.com/asim/go-micro/blob/master/registry/registry.go) based registry allows services in multiple nodes to form a distributed microservice architecture. ### oCIS extensions - Every oCIS extension uses [ocis-pkg](https://github.com/owncloud/ocis/tree/master/ocis-pkg), which implements the [go-micro](https://go-micro.dev/) interfaces for [servers](https://github.com/asim/go-micro/blob/v3.5.0/server/server.go#L17-L37) to register and [clients](https://github.com/asim/go-micro/blob/v3.5.0/client/client.go#L11-L23) to lookup nodes with a service [registry](https://github.com/asim/go-micro/blob/v3.5.0/registry/registry.go). We are following the [12 Factor](https://12factor.net/) methodology with oCIS. The uniformity of services also allows us to use the same command, logging and configuration mechanism. Configurations are forwarded from the oCIS runtime to the individual extensions. ### go-micro - While the [go-micro](https://go-micro.dev/) framework provides abstractions as well as implementations for the different components in a microservice architecture, it uses a more developer focused runtime philosophy: It is used to download services from a repo, compile them on the fly and start them as individual processes. For oCIS we decided to use a more admin friendly runtime: You can download a single binary and start the contained oCIS extensions with a single `bin/ocis server`. This also makes packaging easier. We use [ocis-pkg](https://github.com/owncloud/ocis/tree/master/ocis-pkg) to configure the default implementations for the go-micro [grpc server](https://github.com/asim/go-micro/tree/v3.5.0/plugins/server/grpc), [client](https://github.com/asim/go-micro/tree/v3.5.0/plugins/client/grpc) and [mdns registry](https://github.com/asim/go-micro/blob/v3.5.0/registry/mdns_registry.go), swapping them out as needed, eg. to use the [kubernetes registry plugin](https://github.com/asim/go-micro/tree/v3.5.0/plugins/registry/kubernetes). @@ -62,7 +59,6 @@ Interacting with oCIS involves a multitude af APIs. The server and all clients r We run a huge [test suite](https://github.com/owncloud/core/tree/master/tests), which originated in ownCloud 10 and continues to grow. A detailed description can be found in the developer docs for [testing]({{< ref "development/testing" >}}). ### Architecture Overview - Running `bin/ocis server` will start the below services, all of which can be scaled and deployed on a single node or in a cloud native environment, as needed. {{< svg src="ocis/static/architecture-overview.drawio.svg" >}} diff --git a/docs/ocis/migration.md b/docs/ocis/migration.md index 12583248ac..8fd1ecd414 100644 --- a/docs/ocis/migration.md +++ b/docs/ocis/migration.md @@ -9,14 +9,6 @@ geekdocFilePath: migration.md The migration happens in subsequent stages while the service is online. First all users need to migrate to the new architecture, then the global namespace needs to be introduced. Finally, the data on disk can be migrated user by user by switching the storage driver. -
- -{{< hint warning >}} -@jfd: It might be easier to introduce the spaces api in oc10 and then migrate to oCIS. We cannot migrate both at the same time, the architecture to oCIS (which will change fileids) and introduce a global namespace (which requires stable fileids to let clients handle moves without redownloading). Either we implement arbitrary mounting of shares in oCIS / reva or we make clients and oc10 spaces aware. -{{< /hint >}} - -
- ## Migration Stages ### Stage 0: pre migration @@ -56,7 +48,7 @@ The ownCloud 10 demo instance uses OAuth to obtain a token for ownCloud web and
-_TODO make oauth2 in oc10 trust the new web ui, based on `redirect_uri` and CSRF so no explicit consent is needed_ +_TODO make oauth2 in oc10 trust the new web ui, based on `redirect_uri` and CSRF so no explicit consent is needed?_ #### FAQ _Feel free to add your question as a PR to this document using the link at the top of this page!_ @@ -72,13 +64,12 @@ While SAML and Shibboleth are protocols that solve that problem, they are limite
-_TODO @butonic add ADR for OpenID Connect_ +_TODO @butonic add ADR for OpenID Connect and flesh out pros and cons of the above_
#### User impact -When introducing OpenID Connect, the clients will detect the new authentication scheme when their current way of authenticating returns an error. Users will then have to -reauthorize at the OpenID Connecd IdP, which again, may be configured to skip the consent step for trusted clients. +When introducing OpenID Connect, the clients will detect the new authentication scheme when their current way of authenticating returns an error. Users will then have to reauthorize at the OpenID Connect IdP, which again, may be configured to skip the consent step for trusted clients. #### Steps 1. There are multiple products that can be used as an OpenID Connect IdP. We test with [LibreGraph Connect](https://github.com/libregraph/lico), which is also [embedded in oCIS](https://github.com/owncloud/web/). Other alternatives include [Keycloak](https://www.keycloak.org/) or [Ping](https://www.pingidentity.com/). Please refer to the corresponding setup instructions for the product you intent to use. @@ -106,7 +97,7 @@ Should there be problems with OpenID Connect at this point you can disable the a
Legacy clients relying on Basic auth or app passwords need to be migrated to OpenId Connect to work with oCIS. For a transition period Basic auth in oCIS can be enabled with `PROXY_ENABLE_BASIC_AUTH=true`, but we strongly recommend adopting OpenID Connect for other tools as well. -While OpenID Connect providers will send an `iss` and `sub` claim that relying parties (services like oCIS or ownCloud 10) can use to identify users we recommend introducing a dedicated, globally unique, persistent, non-reassignable user identifier like a UUID for every user. This `ownclouduuid` shold be sent as an additional claim to save additional lookups on the server side. It will become the user id in oCIS, e.g. when searching for recipients the `ownclouduuid` will be used to persist permissions with the share manager. It has a different purpose than the ownCloud 10 username, which is used to login. Using UUIDs we can not only mitigate username collisions when merging multiple instances but also allow renaming usernames after the migration to oCIS has been completed. +While OpenID Connect providers will send an `iss` and `sub` claim that relying parties (services like oCIS or ownCloud 10) can use to identify users we recommend introducing a dedicated, globally unique, persistent, non-reassignable user identifier like a UUID for every user. This `ownclouduuid` should be sent as an additional claim to save additional lookups on the server side. It will become the user id in oCIS, e.g. when searching for recipients the `ownclouduuid` will be used to persist permissions with the share manager. It has a different purpose than the ownCloud 10 username, which is used to login. Using UUIDs we can not only mitigate username collisions when merging multiple instances but also allow renaming usernames after the migration to oCIS has been completed.
@@ -117,9 +108,9 @@ _Feel free to add your question as a PR to this document using the link at the t
-### Stage 3: introduce oCIS interally +### Stage 3: introduce oCIS internally -Befor letting oCIS handle end user requests we will first make it available in the internal network. By subsequently adding services we can add functionality and verify the services work as intended. +Before letting oCIS handle end user requests we will first make it available in the internal network. By subsequently adding services we can add functionality and verify the services work as intended. Start oCIS backend and make read only tests on existing data using the `owncloudsql` storage driver which will read (and write) - blobs from the same datadirectory layout as in ownCloud 10 @@ -139,7 +130,7 @@ None, only administrators will be able to explore oCIS during this stage. #### Steps and verifications -We are going to run and explore a series of services that will together handle the same requests as ownCloud 10. For initial exploration the oCIS binary is recommended. The services can later be deployed using a single oCIS runtime or in multiple cotainers. +We are going to run and explore a series of services that will together handle the same requests as ownCloud 10. For initial exploration the oCIS binary is recommended. The services can later be deployed using a single oCIS runtime or in multiple containers. ##### Storage provider for file metadata @@ -172,7 +163,7 @@ Enable spaces API in oc10: {{< hint warning >}} **Alternative 2** -An additional `uuid` property used only to detect moves. A lookup by uuid is not necessary for this. The `/dav/meta` endpoint would still take the fileid. Clients would use the `uuid` to detect moves and set up new sync pairs when migrating to a global namespace. +An additional `uuid` property used only to detect moves. A lookup by uuid is not necessary for this. The `/dav/meta` endpoint would still take the fileid. Clients would use the `uuid` to detect moves and set up new sync pairs when migrating to a global namespace. ### Stage-3.1 Generate a `uuid` for every file as a file property. Clients can submit a `uuid` when creating files. The server will create a `uuid` if the client did not provide one. @@ -280,7 +271,7 @@ The IP address of the ownCloud host changes. There is no change for the file syn 2. Verify the requests are routed based on the ownCloud 10 routing policy `oc10` by default ##### Test user based routing -1. Change the routing policy for a user or an early adoptors group to `ocis`
_TODO @butonic currently, the migration selector will use the `ocis` policy for users that have been added to the accounts service. IMO we need to evaluate a claim from the IdP._
+1. Change the routing policy for a user or an early adopters group to `ocis`
_TODO @butonic currently, the migration selector will use the `ocis` policy for users that have been added to the accounts service. IMO we need to evaluate a claim from the IdP._
2. Verify the requests are routed based on the oCIS routing policy `oc10` for 'migrated' users. At this point you are ready to rock & roll! @@ -322,8 +313,8 @@ _TODO @butonic update performance comparisons nightly_ #### Steps There are several options to move users to the oCIS backend: -- Use a canary app to let users decide thamselves -- Use an early adoptors group with an opt in +- Use a canary app to let users decide themselves +- Use an early adopters group with an opt in - Force migrate users in batch or one by one at the administrators will #### Verification @@ -333,7 +324,7 @@ The same verification steps as for the internal testing stage apply. Just from t Until now, the oCIS configuration mimics ownCloud 10 and uses the old data directory layout and the ownCloud 10 database. Users can seamlessly be switched from ownCloud 10 to oCIS and back again.
-_TODO @butonic we need a canary app that allows users to decide for themself which backend to use_ +_TODO @butonic we need a canary app that allows users to decide for themselves which backend to use_
@@ -401,12 +392,12 @@ Noticeable performance improvements because we effectively shard the storage log _TODO @butonic implement `ownclouds3` based on `s3ng`_ _TODO @butonic implement tiered storage provider for seamless migration_ -_TODO @butonic document how to manually do that until the storge registry can discover that on its own._ +_TODO @butonic document how to manually do that until the storage registry can discover that on its own._
#### Verification -Start with a test user, then move to early adoptors and finally migrate all users. +Start with a test user, then move to early adopters and finally migrate all users. #### Rollback To switch the storage provider again the same storage space migration can be performed again: copy medatata and blob data using the CS3 api, then change the responsible storage provider in the storage registry. @@ -432,7 +423,7 @@ Migrate share data to _yet to determine_ share manager backend and shut down own The ownCloud 10 database still holds share information in the `oc_share` and `oc_share_external` tables. They are used to efficiently answer queries about who shared what with whom. In oCIS shares are persisted using a share manager and if desired these grants are also sent to the storage provider so it can set ACLs if possible. Only one system should be responsible for the shares, which in case of treating the storage as the primary source effectively turns the share manager into a cache. #### User impact -Depending on chosen the share manager provider some sharing requests should be faster: listing incoming and outgoing shares is no longer bound to the ownCloud 10 database but to whatever technology is used by the share provdier: +Depending on chosen the share manager provider some sharing requests should be faster: listing incoming and outgoing shares is no longer bound to the ownCloud 10 database but to whatever technology is used by the share provider: - For non HA scenarios they can be served from memory, backed by a simple json file. - TODO: implement share manager with redis / nats / ... key value store backend: use the micro store interface please ... @@ -446,13 +437,13 @@ Depending on chosen the share manager provider some sharing requests should be f _TODO for HA implement share manager with redis / nats / ... key value store backend: use the micro store interface please ..._ _TODO for batch migration implement share data migration cli with progress that reads all shares via the cs3 api from one provider and writes them into another provider_ -_TODO for seamless migration implement tiered/chained share provider that reads share data from the old provider and writes newc shares to the new one_ +_TODO for seamless migration implement tiered/chained share provider that reads share data from the old provider and writes new shares to the new one_ _TODO for storage provider as source of truth persist ALL share data in the storage provider. Currently, part is stored in the share manager, part is in the storage provider. We can keep both, but the the share manager should directly persist its metadata to the storage system used by the storage provider so metadata is kept in sync_
#### Verification -After copying all metadata start a dedicated gateway and change the configuration to use the new share manager. Route a test user, a test group and early adoptors to the new gateway. When no problems occur you can stirt the desired number of share managers and roll out the change to all gateways. +After copying all metadata start a dedicated gateway and change the configuration to use the new share manager. Route a test user, a test group and early adopters to the new gateway. When no problems occur you can start the desired number of share managers and roll out the change to all gateways.
@@ -461,12 +452,12 @@ _TODO let the gateway write updates to multiple share managers ... or rely on th
#### Rollback -To switch the share manager to the database one revert routing users to the new share manager. If you already shut down the old share manager start it again. Use the tiered/chained share manager provider in reverse configuration (new share provider as read only, old as write) and migrate the shares again. You can alse restore a database backup if needed. +To switch the share manager to the database one revert routing users to the new share manager. If you already shut down the old share manager start it again. Use the tiered/chained share manager provider in reverse configuration (new share provider as read only, old as write) and migrate the shares again. You can also restore a database backup if needed.
### Stage-10 -Profit! Well, on the one hand you do not need to maintain a clustered database setup and can rely on the storage system. On the other hand you are now in microservice wonderland and will have to relearn how to identify bottlenecks and scale oCIS accordingly. The good thing is that tools like jaeger and prometheus have evolved and will help you understand what is going on. But this is a different Topic. See you on the other side! +Profit! Well, on the one hand you do not need to maintain a clustered database setup and can rely on the storage system. On the other hand you are now in micro service wonderland and will have to relearn how to identify bottlenecks and scale oCIS accordingly. The good thing is that tools like jaeger and prometheus have evolved and will help you understand what is going on. But this is a different Topic. See you on the other side! #### FAQ _Feel free to add your question as a PR to this document using the link at the top of this page!_ @@ -709,7 +700,7 @@ _TODO clarify if metadata from ldap & user_shibboleth needs to be migrated_
-The `dn` -> *owncloud internal username* mapping that currently lives in the `oc_ldap_user_mapping` table needs to move into a dedicated ownclouduuid attribute in the LDAP server. The idp should send it as a claim so the proxy does not have to look up the user using LDAP again. The username cannot be changed in ownCloud 10 and the oCIS provisioning API will not allow changing it as well. When we introduce the graph api we may allow changing usernames when all clients have moved to that api. +The `dn` -> *owncloud internal username* mapping that currently lives in the `oc_ldap_user_mapping` table needs to move into a dedicated `ownclouduuid` attribute in the LDAP server. The idp should send it as a claim so the proxy does not have to look up the user using LDAP again. The username cannot be changed in ownCloud 10 and the oCIS provisioning API will not allow changing it as well. When we introduce the graph api we may allow changing usernames when all clients have moved to that api. The problem is that the username in owncloud 10 and in oCIS also need to be the same, which might not be the case when the ldap mapping used a different column. In that case we should add another owncloudusername attribute to the ldap server. From fea6728e6b2a2a42b311e706e3a06c5c3cd9429b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rn=20Friedrich=20Dreyer?= Date: Thu, 22 Jul 2021 19:28:46 +0000 Subject: [PATCH 06/56] add space id vs resource id vs storage id section MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Jörn Friedrich Dreyer --- docs/extensions/storage/proposedchanges.md | 73 +++++++++++++++++++++- 1 file changed, 72 insertions(+), 1 deletion(-) diff --git a/docs/extensions/storage/proposedchanges.md b/docs/extensions/storage/proposedchanges.md index e8c1a14cec..29fe8c9d40 100644 --- a/docs/extensions/storage/proposedchanges.md +++ b/docs/extensions/storage/proposedchanges.md @@ -111,4 +111,75 @@ Technically, this means that every storage driver needs to have a map of a `uuid - to prevent excessive writes when deep folders are renamed a reverse map might be used: it will map the `uuid` to `:`, in order to trade writes for reads - as a fallback a sync job can read the file id from the metadata of the resources and populate the uuid to internal id map. -The TUS upload can take metadata, for PUT we might need a header. \ No newline at end of file +The TUS upload can take metadata, for PUT we might need a header. + +### Space id vs resource id vs storage id + +We have `/dav/meta/` where the `fileid` is a string that was returned by a PROPFIND or by the `/graph/v1.0/me/drives/` endpoint? That returns a space id and the root drive item which has an `id` + +Does that `id` have a specific format? We currently concatenate as `!`. + +A request against `/dav/meta/fileid` will use the reva storage registry to look up a path. + +What if the storage space is moved to another storage provider. This happens during a migration: + +1. the current oc10 fileids need to be prefixed with at least the numeric storage id to shard them. + +`123` becomes `instanceprefix$345!123` if we use a custom prefix that identifies an instance (so we can merge multiple instances into one ocis instance) and append the numeric storageid `345`. The pattern is `$!`. + +Every `$` identifies a space. + +- [ ] the owncloudsql driver can return these spaceids when listing spaces. + +Why does it not work if we just use the fileid of the root node in the db? + +Say we have a space with three resources: +`$!` +`instanceprefix$345!1` +`instanceprefix$345!2` +`instanceprefix$345!3` + +All users have moved to ocis and the registry contains a regex to route all `instanceprefix.*` references to the storageprovider with the owncloudsql driver. It is up to the driver to locate the correct resource by using the filecache table. In this case the numeric storage id is unnecessary. + +Now we migrate the space `345` to another storage driver: +- the storage registry contains a new entry for `instanceprefix$345` to send all resource ids for that space to the new storage provider +- the new storage driver has to take into account the full storageid because the nodeid may only be unique per storage space. + +If we now have to fetch the path on the `/dav/meta/` endpoint: +`/dav/meta/instanceprefix$345!1` +`/dav/meta/instanceprefix$345!2` +`/dav/meta/instanceprefix$345!3` + +This would work because the registry always sees `instanceprefix$345` as the storageid. + +Now if we use the fileids directly and leave out the numeric storageid: +`!` +`instanceprefix!1` +`instanceprefix!2` +`instanceprefix!3` + +This is the current `!` format. + +The reva storage registry contains a `instanceid` entry pointing to the storage provider with the owncloudsql driver. + +Resources can be looked up because the oc_filecache has a unique fileid over all storages. + +Now we again migrate the space `345` to another storage driver: +- the storage registry contains a new entry for `instanceprefix!1` so the storage space root now points to the new storage provider +- The registry needs to be aware of node ids to route properly. This is a no go. We don't want to keep a cache of *all* nodeids in the registry. Only the root nodes of spaces. +- The new storage driver only has a nodeid which might collide with other nodeids from other storage spaces, eg when two instances are imported into one ocis instance. Although it would be possible to just set up two storage providers extra care would have to be taken to prevent nodeid collisions when importing a space. + +If we now have to fetch the path on the `/dav/meta/` endpoint: +`/dav/meta/instanceprefix!1` would work because it is the root of a space +`/dav/meta/instanceprefix!2` would cause the gateway to poll all storage providers because the registry has no way to determine the responsible storage provider +`/dav/meta/instanceprefix!3` same + +The problem is that without a part in the storageid that allows differentiating storage spaces we cannot route them individually. + +Now, we could use the nodeid of the root of a storage space as the spaceid ... if it is a uuid. If it is numeric it needs a prefix to distinguish it from other spaces. +`!` would be easy for the decomposedfs. +eos might use numeric ids: `$!`, but it needs a custom prefix to distinguish multiple eos instances. + +Furthermore, when migrating spaces between storage providers we want to stay collision free, which is why we should recommend uuids. + +All this has implications for the decomposedfs, because it needs to split the nodes per space to prevent them from colliding. From a022ab78f354c9c871f331f8267fa9614dfd91f4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rn=20Friedrich=20Dreyer?= Date: Fri, 23 Jul 2021 15:17:18 +0000 Subject: [PATCH 07/56] rewrite with stronger spaces emphasis MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Jörn Friedrich Dreyer --- docs/extensions/storage/_index.md | 29 +- docs/extensions/storage/namespaces.md | 8 +- docs/extensions/storage/releasing.md | 31 - docs/extensions/storage/spaces.md | 28 + .../{architecture.md => spacesprovider.md} | 41 +- docs/extensions/storage/spacesregistry.md | 21 + .../storage/static/spacesprovider.drawio.svg | 352 +++++++++++ .../storage/static/spacesregistry.drawio.svg | 595 +++++++++--------- .../storage/static/storage.drawio.svg | 434 +++++++++++++ .../{storages.md => storagedrivers.md} | 12 +- docs/extensions/storage/terminology.md | 52 +- docs/extensions/storage/updating.md | 19 - docs/extensions/storage/users.md | 14 +- docs/ocis/deployment/ocis_keycloak.md | 2 +- docs/ocis/deployment/ocis_traefik.md | 2 +- 15 files changed, 1177 insertions(+), 463 deletions(-) delete mode 100644 docs/extensions/storage/releasing.md create mode 100644 docs/extensions/storage/spaces.md rename docs/extensions/storage/{architecture.md => spacesprovider.md} (77%) create mode 100644 docs/extensions/storage/spacesregistry.md create mode 100644 docs/extensions/storage/static/spacesprovider.drawio.svg create mode 100644 docs/extensions/storage/static/storage.drawio.svg rename docs/extensions/storage/{storages.md => storagedrivers.md} (97%) delete mode 100644 docs/extensions/storage/updating.md diff --git a/docs/extensions/storage/_index.md b/docs/extensions/storage/_index.md index 1f8bcf5e50..1d5023ed34 100644 --- a/docs/extensions/storage/_index.md +++ b/docs/extensions/storage/_index.md @@ -8,29 +8,12 @@ geekdocFilePath: _index.md geekdocCollapseSection: true --- -## Abstract +## Overview -This service provides an oCIS extension that wraps [reva](https://github.com/cs3org/reva/) and adds an opinionated configuration to it. +The storage extension wraps [reva](https://github.com/cs3org/reva/) and adds an opinionated configuration to provide two core services for the oCIS platform: +1. A [*Spaces Registry*]({{< ref "./spacesregistry.md" >}}) that acts as a dictionary for storage *Spaces* and their metadata +2. A [*Spaces Provider*]({{< ref "./spacesprovider.md" >}}) that organizes *Resources* in storage *Spaces* and persists them in an underlying *Storage System* -## Architecture Overview +*Clients* will use the *Spaces Registry* to poll or get notified about changes in all *Spaces* a user has access to. Every *Space* has a dedicated `/dav/spaces/` WebDAV endpoint that is served by a *Spaces Provider* which uses a specific reva storage driver to wrap an underlying *Storage System*. -The below diagram shows the oCIS services and the contained reva services within as dashed boxes. In general: -1. A request comes in at the proxy and is authenticated using OIDC. -2. It is forwarded to the oCIS frontend which handles ocs and ocdav requests by talking to the reva gateway using the CS3 API. -3. The gateway acts as a facade to the actual CS3 services: storage providers, user providers, group providers and sharing providers. - -{{< svg src="extensions/storage/static/overview.drawio.svg" >}} - -The dashed lines in the diagram indicate requests that are made to authenticate requests or lookup the storage provider: -1. After authenticating a request, the proxy may either use the CS3 `userprovider` or the accounts service to fetch the user information that will be minted into the `x-access-token`. -2. The gateway will verify the JWT signature of the `x-access-token` or try to authenticate the request itself, e.g. using a public link token. - -{{< hint warning >}} -The bottom part is lighter because we will deprecate it in favor of using only the CS3 user and group providers after moving some account functionality into reva and glauth. The metadata storage is not registered in the reva gateway to seperate metadata necessary for running the service from data that is being served directly. -{{< /hint >}} - -## Endpoints and references - -In order to reason about the request flow, two aspects in the architecture need to be understood well: -1. What kind of [*namespaces*]({{< ref "./namespaces.md" >}}) are presented at the different WebDAV and CS3 endpoints? -2. What kind of [*resource*]({{< ref "./terminology.md#resources" >}}) [*references*]({{< ref "./terminology.md#references" >}}) are exposed or required: path or id based? +{{< svg src="extensions/storage/static/storage.drawio.svg" >}} diff --git a/docs/extensions/storage/namespaces.md b/docs/extensions/storage/namespaces.md index 3ca0f9981d..eb68887c73 100644 --- a/docs/extensions/storage/namespaces.md +++ b/docs/extensions/storage/namespaces.md @@ -12,7 +12,7 @@ In ownCloud 10 all paths are considered relative to the users home. The CS3 API {{< svg src="extensions/storage/static/namespaces.drawio.svg" >}} -The different paths in the namespaces need to be translated while passing [*references*]({{< ref "./terminology.md#references" >}}) from service to service. While the oc10 endpoints all work on paths we internally reference shared resources by id, so the shares don't break when a file is renamed or moved inside a [*storage space*]({{< ref "./terminology.md#storage-spaces" >}}). The following table lists the various namespaces, paths and id based references: +The different paths in the namespaces need to be translated while passing [*references*]({{< ref "./terminology.md#references" >}}) from service to service. While the oc10 endpoints all work on paths we internally reference shared resources by id, so the shares don't break when a file is renamed or moved inside a storage [*space*]({{< ref "./spaces" >}}). The following table lists the various namespaces, paths and id based references: | oc10 namespace | CS3 global namespace | storage provider | reference | content | |--------------------------------------------------|----------------------------------------|------------------|-----------|---------| @@ -32,13 +32,13 @@ In the global CS3 namespaces we plan to move `/home/Shares`, which currently lis ## ownCloud namespaces -In contrast to the global namespace of CS3, ownCloud always presented a user specific namespace on all endpoints. It will always list the users private files under `/`. Shares can be mounted at an arbitrary location in the users private spaces. See the [webdav]({{< ref "./architecture#webdav" >}}) and [ocs]({{< ref "./architecture#sharing" >}}) sections for more details end examples. +In contrast to the global namespace of CS3, ownCloud always presented a user specific namespace on all endpoints. It will always list the users private files under `/`. Shares can be mounted at an arbitrary location in the users private spaces. See the [webdav]({{< ref "./spacesprovider#webdav" >}}) and [ocs]({{< ref "./spacesprovider#sharing" >}}) sections for more details end examples. With the spaces concept we are planning to introduce a global namespace to the ownCloud webdav endpoints. This will push the users private space down in the hierarchy: it will move from `/webdav` to `/webdav/home` or `/webdav/users/`. The related [migration stages]({{< ref "../../ocis/migration.md" >}}) are subject to change. ## CS3 global namespaces -The *CS3 global namespace* in oCIS is configured in the [*storage space registry*]({{< ref "./terminology.md#storage-space-registries" >}}). oCIS uses these defaults: +The *CS3 global namespace* in oCIS is configured in the storage [*spaces registry*]({{< ref "./spacesregistry" >}}). oCIS uses these defaults: | global namespace | description | |-|-| @@ -48,7 +48,7 @@ The *CS3 global namespace* in oCIS is configured in the [*storage space registry | `/public/` | a virtual folder listing public shares | | `/spaces/` | *TODO: project or group spaces* | -Technically, the `/home` namespace is not necessary: the [*storage space registry*]({{< ref "./terminology.md#storage-space-registries" >}}) knows the path to a users private space in the `/users` namespace and the gateway can forward the requests to the responsible storage provider. +Technically, the `/home` namespace is not necessary: the storage [*spaces registry*]({{< ref "./spacesregistry" >}}) knows the path to a users private space in the `/users` namespace and the gateway can forward the requests to the responsible storage provider. {{< hint warning >}} *@jfd: Why don't we use `/home/` instead of `/users/`. Then the paths would be consistent with most unix systems. diff --git a/docs/extensions/storage/releasing.md b/docs/extensions/storage/releasing.md deleted file mode 100644 index b55369d32e..0000000000 --- a/docs/extensions/storage/releasing.md +++ /dev/null @@ -1,31 +0,0 @@ ---- -title: "Releasing" -date: 2020-05-22T00:00:00+00:00 -weight: 60 -geekdocRepo: https://github.com/owncloud/ocis -geekdocEditPath: edit/master/docs/extensions/storage -geekdocFilePath: releasing.md ---- - -{{< toc >}} - -To release a new version of the storage submodule, you have to follow a few simple steps. - -## Preparation - -1. Before releasing, make sure that reva has been [updated to the desired version]({{< ref "updating" >}}) - -## Release -1. Check out master -{{< highlight txt >}} -git checkout master -git pull origin master -{{< / highlight >}} -2. Create a new tag (preferably signed) and replace the version number accordingly. Prefix the tag with the submodule `storage/v`. -{{< highlight txt >}} -git tag -s storage/vx.x.x -m "release vx.x.x" -git push origin storage/vx.x.x -{{< / highlight >}} -5. Wait for CI and check that the GitHub release was published. - -Congratulations, you just released the storage submodule! diff --git a/docs/extensions/storage/spaces.md b/docs/extensions/storage/spaces.md new file mode 100644 index 0000000000..f1342b4074 --- /dev/null +++ b/docs/extensions/storage/spaces.md @@ -0,0 +1,28 @@ +--- +title: "Spaces" +date: 2018-05-02T00:00:00+00:00 +weight: 3 +geekdocRepo: https://github.com/owncloud/ocis +geekdocEditPath: edit/master/docs/extensions/storage +geekdocFilePath: spaces.md +--- + +{{< hint warning >}} + +The current implementation in oCIS might not yet fully reflect this concept. Feel free to add links to ADRs, PRs and Issues in short warning boxes like this. + +{{< /hint >}} + +## Storage Spaces +A storage *space* is a logical concept. It organizes a set of [*resources*]({{< ref "#resources" >}}) in a hierarchical tree. It has a single *owner* (*user* or *group*), +a *quota*, *permissions* and is identified by a `storage space id`. + +{{< svg src="extensions/storage/static/storagespace.drawio.svg" >}} + +Examples would be every user's personal storage *space*, project storage *spaces* or group storage *spaces*. While they all serve different purposes and may or may not have workflows like anti virus scanning enabled, we need a way to identify and manage these subtrees in a generic way. By creating a dedicated concept for them this becomes easier and literally makes the codebase cleaner. A storage [*Spaces Registry*]({{< ref "./spacesregistry.md" >}}) then allows listing the capabilities of storage *spaces*, e.g. free space, quota, owner, syncable, root etag, upload workflow steps, ... + +Finally, a logical `storage space id` is not tied to a specific [*spaces provider*]({{< ref "./spacesprovider.md" >}}). If the [*storage driver*]({{< ref "./storagedrivers.md" >}}) supports it, we can import existing files including their `file id`, which makes it possible to move storage *spaces* between [*spaces providers*]({{< ref "./spacesprovider.md" >}}) to implement storage classes, e.g. with or without archival, workflows, on SSDs or HDDs. + +## Shares +*To be clarified: we are aware that [*storage spaces*]({{< ref "#storage-spaces" >}}) may be too 'heavywheight' for ad hoc sharing with groups. That being said, there is no technical reason why group shares should not be treated like storage [*spaces*]({{< ref "#storage-spaces" >}}) that users can provision themselves. They would share the quota with the users home or personal storage [*space*]({{< ref "#storage-spaces" >}}) and the share initiator would be the sole owner. Technically, the mechanism of treating a share like a new storage [*space*]({{< ref "#storage-spaces" >}}) would be the same. This obviously also extends to user shares and even file individual shares that would be wrapped in a virtual collection. It would also become possible to share collections of arbitrary files in a single storage space, e.g. the ten best pictures from a large album.* + diff --git a/docs/extensions/storage/architecture.md b/docs/extensions/storage/spacesprovider.md similarity index 77% rename from docs/extensions/storage/architecture.md rename to docs/extensions/storage/spacesprovider.md index ab8683234b..2100a3fa47 100644 --- a/docs/extensions/storage/architecture.md +++ b/docs/extensions/storage/spacesprovider.md @@ -1,12 +1,25 @@ --- -title: "Architecture" +title: "Spaces Provider" date: 2018-05-02T00:00:00+00:00 -weight: 10 +weight: 6 geekdocRepo: https://github.com/owncloud/ocis geekdocEditPath: edit/master/docs/extensions/storage -geekdocFilePath: architecture.md +geekdocFilePath: spacesprovider.md --- +{{< hint warning >}} + +The current implementation in oCIS might not yet fully reflect this concept. Feel free to add links to ADRs, PRs and Issues in short warning boxes like this. + +{{< /hint >}} + +## Spaces Provider +A *storage provider* manages [*resources*]({{< ref "#resources" >}}) identified by a [*reference*]({{< ref "#references" >}}) +by accessing a [*storage system*]({{< ref "#storage-systems" >}}) with a [*storage driver*]({{< ref "./storagedrivers.md" >}}). + +{{< svg src="extensions/storage/static/spacesprovider.drawio.svg" >}} + + ## Frontend The oCIS frontend service starts all services that handle incoming HTTP requests: @@ -38,16 +51,17 @@ The ocdav service not only handles all WebDAV requests under `(remote.php/)(web) | `(remote.php/)webdav/users` | ocdav | storageprovider | `/users` | | | | `(remote.php/)dav/files/` | ocdav | storageprovider | `/users/` | | | | *Spaces concept also needs a new endpoint:* ||||| -| `(remote.php/)dav/spaces//` | ocdav | storageregistry & storageprovider | bypass path based namespace and directly talk to the responsible storage provider using a relative path | [spaces concept](https://github.com/owncloud/ocis/pull/1827) needs to point to [*storage spaces*]({{< ref "./terminology.md#storage-spaces" >}}) or a global endpoint | allow accessing spaces, listing is done by the graph api | +| `(remote.php/)dav/spaces//` | ocdav | storageregistry & storageprovider | bypass path based namespace and directly talk to the responsible storage provider using a relative path | [spaces concept](https://github.com/owncloud/ocis/pull/1827) needs to point to storage [*spaces*]({{< ref "./spaces.md" >}}) | allow accessing spaces, listing is done by the graph api | -The correct endpoint for a users home [*storage space*]({{< ref "./terminology.md#storage-spaces" >}}) in oc10 is `remote.php/dav/files/`. In oc10 All requests at this endpoint use a path based reference that is relative to the users home. In oCIS this can be configured and defaults to `/home` as well. Other API endpoints like ocs and the web UI still expect this to be the users home. +The correct endpoint for a users home storage [*space*]({{< ref "./spaces.md" >}}) in oc10 is `remote.php/dav/files/`. In oc10 all requests at this endpoint use a path based reference that is relative to the users home. In oCIS this can be configured and defaults to `/home` as well. Other API endpoints like ocs and the web UI still expect this to be the users home. In oc10 we originally had `remote.php/webdav` which would render the current users home [*storage space*]({{< ref "./terminology.md#storage-spaces" >}}). The early versions (pre OC7) would jail all received shares into a `remote.php/webdav/shares` subfolder. The semantics for syncing such a folder are [not trivially predictable](https://github.com/owncloud/core/issues/5349), which is why we made shares [freely mountable](https://github.com/owncloud/core/pull/8026) anywhere in the users home. The current reva implementation jails shares into a `remote.php/webdav/Shares` folder for performance reasons. Obviously, this brings back the [special semantics for syncing](https://github.com/owncloud/product/issues/7). In the future we will follow [a different solution](https://github.com/owncloud/product/issues/302) and jail the received shares into a dedicated `/shares` space, on the same level as `/home` and `/spaces`. We will add a dedicated [API to list all *storage spaces*](https://github.com/owncloud/ocis/pull/1827) a user has access to and where they are mounted in the users *namespace*. {{< hint warning >}} +TODO rewrite this hint with `/dav/spaces` Existing folder sync pairs in legacy clients will break when moving the user home down in the path hierarchy like CernBox did. For legacy clients the `remote.php/webdav` endpoint will no longer list the users home directly, but instead present the different types of storage spaces: - `remote.php/webdav/home`: the users home is pushed down into a new `home` [*storage space*]({{< ref "./terminology.md#storage-spaces" >}}) @@ -55,11 +69,6 @@ For legacy clients the `remote.php/webdav` endpoint will no longer list the user - `remote.php/webdav/spaces`: other [*storage spaces*]({{< ref "./terminology.md#storage-spaces" >}}) the user has access to, e.g. group or project drives {{< /hint >}} -{{< hint warning >}} -An alternative would be to introduce a new `remote.php/dav/spaces` or `remote.php/dav/global` endpoint. However, `remote.php/dav` properly follows the WebDAV RFCs strictly. To ensure that all resources under that [*namespace*]({{< ref "./terminology.md#namespaces" >}}) are scoped to the user the URL would have to include the principal like `remote.php/dav/spaces/`, a precondition for e.g. WebDAV [RFC5397](https://tools.ietf.org/html/rfc5397). For a history lesson start at [Replace WebDAV with REST -owncloud/core#12504](https://github.com/owncloud/core/issues/12504#issuecomment-65218491) which spawned [Add extra layer in DAV to accomodate for other services like versions, trashbin, etc owncloud/core#12543](https://github.com/owncloud/core/issues/12543) -{{< /hint >}} - ### Sharing @@ -92,12 +101,12 @@ The user and public share provider implementations identify the file using the [ The OCM API takes an id based reference on the CS3 api, even if the OCM HTTP endpoint takes a path argument. *@jfd: Why? Does it not need the owner? It only stores the owner of the share, which is always the currently looged in user, when creating a share. Afterwards only the owner can update a share ... so collaborative management of shares is not possible. At least for OCM shares.* {{< /hint >}} -### User and Group provisioning -In oc10 users are identified by a username, which cannot change, because it is used as a foreign key in several tables. For oCIS we are internally identifying users by a UUID, while using the username in the WebDAV and OCS APIs for backwards compatability. To distinguish this in the URLs we are using `` instead of ``. You may have encountered ``, which refers to a template that can be configured to build several path segments by filling in user properties, e.g. the first character of the username (`{{substr 0 1 .Username}}/{{.Username}}`), the identity provider (`{{.Id.Idp}}/{{.Username}}`) or the email (`{{.Mail}}`) +## REVA Storage Registry -{{< hint warning >}} -Make no mistake, the [OCS Provisioning API](https://doc.owncloud.com/server/developer_manual/core/apis/provisioning-api.html) uses `userid` while it actually is the username, because it is what you use to login. -{{< /hint >}} +The reva *storage registry* manages the [*CS3 global namespace*]({{< ref "./namespaces.md#cs3-global-namespaces" >}}): +It is used by the reva *gateway* +to look up `address` and `port` of the [*storage provider*]({{< ref "#storage-providers" >}}) +that should handle a [*reference*]({{< ref "#references" >}}). -We are currently working on adding [user management through the CS3 API](https://github.com/owncloud/ocis/pull/1930) to handle user and group provisioning (and deprovisioning). +{{< svg src="extensions/storage/static/storageregistry.drawio.svg" >}} \ No newline at end of file diff --git a/docs/extensions/storage/spacesregistry.md b/docs/extensions/storage/spacesregistry.md new file mode 100644 index 0000000000..d5be48f8ab --- /dev/null +++ b/docs/extensions/storage/spacesregistry.md @@ -0,0 +1,21 @@ +--- +title: "Spaces Registry" +date: 2018-05-02T00:00:00+00:00 +weight: 9 +geekdocRepo: https://github.com/owncloud/ocis +geekdocEditPath: edit/master/docs/extensions/storage +geekdocFilePath: spacesregistry.md +--- + +{{< hint warning >}} + +The current implementation in oCIS might not yet fully reflect this concept. Feel free to add links to ADRs, PRs and Issues in short warning boxes like this. + +{{< /hint >}} + +## Storage Space Registries + +A storage *spaces registry* manages the [*namespace*]({{< ref "./namespaces.md" >}}) for a *user*: it is used by *clients* to look up storage spaces a user has access to, the `/dav/spaces` endpoint to access it via WabDAV, and where the client should mount it in the users personal namespace. + +{{< svg src="extensions/storage/static/spacesregistry.drawio.svg" >}} + diff --git a/docs/extensions/storage/static/spacesprovider.drawio.svg b/docs/extensions/storage/static/spacesprovider.drawio.svg new file mode 100644 index 0000000000..d122c58f7a --- /dev/null +++ b/docs/extensions/storage/static/spacesprovider.drawio.svg @@ -0,0 +1,352 @@ + + + + + + + +
+
+
+ oCIS spaces provider +
+ [Software System] +
+
+
+ + + oCIS spaces provider... + + + + + + + +
+
+
+ + reva storage provider + +
+ [Component: golang] +
+
+
+ hosts multiple storage spaces using a storage driver +
+
+
+
+ + + reva storage provider... + + + + + + + +
+
+
+ + reva gateway + +
+ [Component: golang] +
+
+
+ API facade for internal reva services +
+
+
+
+ + + reva gateway... + + + + + + + +
+
+
+ + Storage System + +
+ [Software System] +
+
+
+ provides persistent storage +
+
+
+
+ + + Storage System... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [POSIX, S3] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + +
+
+
+ + reva frontend + +
+ [Component: golang] +
+
+
+ handles protocol translation +
+
+
+
+ + + reva frontend... + + + + + + + +
+
+
+ + oCIS proxy + +
+ [Component: golang] +
+
+
+ Routes requests to oc10 or ecis +
+
+
+
+ + + oCIS proxy... + + + + + + + + +
+
+
+
+
+ + Mints an internal JWT +
+ and torwards requests to + +
+
+ [WebDAV, OCS, OCM, tus] +
+
+
+
+
+ + + Mints an internal JWT... + + + + + + + +
+
+
+ + Client + +
+ [Container: C++, Kotlin, +
+ Swift or Vue] +
+
+
+ A desktop, mobile or web Client +
+
+
+
+ + + Client... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [WebDAV, libregraph, CS3] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [CS3, tus] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + + +
+
+
+
+
+ + Forwards to + +
+
+ [CS3, storage registry] +
+
+
+
+
+ + + Forwards to... + + + + + + + +
+
+
+

+ C4 Component diagram for an oCIS spaces provider +

+

+ An oCIS spaces provider manages resources in storage spaces by persisting them with a specific storage driver in a storage system. +

+

+ Date: 2021-07-22T12:40 +

+
+
+
+ + + C4 Component diagram for an oCIS spaces provider... + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/docs/extensions/storage/static/spacesregistry.drawio.svg b/docs/extensions/storage/static/spacesregistry.drawio.svg index e3a771b07e..f716b0bb0d 100644 --- a/docs/extensions/storage/static/spacesregistry.drawio.svg +++ b/docs/extensions/storage/static/spacesregistry.drawio.svg @@ -1,136 +1,82 @@ - + - + -
+
- oCIS System + oCIS spaces registry
[Software System]
- - oCIS System... + + oCIS spaces registry... - - - + -
-
-
- - Einstein - -
- [Person] -
-
-
- End user -
-
-
-
- - - Einstein... - - - - - - - -
+
- Client + reva storage registry
- [Container: C++, Kotlin, Swift or Vue] + [Component: golang]

- A desktop, mobile or web Client + manages and caches storage space metadata
- - Client... + + reva storage registry... - + -
+
- Storage Space Registry + reva gateway
- [Container: golang, HTTP, libregraph] + [Component: golang]

- Manages spaces for users + API facade for internal reva services
- - Storage Space Registry... + + reva gateway... - + -
-
-
- - Storage Provider - -
- [Container: golang] -
-
-
- Persists storage spaces using reva -
-
-
-
- - - Storage Provider... - - - - - - - -
+
@@ -147,45 +93,285 @@
- + Storage System... - - - + + -
+
-
- - Moss - -
- [Person] -
-
-
- Administrator +
+
+
+ + Provisions and manages spaces in + +
+
+ [CS3] +
- - Moss... + + Provisions and manages spaces... - - + -
+
+
+
+ + reva frontend + +
+ [Component: golang] +
+
+
+ handles protocol translation +
+
+
+
+ + + reva frontend... + + + + + + + +
+
+
+ + oCIS proxy + +
+ [Component: golang] +
+
+
+ Routes requests to oc10 or ecis +
+
+
+
+ + + oCIS proxy... + + + + + + + + +
+
+
+
+
+ + Mints an internal JWT +
+ and torwards requests to + +
+
+ [libregraph] +
+
+
+
+
+ + + Mints an internal JWT... + + + + + + + +
+
+
+ + Client + +
+ [Container: C++, Kotlin, +
+ Swift or Vue] +
+
+
+ A desktop, mobile or web Client +
+
+
+
+ + + Client... + + + + + + + + +
+
+
+
+
+ + polls or gets notified about changes in + +
+
+ [libregraph] +
+
+
+
+
+ + + polls or gets notified about c... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [CS3, tus] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + + +
+
+
+
+
+ + Lists spaces using + +
+
+ [CS3] +
+
+
+
+
+ + + Lists spaces using... + + + + + + + +
+
+
+

+ C4 Component diagram for an oCIS spaces registry +

+

+ An oCIS spaces provider manages resources in storage spaces by persisting them with a specific storage driver in a storage system. +

+

+ Date: 2021-07-22T12:40 +

+
+
+
+ + + C4 Component diagram for an oCIS spaces registry... + + + + + + + +
+
+
+ + reva storage provider + +
+ [Component: golang] +
+
+
+ hosts multiple storage spaces using a storage driver +
+
+
+
+ + + reva storage provider... + + + + + + + + +
@@ -202,226 +388,11 @@
- + Reads from and writes to... - - - - - -
-
-
-
-
- - Reads from and writes to - -
-
- [WebDAV, libregraph, CS3, tus] -
-
-
-
-
- - - Reads from and writes to... - - - - - - - - -
-
-
-
-
- - Manages the users Storage Spaces in - -
-
- [libregraph] -
-
-
-
-
- - - Manages the users Storage Spac... - - - - - - - - -
-
-
-
-
- - Manages resources with - -
-
- [Web UI or native clients] -
-
-
-
-
- - - Manages resources with... - - - - - - - - -
-
-
-
-
- - Registers itself at and -
- sends space root etag changes to - -
-
- [CS3, libregraph?, PUSH] -
-
-
-
-
- - - Registers itself at and... - - - - - - - - -
-
-
-
-
- - Manages organizational Storage Spaces in - -
-
- [WebDAV, libregraph, CS3, CLI] -
-
-
-
-
- - - Manages organizational Storage... - - - - - - - -
-
-
- - Identity Management System - -
- [Software System] -
-
-
- provides users and groups -
-
-
-
- - - Identity Management System... - - - - - - - - -
-
-
-
-
- - Authenticates users and searches recipients with - -
-
- [OpenID Connect, LDAP, REST] -
-
-
-
-
- - - Authenticates users and search... - - - - - - - -
-
-
-

- C4 Container diagram for the oCIS System -

-

- As a platform, the oCIS system may not only includes web, mobile and desktop clients but also the underlying storage system or an identity management system -

-

- Date: 2021-07-22T16:43 -

-
-
-
- - - C4 Container diagram for the oCIS System... - - - diff --git a/docs/extensions/storage/static/storage.drawio.svg b/docs/extensions/storage/static/storage.drawio.svg new file mode 100644 index 0000000000..fd6e759cf3 --- /dev/null +++ b/docs/extensions/storage/static/storage.drawio.svg @@ -0,0 +1,434 @@ + + + + + + + +
+
+
+ oCIS System +
+ [Software System] +
+
+
+ + + oCIS System... + + + + + + + + + +
+
+
+ + Einstein + +
+ [Person] +
+
+
+ End user +
+
+
+
+ + + Einstein... + + + + + + + +
+
+
+ + Client + +
+ [Container: C++, Kotlin, Swift or Vue] +
+
+
+ A desktop, mobile or web Client +
+
+
+
+ + + Client... + + + + + + + +
+
+
+ + Storage Space Registry + +
+ [Container: golang, HTTP, libregraph] +
+
+
+ Manages spaces for users +
+
+
+
+ + + Storage Space Registry... + + + + + + + +
+
+
+ + Storage Space Provider + +
+ [Container: golang] +
+
+
+ Persists storage spaces using reva +
+
+
+
+ + + Storage Space Provider... + + + + + + + +
+
+
+ + Storage System + +
+ [Software System] +
+
+
+ provides persistent storage +
+
+
+
+ + + Storage System... + + + + + + + + + +
+
+
+ + Moss + +
+ [Person] +
+
+
+ Administrator +
+
+
+
+ + + Moss... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [POSIX, S3] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [WebDAV, libregraph, CS3, tus] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + + +
+
+
+
+
+ + Manages the users Storage Spaces in + +
+
+ [libregraph] +
+
+
+
+
+ + + Manages the users Storage Spac... + + + + + + + + +
+
+
+
+
+ + Manages resources with + +
+
+ [Web UI or native clients] +
+
+
+
+
+ + + Manages resources with... + + + + + + + + +
+
+
+
+
+ + Registers itself at and +
+ sends space root etag changes to + +
+
+ [CS3, libregraph?, PUSH] +
+
+
+
+
+ + + Registers itself at and... + + + + + + + + +
+
+
+
+
+ + Manages organizational Storage Spaces in + +
+
+ [WebDAV, libregraph, CS3, CLI] +
+
+
+
+
+ + + Manages organizational Storage... + + + + + + + +
+
+
+ + Identity Management System + +
+ [Software System] +
+
+
+ provides users and groups +
+
+
+
+ + + Identity Management System... + + + + + + + + +
+
+
+
+
+ + Authenticates users and searches recipients with + +
+
+ [OpenID Connect, LDAP, REST] +
+
+
+
+
+ + + Authenticates users and search... + + + + + + + +
+
+
+

+ C4 Container diagram for the oCIS System +

+

+ As a platform, the oCIS system may not only includes web, mobile and desktop clients but also the underlying storage system or an identity management system +

+

+ Date: 2021-07-22T16:43 +

+
+
+
+ + + C4 Container diagram for the oCIS System... + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/docs/extensions/storage/storages.md b/docs/extensions/storage/storagedrivers.md similarity index 97% rename from docs/extensions/storage/storages.md rename to docs/extensions/storage/storagedrivers.md index 59927fd933..9a9d7fb4fd 100644 --- a/docs/extensions/storage/storages.md +++ b/docs/extensions/storage/storagedrivers.md @@ -1,15 +1,18 @@ --- -title: "Storages" +title: "Storage drivers" date: 2020-04-27T18:46:00+01:00 -weight: 37 +weight: 12 geekdocRepo: https://github.com/owncloud/ocis geekdocEditPath: edit/master/docs/extensions/storage geekdocFilePath: storages.md --- -## Storage commands +A *storage driver* implements access to a [*storage system*]({{< ref "#storage-systems" >}}): -`storage` has multiple storage provider commands to preconfigure different default configurations for the reva *storage provider* service. While you could rerun `storage storage-oc` multiple times with different flags to get multiple instances we are giving the different commands the necessary default configuration to allow the `ocis` binary to simply start them and not deal with configuration. +It maps the *path* and *id* based CS3 *references* to an appropriate [*storage system*]({{< ref "#storage-systems" >}}) specific reference, e.g.: +- eos file ids +- posix inodes or paths +- deconstructed filesystem nodes ## Storage providers @@ -25,7 +28,6 @@ A lot of different storage technologies exist, ranging from general purpose file Unfortunately, no POSIX filesystem natively supports all storage aspects that ownCloud 10 requires: - ### A hierarchical file tree An important aspect of a filesystem is organizing files and directories in a file hierarchy, or tree. It allows you to create, move and delete nodes. Beside the name a node also has well known metadata like size and mtime that are persisted in the tree as well. diff --git a/docs/extensions/storage/terminology.md b/docs/extensions/storage/terminology.md index 2b883b5705..426c8448be 100644 --- a/docs/extensions/storage/terminology.md +++ b/docs/extensions/storage/terminology.md @@ -19,7 +19,6 @@ A *resource* is the basic building block that oCIS manages. It can be of [differ - a [*reference*]({{< ref "#references" >}}) which can point to a resource in another [*storage provider*]({{< ref "#storage-providers" >}}) ### References - A *reference* identifies a [*resource*]({{< ref "#resources" >}}). A [*CS3 reference*](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.Reference) can carry a *path* and a [CS3 *resource id*](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId). The references come in two flavors: absolute and combined. Absolute references have either the *path* or the *resource id* set: - An absolute *path* MUST start with a `/`. The *resource id* MUST be empty. @@ -28,59 +27,12 @@ Combined references have both, *path* and *resource id* set: - the *resource id* identifies the root [*resource*]({{< ref "#resources" >}}) - the *path* is relative to that root. It MUST start with `.` - -### Storage Spaces -A *storage space* organizes a set of [*resources*]({{< ref "#resources" >}}) in a hierarchical tree. It has a single *owner* (*user* or *group*), -a *quota*, *permissions* and is identified by a `storage space id`. - -{{< svg src="extensions/storage/static/storagespace.drawio.svg" >}} - -Examples would be every user's personal storage space, project storage spaces or group storage spaces. While they all serve different purposes and may or may not have workflows like anti virus scanning enabled, we need a way to identify and manage these subtrees in a generic way. By creating a dedicated concept for them this becomes easier and literally makes the codebase cleaner. A [*storage space registry*]({{< ref "#storage-space-registries" >}}) then allows listing the capabilities of [*storage spaces*]({{< ref "#storage-spaces" >}}), e.g. free space, quota, owner, syncable, root etag, upload workflow steps, ... - -Finally, a logical `storage space id` is not tied to a specific [*storage provider*]({{< ref "#storage-providers" >}}). If the [*storage driver*]({{< ref "#storage-drivers" >}}) supports it, we can import existing files including their `file id`, which makes it possible to move [*storage spaces*]({{< ref "#storage-spaces" >}}) between [*storage providers*]({{< ref "#storage-providers" >}}) to implement storage classes, e.g. with or without archival, workflows, on SSDs or HDDs. - -### Shares -*To be clarified: we are aware that [*storage spaces*]({{< ref "#storage-spaces" >}}) may be too 'heavywheight' for ad hoc sharing with groups. That being said, there is no technical reason why group shares should not be treated like [*storage spaces*]({{< ref "#storage-spaces" >}}) that users can provision themselves. They would share the quota with the users home [*storage space*]({{< ref "#storage-spaces" >}}) and the share initiator would be the sole owner. Technically, the mechanism of treating a share like a new [*storage space*]({{< ref "#storage-spaces" >}}) would be the same. This obviously also extends to user shares and even file indvidual shares that would be wrapped in a virtual collection. It would also become possible to share collections of arbitrary files in a single storage space, e.g. the ten best pictures from a large album.* - - -### Storage Space Registries - -A *storage space registry* manages the [*namespace*]({{< ref "./namespaces.md" >}}) for a *user*: it is used by *clients* to look up storage spaces a user has access to, the `/dav/spaces` endpoint to access it via WabDAV, and where the client should mount it in the users personal namespace. - -{{< svg src="extensions/storage/static/spacesregistry.drawio.svg" >}} - - ## Technical concepts -### Storage Drivers - -A *storage driver* implements access to a [*storage system*]({{< ref "#storage-systems" >}}): - -It maps the *path* and *id* based CS3 *references* to an appropriate [*storage system*]({{< ref "#storage-systems" >}}) specific reference, e.g.: -- eos file ids -- posix inodes or paths -- deconstructed filesystem nodes - -### Storage Providers - -A *storage provider* manages [*resources*]({{< ref "#resources" >}}) identified by a [*reference*]({{< ref "#references" >}}) -by accessing a [*storage system*]({{< ref "#storage-systems" >}}) with a [*storage driver*]({{< ref "#storage-drivers" >}}). - -{{< svg src="extensions/storage/static/storageprovider.drawio.svg" >}} - -### Storage Registry - -A *storage registry* manages the [*CS3 global namespace*]({{< ref "./namespaces.md#cs3-global-namespaces" >}}): -It is used by the *gateway* -to look up `address` and `port` of the [*storage provider*]({{< ref "#storage-providers" >}}) -that should handle a [*reference*]({{< ref "#references" >}}). - -{{< svg src="extensions/storage/static/storageregistry.drawio.svg" >}} - ### Storage Systems Every *storage system* has different native capabilities like id and path based lookups, recursive change time propagation, permissions, trash, versions, archival and more. -A [*storage provider*]({{< ref "#storage-providers" >}}) makes the storage system available in the CS3 API by wrapping the capabilities as good as possible using a [*storage driver*]({{< ref "#storage-drivers" >}}). -There migt be multiple [*storage drivers*]({{< ref "#storage-drivers" >}}) for a *storage system*, implementing different tradeoffs to match varying requirements. +A [*storage provider*]({{< ref "#storage-providers" >}}) makes the storage system available in the CS3 API by wrapping the capabilities as good as possible using a [*storage driver*]({{< ref "./storagedrivers.md" >}}). +There might be multiple [*storage drivers*]({{< ref "./storagedrivers.md" >}}) for a *storage system*, implementing different tradeoffs to match varying requirements. ### Gateways A *gateway* acts as a facade to the storage related services. It authenticates and forwards API calls that are publicly accessible. diff --git a/docs/extensions/storage/updating.md b/docs/extensions/storage/updating.md deleted file mode 100644 index dd19203766..0000000000 --- a/docs/extensions/storage/updating.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -title: "Updating reva" -date: 2020-05-22T00:00:00+00:00 -weight: 50 -geekdocRepo: https://github.com/owncloud/ocis -geekdocEditPath: edit/master/docs/extensions/storage -geekdocFilePath: updating.md ---- - -{{< toc >}} - -## Updating reva - -1. Run `go get github.com/cs3org/reva@master` in all repos that depend on reva -2. Create a changelog entry containing changes that were done in [reva](https://github.com/cs3org/reva/commits/master) -3. Create a Pull Request to ocis master with those changes -4. If test issues appear, you might need to adjust the tests -5. After the PR is merged, consider doing a [release of the storage submodule]({{< ref "releasing" >}}) - diff --git a/docs/extensions/storage/users.md b/docs/extensions/storage/users.md index 4a5e716faf..8e5d034bad 100644 --- a/docs/extensions/storage/users.md +++ b/docs/extensions/storage/users.md @@ -1,12 +1,24 @@ --- title: "Users" date: 2020-01-16T00:00:00+00:00 -weight: 35 +weight: 17 geekdocRepo: https://github.com/owncloud/ocis geekdocEditPath: edit/master/docs/extensions/storage geekdocFilePath: users.md --- +TODO add this to the storage overview? or is this a different part? That should be started as a separate service ? And documented elsewhere, eg. in the accounts? + +### User and Group provisioning + +In oc10 users are identified by a username, which cannot change, because it is used as a foreign key in several tables. For oCIS we are internally identifying users by a UUID, while using the username in the WebDAV and OCS APIs for backwards compatability. To distinguish this in the URLs we are using `` instead of ``. You may have encountered ``, which refers to a template that can be configured to build several path segments by filling in user properties, e.g. the first character of the username (`{{substr 0 1 .Username}}/{{.Username}}`), the identity provider (`{{.Id.Idp}}/{{.Username}}`) or the email (`{{.Mail}}`) + +{{< hint warning >}} +Make no mistake, the [OCS Provisioning API](https://doc.owncloud.com/server/developer_manual/core/apis/provisioning-api.html) uses `userid` while it actually is the username, because it is what you use to login. +{{< /hint >}} + +We are currently working on adding [user management through the CS3 API](https://github.com/owncloud/ocis/pull/1930) to handle user and group provisioning (and deprovisioning). + ### Demo driver This is a simple user driver for testing. It contains three users: diff --git a/docs/ocis/deployment/ocis_keycloak.md b/docs/ocis/deployment/ocis_keycloak.md index 46ef4ad061..4cc2be6cc0 100644 --- a/docs/ocis/deployment/ocis_keycloak.md +++ b/docs/ocis/deployment/ocis_keycloak.md @@ -21,7 +21,7 @@ The docker stack consists 4 containers. One of them is Traefik, a proxy which is Keykloak add two containers: Keycloak itself and a PostgreSQL as database. Keycloak will be configured as oCIS' IDP instead of the internal IDP [LibreGraph Connect]({{< ref "../../extensions/idp" >}}) -The other container is oCIS itself running all extensions in one container. In this example oCIS uses [oCIS storage driver]({{< ref "../../extensions/storage/storages#storage-drivers" >}}) +The other container is oCIS itself running all extensions in one container. In this example oCIS uses the [oCIS storage driver]({{< ref "../../extensions/storage/storagedrivers" >}}) ## Server Deployment diff --git a/docs/ocis/deployment/ocis_traefik.md b/docs/ocis/deployment/ocis_traefik.md index 6bb3088776..cb44362fee 100644 --- a/docs/ocis/deployment/ocis_traefik.md +++ b/docs/ocis/deployment/ocis_traefik.md @@ -18,7 +18,7 @@ geekdocFilePath: ocis_traefik.md The docker stack consists of two containers. One of them is Traefik, a proxy which is terminating ssl and forwards the requests to oCIS in the internal docker network. -The other one is oCIS itself running all extensions in one container. In this example oCIS uses its internal IDP [LibreGraph Connect]({{< ref "../../extensions/idp" >}}) and the [oCIS storage driver]({{< ref "../../extensions/storage/storages#storage-drivers" >}}) +The other one is oCIS itself running all extensions in one container. In this example oCIS uses its internal IDP [LibreGraph Connect]({{< ref "../../extensions/idp" >}}) and the [oCIS storage driver]({{< ref "../../extensions/storage/storagedrivers" >}}) ## Server Deployment From 27863fdb4316e48c5e42dbfda94a5b51b98c17cc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rn=20Friedrich=20Dreyer?= Date: Mon, 13 Sep 2021 13:06:02 +0000 Subject: [PATCH 08/56] work MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Jörn Friedrich Dreyer --- docs/extensions/storage/namespaces.md | 6 +- docs/extensions/storage/proposedchanges.md | 30 +++---- docs/extensions/storage/spaces.md | 11 +++ docs/extensions/storage/spacesprovider.md | 2 +- .../storage/static/namespaces.drawio.svg | 83 +++++++++---------- docs/extensions/storage/users.md | 2 +- 6 files changed, 67 insertions(+), 67 deletions(-) diff --git a/docs/extensions/storage/namespaces.md b/docs/extensions/storage/namespaces.md index eb68887c73..19864a6e7f 100644 --- a/docs/extensions/storage/namespaces.md +++ b/docs/extensions/storage/namespaces.md @@ -16,7 +16,7 @@ The different paths in the namespaces need to be translated while passing [*refe | oc10 namespace | CS3 global namespace | storage provider | reference | content | |--------------------------------------------------|----------------------------------------|------------------|-----------|---------| -| `/webdav/path/to/file.ext` `/dav/files//path/to/file.ext` | `/home/path/to/file.ext` | home | `//path/to/file.ext` | currently logged in users home | +| `/webdav/path/to/file.ext` `/dav/files//path/to/file.ext` | `/home/path/to/file.ext` | home | `//path/to/file.ext` | currently logged in users home | | `/webdav/Shares/foo` `/dav/files//Shares/foo` | `/home/Shares/foo` | users | id based access | all users, used to access collaborative shares | | `/dav/public-files//rel/path/to/file.ext` | `/public//rel/path/to/file.ext` | public | id based access | publicly shared files, used to access public links | @@ -43,7 +43,7 @@ The *CS3 global namespace* in oCIS is configured in the storage [*spaces registr | global namespace | description | |-|-| | `/home` | an alias for the currently logged in uses private space | -| `/users/` | user private spaces | +| `/users/` | user private spaces | | `/shares` | a virtual listing of share spaces a user has access to | | `/public/` | a virtual folder listing public shares | | `/spaces/` | *TODO: project or group spaces* | @@ -51,7 +51,7 @@ The *CS3 global namespace* in oCIS is configured in the storage [*spaces registr Technically, the `/home` namespace is not necessary: the storage [*spaces registry*]({{< ref "./spacesregistry" >}}) knows the path to a users private space in the `/users` namespace and the gateway can forward the requests to the responsible storage provider. {{< hint warning >}} -*@jfd: Why don't we use `/home/` instead of `/users/`. Then the paths would be consistent with most unix systems. +*@jfd: Why don't we use `/home/` instead of `/users/`. Then the paths would be consistent with most unix systems. {{< /hint >}} The `/shares` namespace is used to solve two problems: diff --git a/docs/extensions/storage/proposedchanges.md b/docs/extensions/storage/proposedchanges.md index 29fe8c9d40..c306a4005b 100644 --- a/docs/extensions/storage/proposedchanges.md +++ b/docs/extensions/storage/proposedchanges.md @@ -13,29 +13,21 @@ Some architectural changes still need to be clarified or changed. Maybe an ADR i ## A dedicated shares storage provider -Currently, the *gateway* treats `/home/shares` different than any other path: it will stat all children and calculate an etag to allow clients to discover changes in accepted shares. This requires the storage provider to cooperate and provide this special `/shares` folder in the root of a users home when it is accessed as a home storage, which is a config flag that needs to be set for every storage driver. +Currently, when a user accepts a share, a cs3 reference is created in the users `/home/shares` folder. This reference represents the mount point of a share and can be renamed, similar to the share jail in ownCloud 10. This spreads the metadata of a share in two places: +- the share is persisted in the *share manager* +- the mount point of a share is persisted in the home *storage provider* -The `enable_home` flag will cause drivers to jail path based requests into a `` subfolder. In effect it divides a storage provider into multiple [*storage spaces*]({{< ref "#storage-spaces" >}}): when calling `CreateHome` a subfolder following the `` is created and market as the root of a users home. Both, the eos and ocis storage drivers use extended attributes to mark the folder as the end of the size aggregation and tree mtime propagation mechanism. Even setting the quota is possible like that. All this literally is a [*storage space*]({{< ref "#storage-spaces" >}}). +Furthermore, the *gateway* treats `/home/shares` different than any other path: it will stat all children and calculate an etag to allow clients to discover changes in accepted shares. This requires the storage provider to cooperate and provide this special `/shares` folder in the root of a users home when it is accessed as a home storage. That is the origin of the `enable_home` config flag that needs to be implemented for every storage driver. -We can implement [ListStorageSpaces](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ListStorageSpacesRequest) by either -- iterating over the root of the storage and treating every folder following the `` as a `home` *storage space*, -- iterating over the root of the storage and treating every folder following a new `` as a `project` *storage space*, or -- iterating over the root of the storage and treating every folder following a generic `` as a *storage space* for a configurable space type, or -- we allow configuring a map of `space type` to `layout` (based on the [CreateStorageSpaceRequest](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.CreateStorageSpaceRequest)) which would allow things like -``` -home=/var/lib/ocis/storage/home/{{substr 0 1 .Owner.Username}}/{{.Owner.Username}} -spaces=/spaces/var/lib/ocis/storage/projects/{{.Name}} -``` +In order to have a single source of truth we need to make the *share manager* aware of the mount point. We can then move all the logic that aggregates the etag in the share folder to a dedicated *shares storage provider* that is using the *share manager* for persistence. The *shares storage provider* would provide a `/shares` namespace outside of `/home` that lists all accepted shares for the current user. As a result the storage drivers no longer need to have a `enable_home` flag that jails users into their home. The `/home/shares` folder would move outside of the `/home`. In fact `/home` will no longer be needed, because the home folder concept can be implemented as a space: `CreateHome` would create a `personal` space on the. -This would make the `GetHome()` call return the path to the *storage provider* including the relative path to the *storage space*. No need for a *storage provider* mounted at `/home`. This is just a UI alias for `/users/`. Just like a normal `/home/` on a linux machine. +Work on this is done in https://github.com/cs3org/reva/pull/2023 -But if we have no `/home` where do we find the shares, and how can clients discover changes in accepted shares? - -The `/shares` namespace should be provided by a *shares storage provider* that lists all accepted shares for the current user... but what about copy pasting links from the browser? Well this storage is only really needed to have a path to ocm shares that actually reside on other instances. In the UI the shares would be listed by querying a *share manager*. It returns ResourceIds, which can be stated to fetch a path that is then accessible in the CS3 global namespace. Two caveats: +{{< hint warning >}} +What about copy pasting links from the browser? Well this storage is only really needed to have a path to ocm shares that actually reside on other instances. In the UI the shares would be listed by querying a *share manager*. It returns ResourceIds, which can be stated to fetch a path that is then accessible in the CS3 global namespace. Two caveats: - This only works for resources that are actually hosted by the current instance. For those it would leak the parent path segments to a shared resource. - For accepted OCM shares there must be a path in the [*CS3 global namespace*]({{< ref "./namespaces.md#cs3-global-namespaces" >}}) that has to be the same for all users, otherwise they cannot copy and share those URLs. - -Work on this is done in https://github.com/cs3org/reva/pull/1846 +{{< /hint >}} ### The gateway should be responsible for path transformations @@ -47,9 +39,9 @@ Work is done in https://github.com/cs3org/reva/pull/1866 ## URL escaped string representation of a CS3 reference -For the `/dav/spaces/` endpoint we need to encode the *reference* in a url compatible way. +For the spaces concept we introduced the `/dav/spaces/` endpoint. It encodes a cs3 *reference* in a URL compatible way. 1. We can separate the path using a `/`: `/dav/spaces//` -2. The `spaceid` currently is a cs3 resourceid, consisting of `` and ``. Since the nodeid might contain `/` eg. for the local driver we have to urlencode the spaceid. +2. The `spaceid` currently is a cs3 resourceid, consisting of `` and ``. Since the opaqueid might contain `/` eg. for the local driver we have to urlencode the spaceid. To access resources by id we need to make the `/dav/meta/` able to list directories... Otherwise id based navigation first has to look up the path. Or we use the libregraph api for id based navigation. diff --git a/docs/extensions/storage/spaces.md b/docs/extensions/storage/spaces.md index f1342b4074..3f87f3dd96 100644 --- a/docs/extensions/storage/spaces.md +++ b/docs/extensions/storage/spaces.md @@ -26,3 +26,14 @@ Finally, a logical `storage space id` is not tied to a specific [*spaces provide ## Shares *To be clarified: we are aware that [*storage spaces*]({{< ref "#storage-spaces" >}}) may be too 'heavywheight' for ad hoc sharing with groups. That being said, there is no technical reason why group shares should not be treated like storage [*spaces*]({{< ref "#storage-spaces" >}}) that users can provision themselves. They would share the quota with the users home or personal storage [*space*]({{< ref "#storage-spaces" >}}) and the share initiator would be the sole owner. Technically, the mechanism of treating a share like a new storage [*space*]({{< ref "#storage-spaces" >}}) would be the same. This obviously also extends to user shares and even file individual shares that would be wrapped in a virtual collection. It would also become possible to share collections of arbitrary files in a single storage space, e.g. the ten best pictures from a large album.* +## Notes + +We can implement [ListStorageSpaces](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ListStorageSpacesRequest) by either +- iterating over the root of the storage and treating every folder following the `` as a `home` *storage space*, +- iterating over the root of the storage and treating every folder following a new `` as a `project` *storage space*, or +- iterating over the root of the storage and treating every folder following a generic `` as a *storage space* for a configurable space type, or +- we allow configuring a map of `space type` to `layout` (based on the [CreateStorageSpaceRequest](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.CreateStorageSpaceRequest)) which would allow things like +``` +home=/var/lib/ocis/storage/home/{{substr 0 1 .Owner.Username}}/{{.Owner.Username}} +spaces=/spaces/var/lib/ocis/storage/projects/{{.Name}} +``` diff --git a/docs/extensions/storage/spacesprovider.md b/docs/extensions/storage/spacesprovider.md index 2100a3fa47..a30c723780 100644 --- a/docs/extensions/storage/spacesprovider.md +++ b/docs/extensions/storage/spacesprovider.md @@ -49,7 +49,7 @@ The ocdav service not only handles all WebDAV requests under `(remote.php/)(web) | *Note: existing folder sync pairs in legacy clients will break when moving the user home down in the path hierarchy* ||||| | `(remote.php/)webdav/home` | ocdav | storageprovider | `/home` | | | | `(remote.php/)webdav/users` | ocdav | storageprovider | `/users` | | | -| `(remote.php/)dav/files/` | ocdav | storageprovider | `/users/` | | | +| `(remote.php/)dav/files/` | ocdav | storageprovider | `/users/` | | | | *Spaces concept also needs a new endpoint:* ||||| | `(remote.php/)dav/spaces//` | ocdav | storageregistry & storageprovider | bypass path based namespace and directly talk to the responsible storage provider using a relative path | [spaces concept](https://github.com/owncloud/ocis/pull/1827) needs to point to storage [*spaces*]({{< ref "./spaces.md" >}}) | allow accessing spaces, listing is done by the graph api | diff --git a/docs/extensions/storage/static/namespaces.drawio.svg b/docs/extensions/storage/static/namespaces.drawio.svg index 5440f46b43..b3baa5895a 100644 --- a/docs/extensions/storage/static/namespaces.drawio.svg +++ b/docs/extensions/storage/static/namespaces.drawio.svg @@ -1,4 +1,4 @@ - + @@ -35,7 +35,7 @@ /home
- /users/<userlayout> + /users/<user_layout>
/public @@ -52,13 +52,13 @@ - +
-
+
storage home
@@ -69,16 +69,15 @@ - - - - + + +
-
+
storageprovider
@@ -89,13 +88,13 @@ - +
-
+
dataprovider
@@ -106,13 +105,13 @@ - +
-
+
frontend
@@ -123,16 +122,15 @@ - - - - + + +
-
+
ocdav
@@ -143,13 +141,13 @@ - +
-
+
ocs
@@ -160,14 +158,14 @@ - - + +
-
+
/webdav
@@ -203,14 +201,14 @@ - - + +
-
+
/ocs/v1.php/apps/files_sharing/api/v1/shares @@ -223,13 +221,13 @@ - +
-
+
gateway
@@ -240,16 +238,15 @@ - - - - + + +
-
+
gateway
@@ -260,13 +257,13 @@ - +
-
+
authregistry
@@ -277,13 +274,13 @@ - +
-
+
storageregistry
@@ -294,15 +291,15 @@ - - - + + +
-
+
oc10 namespace
(all paths aere relative to the users home) @@ -315,13 +312,13 @@ - +
-
+
CS3 global namespace
diff --git a/docs/extensions/storage/users.md b/docs/extensions/storage/users.md index 8e5d034bad..ea92720f82 100644 --- a/docs/extensions/storage/users.md +++ b/docs/extensions/storage/users.md @@ -11,7 +11,7 @@ TODO add this to the storage overview? or is this a different part? That should ### User and Group provisioning -In oc10 users are identified by a username, which cannot change, because it is used as a foreign key in several tables. For oCIS we are internally identifying users by a UUID, while using the username in the WebDAV and OCS APIs for backwards compatability. To distinguish this in the URLs we are using `` instead of ``. You may have encountered ``, which refers to a template that can be configured to build several path segments by filling in user properties, e.g. the first character of the username (`{{substr 0 1 .Username}}/{{.Username}}`), the identity provider (`{{.Id.Idp}}/{{.Username}}`) or the email (`{{.Mail}}`) +In oc10 users are identified by a username, which cannot change, because it is used as a foreign key in several tables. For oCIS we are internally identifying users by a UUID, while using the username in the WebDAV and OCS APIs for backwards compatability. To distinguish this in the URLs we are using `` instead of ``. You may have encountered ``, which refers to a template that can be configured to build several path segments by filling in user properties, e.g. the first character of the username (`{{substr 0 1 .Username}}/{{.Username}}`), the identity provider (`{{.Id.Idp}}/{{.Username}}`) or the email (`{{.Mail}}`) {{< hint warning >}} Make no mistake, the [OCS Provisioning API](https://doc.owncloud.com/server/developer_manual/core/apis/provisioning-api.html) uses `userid` while it actually is the username, because it is what you use to login. From b7d4f2b3314d118ffaa302c37f0c7722abd54d4e Mon Sep 17 00:00:00 2001 From: Willy Kloucek <34452982+wkloucek@users.noreply.github.com> Date: Tue, 7 Dec 2021 12:19:55 +0000 Subject: [PATCH 09/56] Automated changelog update [skip ci] --- CHANGELOG.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4094acfe47..3514a787a4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ The following sections list the changes for unreleased. * Bugfix - Fix claim selector based routing for basic auth: [#2779](https://github.com/owncloud/ocis/pull/2779) * Bugfix - Disallow creation of a group with empty name via the OCS api: [#2825](https://github.com/owncloud/ocis/pull/2825) * Bugfix - Fix using s3ng as the metadata storage backend: [#2807](https://github.com/owncloud/ocis/pull/2807) +* Bugfix - Use the CS3api up- and download workflow for the accounts service: [#2837](https://github.com/owncloud/ocis/pull/2837) * Change - Rename `APP_PROVIDER_BASIC_*` environment variables: [#2812](https://github.com/owncloud/ocis/pull/2812) * Change - Restructure Configuration Parsing: [#2708](https://github.com/owncloud/ocis/pull/2708) * Change - OIDC: fallback if IDP doesn't provide "preferred_username" claim: [#2644](https://github.com/owncloud/ocis/issues/2644) @@ -46,6 +47,15 @@ The following sections list the changes for unreleased. https://github.com/owncloud/ocis/issues/2668 https://github.com/owncloud/ocis/pull/2807 +* Bugfix - Use the CS3api up- and download workflow for the accounts service: [#2837](https://github.com/owncloud/ocis/pull/2837) + + We've fixed the interaction of the accounts service with the metadata storage after bypassing + the InitiateUpload and InitiateDownload have been removed from various storage drivers. The + accounts service now uses the proper CS3apis workflow for up- and downloads. + + https://github.com/owncloud/ocis/pull/2837 + https://github.com/cs3org/reva/pull/2309 + * Change - Rename `APP_PROVIDER_BASIC_*` environment variables: [#2812](https://github.com/owncloud/ocis/pull/2812) We've renamed the `APP_PROVIDER_BASIC_*` to `APP_PROVIDER_*` since the `_BASIC_` part is a @@ -89,6 +99,7 @@ The following sections list the changes for unreleased. * #TODO: update this changelog before the next oCIS release https://github.com/owncloud/ocis/pull/2835 + https://github.com/owncloud/ocis/pull/2837 # Changelog for [1.15.0] (2021-11-19) The following sections list the changes for 1.15.0. From 0e1d9b990f266700d02c8d4a1b1cac84e13dd928 Mon Sep 17 00:00:00 2001 From: Pascal Wengerter Date: Wed, 8 Dec 2021 12:23:28 +0100 Subject: [PATCH 10/56] [full-ci] Bump web to v4.6.0 (#2846) * Bump web to v4.6.0 * [full-ci]Bump web commitic in CI * Start middleware in accounts & settings CI pipelines, update expected failures Co-authored-by: Dipak Acharya --- .drone.env | 2 +- .drone.star | 32 +++++++++++++++++-- changelog/unreleased/update-web-4.6.0.md | 8 +++++ ...expected-failures-webUI-on-OCIS-storage.md | 16 ++++++---- web/Makefile | 2 +- 5 files changed, 49 insertions(+), 11 deletions(-) create mode 100644 changelog/unreleased/update-web-4.6.0.md diff --git a/.drone.env b/.drone.env index 109fb7a772..04c78573c2 100644 --- a/.drone.env +++ b/.drone.env @@ -3,5 +3,5 @@ CORE_COMMITID=4bbd91de68aefdf94c03d2920a3353702f856280 CORE_BRANCH=acceptance-test-changes-waiting-2021-11 # The test runner source for UI tests -WEB_COMMITID=82e69d203bfb44a2a652a4c052fe7ce795bb1100 +WEB_COMMITID=ed4a8b32240b59666a6c2b162d95b50ff7191eb8 WEB_BRANCH=master diff --git a/.drone.star b/.drone.star index f718437f4d..b50e5611e4 100644 --- a/.drone.star +++ b/.drone.star @@ -594,6 +594,7 @@ def uiTestPipeline(ctx, filterTags, early_fail, runPart = 1, numberOfParts = 1, "RUN_PART": runPart, "DIVIDE_INTO_NUM_PARTS": numberOfParts, "EXPECTED_FAILURES_FILE": "/drone/src/tests/acceptance/expected-failures-webUI-on-%s-storage%s.md" % (storage.upper(), expectedFailuresFileFilterTags), + "MIDDLEWARE_HOST": "http://middleware:3000", }, "commands": [ ". /drone/src/.drone.env", @@ -615,7 +616,7 @@ def uiTestPipeline(ctx, filterTags, early_fail, runPart = 1, numberOfParts = 1, }], }, ] + failEarly(ctx, early_fail), - "services": selenium(), + "services": selenium() + middlewareService(), "volumes": [pipelineVolumeOC10Tests] + [{ "name": "uploads", @@ -658,6 +659,7 @@ def accountsUITests(ctx, storage = "ocis", accounts_hash_difficulty = 4): "NODE_TLS_REJECT_UNAUTHORIZED": 0, "WEB_PATH": "/srv/app/web", "FEATURE_PATH": "/drone/src/accounts/ui/tests/acceptance/features", + "MIDDLEWARE_HOST": "http://middleware:3000", }, "commands": [ ". /drone/src/.drone.env", @@ -678,7 +680,7 @@ def accountsUITests(ctx, storage = "ocis", accounts_hash_difficulty = 4): }], }, ] + failEarly(ctx, early_fail), - "services": selenium(), + "services": selenium() + middlewareService(), "volumes": [stepVolumeOC10Tests] + [{ "name": "uploads", @@ -721,6 +723,7 @@ def settingsUITests(ctx, storage = "ocis", accounts_hash_difficulty = 4): "NODE_TLS_REJECT_UNAUTHORIZED": 0, "WEB_PATH": "/srv/app/web", "FEATURE_PATH": "/drone/src/settings/ui/tests/acceptance/features", + "MIDDLEWARE_HOST": "http://middleware:3000", }, "commands": [ ". /drone/src/.drone.env", @@ -746,7 +749,7 @@ def settingsUITests(ctx, storage = "ocis", accounts_hash_difficulty = 4): "name": "redis", "image": "redis:6-alpine", }, - ] + selenium(), + ] + selenium() + middlewareService(), "volumes": [stepVolumeOC10Tests] + [{ "name": "uploads", @@ -1417,6 +1420,29 @@ def ocisServer(storage, accounts_hash_difficulty = 4, volumes = []): }, ] +def middlewareService(): + return [{ + "name": "middleware", + "image": "owncloud/owncloud-test-middleware", + "pull": "always", + "environment": { + "BACKEND_HOST": "https://ocis-server:9200", + "OCIS_REVA_DATA_ROOT": "/srv/app/tmp/ocis/storage/owncloud/", + "RUN_ON_OCIS": "true", + "HOST": "middleware", + "REMOTE_UPLOAD_DIR": "/uploads", + "NODE_TLS_REJECT_UNAUTHORIZED": "0", + "MIDDLEWARE_HOST": "middleware", + }, + "volumes": [{ + "name": "uploads", + "path": "/uploads", + }, { + "name": "gopath", + "path": "/srv/app", + }], + }] + def cloneCoreRepos(): return [ { diff --git a/changelog/unreleased/update-web-4.6.0.md b/changelog/unreleased/update-web-4.6.0.md new file mode 100644 index 0000000000..05cb15e9bb --- /dev/null +++ b/changelog/unreleased/update-web-4.6.0.md @@ -0,0 +1,8 @@ +Enhancement: Update ownCloud Web to v4.6.0 + +Tags: web + +We updated ownCloud Web to v4.6.0. Please refer to the changelog (linked) for details on the web release. + +https://github.com/owncloud/ocis/pull/2846 +https://github.com/owncloud/web/releases/tag/v4.6.0 diff --git a/tests/acceptance/expected-failures-webUI-on-OCIS-storage.md b/tests/acceptance/expected-failures-webUI-on-OCIS-storage.md index 30ca76a42f..6f471f4716 100644 --- a/tests/acceptance/expected-failures-webUI-on-OCIS-storage.md +++ b/tests/acceptance/expected-failures-webUI-on-OCIS-storage.md @@ -125,12 +125,12 @@ Other free text and markdown formatting can be used elsewhere in the document if - [webUIFilesCopy/copyPrivateLinks.feature:21](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIFilesCopy/copyPrivateLinks.feature#L21) ### [Versions Cleanup not available](https://github.com/owncloud/ocis/issues/1310) +- [webUIFilesActionMenu/versions.feature:37](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIFilesActionMenu/versions.feature#L37) - [webUIFilesActionMenu/versions.feature:48](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIFilesActionMenu/versions.feature#L48) -- [webUIFilesActionMenu/versions.feature:59](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIFilesActionMenu/versions.feature#L59) -- [webUIFilesActionMenu/versions.feature:74](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIFilesActionMenu/versions.feature#L74) +- [webUIFilesActionMenu/versions.feature:63](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIFilesActionMenu/versions.feature#L63) ### [Change the file content of a received shared file](https://github.com/owncloud/ocis/issues/2319) -- [webUIFilesActionMenu/versions.feature:88](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIFilesActionMenu/versions.feature#L88) +- [webUIFilesActionMenu/versions.feature:77](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIFilesActionMenu/versions.feature#L77) ### [No occ command in ocis](https://github.com/owncloud/ocis/issues/1317) - [webUIRestrictSharing/restrictReSharing.feature:23](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIRestrictSharing/restrictReSharing.feature#L23) @@ -145,6 +145,7 @@ Other free text and markdown formatting can be used elsewhere in the document if - [webUISharingAutocompletion/shareAutocompletion.feature:128](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingAutocompletion/shareAutocompletion.feature#L128) - [webUISharingAutocompletion/shareAutocompletion.feature:141](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingAutocompletion/shareAutocompletion.feature#L141) +### [No occ command in ocis](https://github.com/owncloud/ocis/issues/1317) ### [First request with a recreated user returns a 401 error](https://github.com/owncloud/ocis/issues/1675) - [webUISharingAutocompletion/shareAutocompletion.feature:76](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingAutocompletion/shareAutocompletion.feature#L76) - [webUISharingAutocompletion/shareAutocompletion.feature:115](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingAutocompletion/shareAutocompletion.feature#L115) @@ -215,6 +216,7 @@ Other free text and markdown formatting can be used elsewhere in the document if - [webUIFilesCopy/copy.feature:98](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIFilesCopy/copy.feature#L98) - [webUIMoveFilesFolders/moveFiles.feature:97](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIMoveFilesFolders/moveFiles.feature#L97) - [webUIMoveFilesFolders/moveFolders.feature:72](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIMoveFilesFolders/moveFolders.feature#L72) +- [webUIFilesActionMenu/versions.feature:90](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIFilesActionMenu/versions.feature#L90) ### [Accepting different shares with same filename from different users overwrites one file](https://github.com/owncloud/ocis/issues/713) - [webUISharingAcceptShares/acceptShares.feature:212](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingAcceptShares/acceptShares.feature#L212) @@ -228,6 +230,7 @@ Other free text and markdown formatting can be used elsewhere in the document if - [webUISharingFolderAdvancedPermissionsGroups/shareAdvancePermissionsGroup.feature:71](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingFolderAdvancedPermissionsGroups/shareAdvancePermissionsGroup.feature#L71) ### [Share Permissions](https://github.com/owncloud/ocis/issues/1277) +### [Deletion of a selected user/group as a collaborator has unusual behavior in UI](https://github.com/owncloud/web/issues/5857) - [webUISharingFilePermissionsGroups/sharePermissionsGroup.feature:62](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingFilePermissionsGroups/sharePermissionsGroup.feature#L62) - [webUISharingFilePermissionsGroups/sharePermissionsGroup.feature:63](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingFilePermissionsGroups/sharePermissionsGroup.feature#L63) - [webUISharingFilePermissionMultipleUsers/shareFileWithMultipleUsers.feature:63](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingFilePermissionMultipleUsers/shareFileWithMultipleUsers.feature#L63) @@ -240,6 +243,7 @@ Other free text and markdown formatting can be used elsewhere in the document if - [webUISharingFolderPermissionMultipleUsers/shareFolderWithMultipleUsers.feature:62](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingFolderPermissionMultipleUsers/shareFolderWithMultipleUsers.feature#L62) ### [Advanced Permissions role is not displayed](https://github.com/owncloud/ocis/issues/1922) +### [Deletion of a selected user/group as a collaborator has unusual behavior in UI](https://github.com/owncloud/web/issues/5857) - [webUISharingFilePermissionsGroups/sharePermissionsGroup.feature:64](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingFilePermissionsGroups/sharePermissionsGroup.feature#L64) - [webUISharingFilePermissionsGroups/sharePermissionsGroup.feature:65](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingFilePermissionsGroups/sharePermissionsGroup.feature#L65) - [webUISharingFilePermissionsGroups/sharePermissionsGroup.feature:66](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingFilePermissionsGroups/sharePermissionsGroup.feature#L66) @@ -392,12 +396,12 @@ Other free text and markdown formatting can be used elsewhere in the document if - [webUISharingInternalUsersShareWithPage/shareWithUsers.feature:187](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingInternalUsersShareWithPage/shareWithUsers.feature#L187) ### [Deletion of a recursive folder from trashbin is not possible](https://github.com/owncloud/product/issues/188) -- [webUITrashbinDelete/trashbinDelete.feature:105](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUITrashbinDelete/trashbinDelete.feature#L105) -- [webUITrashbinDelete/trashbinDelete.feature:85](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUITrashbinDelete/trashbinDelete.feature#L85) +- [webUITrashbinDelete/trashbinDelete.feature:86](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUITrashbinDelete/trashbinDelete.feature#L86) +- [webUITrashbinDelete/trashbinDelete.feature:72](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUITrashbinDelete/trashbinDelete.feature#L72) - [webUITrashbinDelete/trashbinDelete.feature:49](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUITrashbinDelete/trashbinDelete.feature#L49) ### [Tags page not implemented yet](https://github.com/owncloud/web/issues/5017) -- [webUIDeleteFilesFolders/deleteFilesFolders.feature:144](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIDeleteFilesFolders/deleteFilesFolders.feature#L144) +- [webUIDeleteFilesFolders/deleteFilesFolders.feature:131](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIDeleteFilesFolders/deleteFilesFolders.feature#L131) - [webUIFilesSearch/search.feature:63](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIFilesSearch/search.feature#L63) - [webUIFilesSearch/search.feature:71](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIFilesSearch/search.feature#L71) - [webUIFilesSearch/search.feature:84](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIFilesSearch/search.feature#L84) diff --git a/web/Makefile b/web/Makefile index faf3c5cca3..2d9c78e628 100644 --- a/web/Makefile +++ b/web/Makefile @@ -29,7 +29,7 @@ ci-go-generate: # CI runs ci-node-generate automatically before this target .PHONY: ci-node-generate ci-node-generate: pull-assets -WEB_ASSETS_VERSION = v4.5.0 +WEB_ASSETS_VERSION = v4.6.0 .PHONY: pull-assets pull-assets: From 68a6370821b74210e8723dfa1e6de485d9a86109 Mon Sep 17 00:00:00 2001 From: Pascal Wengerter Date: Wed, 8 Dec 2021 11:23:52 +0000 Subject: [PATCH 11/56] Automated changelog update [skip ci] --- CHANGELOG.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3514a787a4..48d57b187f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,7 @@ The following sections list the changes for unreleased. * Enhancement - Cleanup ocis-pkg config: [#2813](https://github.com/owncloud/ocis/pull/2813) * Enhancement - Correct shutdown of services under runtime: [#2843](https://github.com/owncloud/ocis/pull/2843) * Enhancement - Update REVA to v1.17.0: [#2835](https://github.com/owncloud/ocis/pull/2835) +* Enhancement - Update ownCloud Web to v4.6.0: [#2846](https://github.com/owncloud/ocis/pull/2846) ## Details @@ -100,6 +101,16 @@ The following sections list the changes for unreleased. https://github.com/owncloud/ocis/pull/2835 https://github.com/owncloud/ocis/pull/2837 + +* Enhancement - Update ownCloud Web to v4.6.0: [#2846](https://github.com/owncloud/ocis/pull/2846) + + Tags: web + + We updated ownCloud Web to v4.6.0. Please refer to the changelog (linked) for details on the web + release. + + https://github.com/owncloud/ocis/pull/2846 + https://github.com/owncloud/web/releases/tag/v4.6.0 # Changelog for [1.15.0] (2021-11-19) The following sections list the changes for 1.15.0. From 54de444c29dce1c954055ab4ea5d79661ca8bb40 Mon Sep 17 00:00:00 2001 From: Willy Kloucek Date: Wed, 8 Dec 2021 13:17:15 +0100 Subject: [PATCH 12/56] update REVA --- changelog/unreleased/update-reva.md | 1 + go.mod | 2 +- go.sum | 4 ++-- 3 files changed, 4 insertions(+), 3 deletions(-) diff --git a/changelog/unreleased/update-reva.md b/changelog/unreleased/update-reva.md index 810cb656a8..e12cfb644f 100644 --- a/changelog/unreleased/update-reva.md +++ b/changelog/unreleased/update-reva.md @@ -6,5 +6,6 @@ This update includes: * #TODO: update this changelog before the next oCIS release +https://github.com/owncloud/ocis/pull/2849 https://github.com/owncloud/ocis/pull/2835 https://github.com/owncloud/ocis/pull/2837 diff --git a/go.mod b/go.mod index 4d989478e7..983f1dcb91 100644 --- a/go.mod +++ b/go.mod @@ -19,7 +19,7 @@ require ( github.com/blevesearch/bleve/v2 v2.2.2 github.com/coreos/go-oidc/v3 v3.1.0 github.com/cs3org/go-cs3apis v0.0.0-20211104090126-8e972dca8304 - github.com/cs3org/reva v1.16.1-0.20211203225713-939768a1af06 + github.com/cs3org/reva v1.16.1-0.20211208085434-d9eefe293789 github.com/disintegration/imaging v1.6.2 github.com/glauth/glauth/v2 v2.0.0-20211021011345-ef3151c28733 github.com/go-chi/chi/v5 v5.0.7 diff --git a/go.sum b/go.sum index 28a54a77ad..cd777f1c59 100644 --- a/go.sum +++ b/go.sum @@ -299,8 +299,8 @@ github.com/crewjam/saml v0.4.5/go.mod h1:qCJQpUtZte9R1ZjUBcW8qtCNlinbO363ooNl02S github.com/cs3org/cato v0.0.0-20200828125504-e418fc54dd5e/go.mod h1:XJEZ3/EQuI3BXTp/6DUzFr850vlxq11I6satRtz0YQ4= github.com/cs3org/go-cs3apis v0.0.0-20211104090126-8e972dca8304 h1:e/nIPR518vyvrulo9goAZTtYD6gFfu/2/9MDe6mTGcw= github.com/cs3org/go-cs3apis v0.0.0-20211104090126-8e972dca8304/go.mod h1:UXha4TguuB52H14EMoSsCqDj7k8a/t7g4gVP+bgY5LY= -github.com/cs3org/reva v1.16.1-0.20211203225713-939768a1af06 h1:NP+Zvli7+9USaDss/+Ywk4KJ0H7n82UHZiU4V+x25I0= -github.com/cs3org/reva v1.16.1-0.20211203225713-939768a1af06/go.mod h1:3n/zVKsKTCL10Mwn2Nhtvn50gP5mA+933lQ2IYNAJso= +github.com/cs3org/reva v1.16.1-0.20211208085434-d9eefe293789 h1:AfPsKObKRsHcUig6hiFIQGNzixJNh4ljHoLG9S+m7Yg= +github.com/cs3org/reva v1.16.1-0.20211208085434-d9eefe293789/go.mod h1:3n/zVKsKTCL10Mwn2Nhtvn50gP5mA+933lQ2IYNAJso= github.com/cubewise-code/go-mime v0.0.0-20200519001935-8c5762b177d8 h1:Z9lwXumT5ACSmJ7WGnFl+OMLLjpz5uR2fyz7dC255FI= github.com/cubewise-code/go-mime v0.0.0-20200519001935-8c5762b177d8/go.mod h1:4abs/jPXcmJzYoYGF91JF9Uq9s/KL5n1jvFDix8KcqY= github.com/cyberdelia/templates v0.0.0-20141128023046-ca7fffd4298c/go.mod h1:GyV+0YP4qX0UQ7r2MoYZ+AvYDp12OF5yg4q8rGnyNh4= From 3c22e886e073b814b469c359f0a209ff762d1608 Mon Sep 17 00:00:00 2001 From: "A.Unger" Date: Thu, 9 Dec 2021 10:14:27 +0100 Subject: [PATCH 13/56] added more background information to the steps --- docs/ocis/development/profiling.md | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/docs/ocis/development/profiling.md b/docs/ocis/development/profiling.md index 22807c00a5..9afb747cda 100644 --- a/docs/ocis/development/profiling.md +++ b/docs/ocis/development/profiling.md @@ -14,19 +14,19 @@ geekdocFilePath: profiling.md - Go development kit of a [supported version](https://golang.org/doc/devel/release.html#policy). Follow [these instructions](http://golang.org/doc/code.html) to install the go tool and set up GOPATH. - -- Graphviz: http://www.graphviz.org/ - Optional, used to generate graphic visualizations of profiles +- Graphviz: http://www.graphviz.org/. Used to generate graphic visualizations of profiles, which this example setup does. The only way to enable the profiler currently is to explicitly select which areas to collect samples for. In order to do this, the following steps have to be followed. ## 1. Clone Reva +Reva is the reference implementation of the CS3 APIs that we use for our daily business between oCIS and its storages. It is in charge of accessing the storage, as well as managing shares. Because of this fact, the examples will modify code in this dependency. You can think of Reva as the framework we use in order to interface with different storage providers. + `git clone github.com/cs3org/reva` ## 2. Patch reva with the area that you want sampled. -For the purposes of this docs let's use the WebDAV `PROPFIND` path. +For the purposes of these docs let's use the WebDAV `PROPFIND` path. This patch is needed in order to have the WebDAV process reporting profiling traces to the `pprof`. ```diff diff --git a/internal/http/services/owncloud/ocdav/propfind.go b/internal/http/services/owncloud/ocdav/propfind.go @@ -60,12 +60,14 @@ index 0e9c99be..f271572f 100644 The previous patch will: 1. import `net/http/pprof`, which will register debug handlers in `DefaultServeMux`. -2. define a `init()` function that starts an http server with the previously registered handlers. +2. define a `init()` function that starts an HTTP server with the previously registered handlers. With everything running one should have access to http://localhost:1234/debug/pprof/ ## 3. Replace reva in oCIS go.mod with local version and build a new binary +In Go, the `go.mod` file controls the dependencies of your module. Because we patched an external library, Go provides with a mechanism to overwrite an existing dependency with one on your local machine, which we previously installed. + ```diff diff --git a/go.mod b/go.mod index 131d14d7b..9668c38e4 100644 @@ -85,6 +87,8 @@ Make sure to replace `github.com/cs3org/reva => path/to/your/reva` with the corr ## 4. Build a new ocis binary +Using the new dependency with the pprof patch. + From owncloud/ocis root: ```console @@ -97,11 +101,13 @@ $ make clean build From owncloud/ocis root: ```console -$ OCIS_LOG_PRETTY=true OCIS_LOG_COLOR=true ocis/bin/ocis server +$ ocis/bin/ocis server ``` ## 6. Run `pprof` +[Pprof](https://github.com/google/pprof) is a tool developed at Google. It is a tool for visualization and analysis of profiling data. It will take the reported profiled data from our server, and represent it in a meaningful manner. + ### Install pprof If `pprof` is not installed make sure to get it; one way of installing it is using the Go tools: @@ -124,6 +130,10 @@ Once the collection is done a browser tab will open with the result `svg`, looki For references on how to interpret this graph, [continue reading here](https://github.com/google/pprof/blob/master/doc/README.md#interpreting-the-callgraph). +## Room for improvement + +Because these docs are intended to be read by developers they are quite technical in content. Requiring the user to alter the code. This is done so that we do not include, or assume, third party dependencies such as Graphviz in our binary, making it heavier. Having said this, the profiler is only meant to be used in development + ## References - https://medium.com/swlh/go-profile-your-code-like-a-master-1505be38fdba From c1d01303d2bbbfd4208fae902bd8d2da83957af9 Mon Sep 17 00:00:00 2001 From: Willy Kloucek Date: Thu, 9 Dec 2021 15:36:18 +0100 Subject: [PATCH 14/56] update REVA to v1.17.0 --- changelog/unreleased/update-reva.md | 15 ++++++++++++++- go.mod | 8 ++++---- go.sum | 16 ++++++++-------- 3 files changed, 26 insertions(+), 13 deletions(-) diff --git a/changelog/unreleased/update-reva.md b/changelog/unreleased/update-reva.md index e12cfb644f..0f3bd354a3 100644 --- a/changelog/unreleased/update-reva.md +++ b/changelog/unreleased/update-reva.md @@ -3,7 +3,20 @@ Enhancement: Update REVA to v1.17.0 Updated REVA to v1.17.0 This update includes: - * #TODO: update this changelog before the next oCIS release + * Fix [cs3org/reva#2305](https://github.com/cs3org/reva/pull/2305): Make sure /app/new takes `target` as absolute path + * Fix [cs3org/reva#2303](https://github.com/cs3org/reva/pull/2303): Fix content disposition header for public links files + * Fix [cs3org/reva#2316](https://github.com/cs3org/reva/pull/2316): Fix the share types in propfinds + * Fix [cs3org/reva#2803](https://github.com/cs3org/reva/pull/2803): Fix app provider for editor public links + * Fix [cs3org/reva#2298](https://github.com/cs3org/reva/pull/2298): Remove share refs from trashbin + * Fix [cs3org/reva#2309](https://github.com/cs3org/reva/pull/2309): Remove early finish for zero byte file uploads + * Fix [cs3org/reva#1941](https://github.com/cs3org/reva/pull/1941): Fix TUS uploads with transfer token only + * Chg [cs3org/reva#2210](https://github.com/cs3org/reva/pull/2210): Fix app provider new file creation and improved error codes + * Enh [cs3org/reva#2217](https://github.com/cs3org/reva/pull/2217): OIDC auth driver for ESCAPE IAM + * Enh [cs3org/reva#2256](https://github.com/cs3org/reva/pull/2256): Return user type in the response of the ocs GET user call + * Enh [cs3org/reva#2315](https://github.com/cs3org/reva/pull/2315): Add new attributes to public link propfinds + * Enh [cs3org/reva#2740](https://github.com/cs3org/reva/pull/2740): Implement space membership endpoints + * Enh [cs3org/reva#2252](https://github.com/cs3org/reva/pull/2252): Add the xattr sys.acl to SysACL (eosgrpc) + * Enh [cs3org/reva#2314](https://github.com/cs3org/reva/pull/2314): OIDC: fallback if IDP doesn't provide "preferred_username" claim https://github.com/owncloud/ocis/pull/2849 diff --git a/go.mod b/go.mod index 983f1dcb91..55aed36748 100644 --- a/go.mod +++ b/go.mod @@ -19,7 +19,7 @@ require ( github.com/blevesearch/bleve/v2 v2.2.2 github.com/coreos/go-oidc/v3 v3.1.0 github.com/cs3org/go-cs3apis v0.0.0-20211104090126-8e972dca8304 - github.com/cs3org/reva v1.16.1-0.20211208085434-d9eefe293789 + github.com/cs3org/reva v1.17.0 github.com/disintegration/imaging v1.6.2 github.com/glauth/glauth/v2 v2.0.0-20211021011345-ef3151c28733 github.com/go-chi/chi/v5 v5.0.7 @@ -38,7 +38,7 @@ require ( github.com/justinas/alice v1.2.0 github.com/libregraph/lico v0.53.0 github.com/mennanov/fieldmask-utils v0.5.0 - github.com/mitchellh/mapstructure v1.4.2 + github.com/mitchellh/mapstructure v1.4.3 github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 github.com/nmcclain/asn1-ber v0.0.0-20170104154839-2661553a0484 github.com/nmcclain/ldap v0.0.0-20210720162743-7f8d1e44eeba @@ -87,7 +87,7 @@ require ( github.com/RoaringBitmap/roaring v0.9.4 // indirect github.com/acomagu/bufpipe v1.0.3 // indirect github.com/asaskevich/govalidator v0.0.0-20210307081110-f21760c49a8d // indirect - github.com/aws/aws-sdk-go v1.42.9 // indirect + github.com/aws/aws-sdk-go v1.42.19 // indirect github.com/beevik/etree v1.1.0 // indirect github.com/beorn7/perks v1.0.1 // indirect github.com/bitly/go-simplejson v0.5.0 // indirect @@ -140,7 +140,7 @@ require ( github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect github.com/golang/mock v1.6.0 // indirect github.com/golang/snappy v0.0.4 // indirect - github.com/gomodule/redigo v1.8.5 // indirect + github.com/gomodule/redigo v1.8.6 // indirect github.com/google/go-cmp v0.5.6 // indirect github.com/google/go-querystring v1.1.0 // indirect github.com/google/uuid v1.3.0 // indirect diff --git a/go.sum b/go.sum index cd777f1c59..3f125c24d7 100644 --- a/go.sum +++ b/go.sum @@ -169,8 +169,8 @@ github.com/aws/aws-sdk-go v1.34.28/go.mod h1:H7NKnBqNVzoTJpGfLrQkkD+ytBA93eiDYi/ github.com/aws/aws-sdk-go v1.37.27/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro= github.com/aws/aws-sdk-go v1.38.35/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro= github.com/aws/aws-sdk-go v1.40.11/go.mod h1:585smgzpB/KqRA+K3y/NL/oYRqQvpNJYvLm+LY1U59Q= -github.com/aws/aws-sdk-go v1.42.9 h1:8ptAGgA+uC2TUbdvUeOVSfBocIZvGE2NKiLxkAcn1GA= -github.com/aws/aws-sdk-go v1.42.9/go.mod h1:585smgzpB/KqRA+K3y/NL/oYRqQvpNJYvLm+LY1U59Q= +github.com/aws/aws-sdk-go v1.42.19 h1:L/aM1QwsqVia9qIqexTHwYN+lgLYuOtf11VDgz0YIyw= +github.com/aws/aws-sdk-go v1.42.19/go.mod h1:585smgzpB/KqRA+K3y/NL/oYRqQvpNJYvLm+LY1U59Q= github.com/beevik/etree v1.1.0 h1:T0xke/WvNtMoCqgzPhkX2r4rjY3GDZFi+FjpRZY2Jbs= github.com/beevik/etree v1.1.0/go.mod h1:r8Aw8JqVegEf0w2fDnATrX9VpkMcyFeM0FhwO62wh+A= github.com/benbjohnson/clock v1.1.0 h1:Q92kusRqC1XV2MjkWETPvjJVqKetz1OzxZB7mHJLju8= @@ -299,8 +299,8 @@ github.com/crewjam/saml v0.4.5/go.mod h1:qCJQpUtZte9R1ZjUBcW8qtCNlinbO363ooNl02S github.com/cs3org/cato v0.0.0-20200828125504-e418fc54dd5e/go.mod h1:XJEZ3/EQuI3BXTp/6DUzFr850vlxq11I6satRtz0YQ4= github.com/cs3org/go-cs3apis v0.0.0-20211104090126-8e972dca8304 h1:e/nIPR518vyvrulo9goAZTtYD6gFfu/2/9MDe6mTGcw= github.com/cs3org/go-cs3apis v0.0.0-20211104090126-8e972dca8304/go.mod h1:UXha4TguuB52H14EMoSsCqDj7k8a/t7g4gVP+bgY5LY= -github.com/cs3org/reva v1.16.1-0.20211208085434-d9eefe293789 h1:AfPsKObKRsHcUig6hiFIQGNzixJNh4ljHoLG9S+m7Yg= -github.com/cs3org/reva v1.16.1-0.20211208085434-d9eefe293789/go.mod h1:3n/zVKsKTCL10Mwn2Nhtvn50gP5mA+933lQ2IYNAJso= +github.com/cs3org/reva v1.17.0 h1:UVdiiK6gUF2pb7qN9TKhMuf55sUKVnCcVjiVSzodluw= +github.com/cs3org/reva v1.17.0/go.mod h1:gtsVzMfDrUiUjH6qlHx+QqiRKsSYjVO6wEcCzANiqUg= github.com/cubewise-code/go-mime v0.0.0-20200519001935-8c5762b177d8 h1:Z9lwXumT5ACSmJ7WGnFl+OMLLjpz5uR2fyz7dC255FI= github.com/cubewise-code/go-mime v0.0.0-20200519001935-8c5762b177d8/go.mod h1:4abs/jPXcmJzYoYGF91JF9Uq9s/KL5n1jvFDix8KcqY= github.com/cyberdelia/templates v0.0.0-20141128023046-ca7fffd4298c/go.mod h1:GyV+0YP4qX0UQ7r2MoYZ+AvYDp12OF5yg4q8rGnyNh4= @@ -602,8 +602,8 @@ github.com/golang/snappy v0.0.3/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEW github.com/golang/snappy v0.0.4 h1:yAGX7huGHXlcLOEtBnF4w7FQwA26wojNCwOYAEhLjQM= github.com/golang/snappy v0.0.4/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= github.com/golangci/lint-1 v0.0.0-20181222135242-d2cdd8c08219/go.mod h1:/X8TswGSh1pIozq4ZwCfxS0WA5JGXguxk94ar/4c87Y= -github.com/gomodule/redigo v1.8.5 h1:nRAxCa+SVsyjSBrtZmG/cqb6VbTmuRzpg/PoTFlpumc= -github.com/gomodule/redigo v1.8.5/go.mod h1:P9dn9mFrCBvWhGE1wpxx6fgq7BAeLBk+UUUzlpkBYO0= +github.com/gomodule/redigo v1.8.6 h1:h7kHSqUl2kxeaQtVslsfUCPJ1oz2pxcyzLy4zezIzPw= +github.com/gomodule/redigo v1.8.6/go.mod h1:P9dn9mFrCBvWhGE1wpxx6fgq7BAeLBk+UUUzlpkBYO0= github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= @@ -922,8 +922,8 @@ github.com/mitchellh/mapstructure v1.3.2/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RR github.com/mitchellh/mapstructure v1.3.3/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo= github.com/mitchellh/mapstructure v1.4.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo= github.com/mitchellh/mapstructure v1.4.1/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo= -github.com/mitchellh/mapstructure v1.4.2 h1:6h7AQ0yhTcIsmFmnAwQls75jp2Gzs4iB8W7pjMO+rqo= -github.com/mitchellh/mapstructure v1.4.2/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo= +github.com/mitchellh/mapstructure v1.4.3 h1:OVowDSCllw/YjdLkam3/sm7wEtOy59d8ndGgCcyj8cs= +github.com/mitchellh/mapstructure v1.4.3/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo= github.com/mitchellh/reflectwalk v1.0.2 h1:G2LzWKi524PWgd3mLHV8Y5k7s6XUvT0Gef6zxSIeXaQ= github.com/mitchellh/reflectwalk v1.0.2/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw= github.com/moby/sys/mount v0.2.0/go.mod h1:aAivFE2LB3W4bACsUXChRHQ0qKWsetY4Y9V7sxOougM= From 068078a47bac67bd016fd57b85a1be5d18d3576c Mon Sep 17 00:00:00 2001 From: Willy Kloucek Date: Fri, 29 Oct 2021 14:04:24 +0200 Subject: [PATCH 15/56] add new file action to app provider docs --- docs/extensions/storage/apps.md | 54 +++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/docs/extensions/storage/apps.md b/docs/extensions/storage/apps.md index 19826cf485..9e0cb865cd 100644 --- a/docs/extensions/storage/apps.md +++ b/docs/extensions/storage/apps.md @@ -271,6 +271,60 @@ There are apps, which need to be opened in the iframe with a GET request. The GE } ``` +### Creating a file with the app provider + +**Endpoint**: specified in the capabilities in `new_file_url`, currently `/app/new` + +**Method**: HTTP POST + +**Authentication** (one of them): + +- `Authorization` header with OIDC Bearer token for authenticated users or basic auth credentials (if enabled in oCIS) +- `Public-Token` header with public link token for public links +- `X-Access-Token` header with a REVA token for authenticated users + +**Query parameters**: + +- `filename` (mandatory): path / name of the new file +- `template` (optional): not yet implemented + +**Request examples**: + +```bash +curl -X POST 'https://ocis.test/app/new?filename=/home/test.odt' +``` + +**Response example**: + +You will receive a file id of the freshly created file, which you can use to open the file in an editor. + +```json +{ + "file_id": "ZmlsZTppZAo=" +} +``` + +**Example responses (error case)**: + +- #TODO: file already exists, currently existing files will be overwritten + + ```json + { + "code": "ERROR", + "message": "file already exists" + } + ``` + +- unauthorized / failed to create the file + + ```json + { + "code": "SERVER_ERROR", + "message": "error creating resource" + } + ``` + + ## App drivers App drivers represent apps, if the app is not able to register itself. Currently there is only the CS3org WOPI server app driver. From a713c90841510b59cb1c00005eb6e1f7001d93ea Mon Sep 17 00:00:00 2001 From: Willy Kloucek Date: Thu, 4 Nov 2021 17:06:09 +0100 Subject: [PATCH 16/56] adapt to changes of cs3org/reva#2210 --- docs/extensions/storage/apps.md | 33 +++++++++++++++++++++++++-------- 1 file changed, 25 insertions(+), 8 deletions(-) diff --git a/docs/extensions/storage/apps.md b/docs/extensions/storage/apps.md index 9e0cb865cd..db7d873622 100644 --- a/docs/extensions/storage/apps.md +++ b/docs/extensions/storage/apps.md @@ -285,13 +285,14 @@ There are apps, which need to be opened in the iframe with a GET request. The GE **Query parameters**: -- `filename` (mandatory): path / name of the new file +- `parent_container_id` (mandatory): ID of the folder in which the file will be created +- `filename` (mandatory): name of the new file - `template` (optional): not yet implemented **Request examples**: ```bash -curl -X POST 'https://ocis.test/app/new?filename=/home/test.odt' +curl -X POST 'https://ocis.test/app/new?parent_container_id=c2lkOmNpZAo=&filename=test.odt' ``` **Response example**: @@ -306,21 +307,37 @@ You will receive a file id of the freshly created file, which you can use to ope **Example responses (error case)**: -- #TODO: file already exists, currently existing files will be overwritten - +- missing parent folder ID ```json { - "code": "ERROR", - "message": "file already exists" + "code": "INVALID_PARAMETER", + "message": "Missing parent container ID" } ``` -- unauthorized / failed to create the file +- missing file name + ```json + { + "code": "INVALID_PARAMETER", + "message": "Missing filename" + } + ``` + +- file already exists ```json { "code": "SERVER_ERROR", - "message": "error creating resource" + "message": "The file already exists" + } + ``` + +- unauthorized / failed to find the parent folder + + ```json + { + "code": "RESOURCE_NOT_FOUND", + "message": "The parent container is not accessible or does not exist" } ``` From 34b2971cb9b06e3e29feaea27e798eb4b7767785 Mon Sep 17 00:00:00 2001 From: Willy Kloucek Date: Thu, 4 Nov 2021 17:10:19 +0100 Subject: [PATCH 17/56] add default_application from cs3org/reva#2230 --- docs/extensions/storage/apps.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/extensions/storage/apps.md b/docs/extensions/storage/apps.md index db7d873622..eeed7955a9 100644 --- a/docs/extensions/storage/apps.md +++ b/docs/extensions/storage/apps.md @@ -132,7 +132,8 @@ curl 'https://ocis.test/app/list' "name": "OpenDocument", "icon": "https://some-website.test/opendocument-text-icon.png", "description": "OpenDocument text document", - "allow_creation": true + "allow_creation": true, + "default_application": "Collabora" }, { "mime_type": "text/markdown", @@ -145,7 +146,8 @@ curl 'https://ocis.test/app/list' ], "name": "Markdown file", "description": "Markdown file", - "allow_creation": true + "allow_creation": true, + "default_application": "CodiMD" }, { "mime_type": "application/vnd.ms-word.document.macroenabled.12", From 11a98997356e0e6b56b716a20d79167c9cb7be00 Mon Sep 17 00:00:00 2001 From: Willy Kloucek Date: Fri, 5 Nov 2021 12:44:15 +0100 Subject: [PATCH 18/56] add https status codes --- docs/extensions/storage/apps.md | 121 ++++++++++++++++++++++++++------ 1 file changed, 99 insertions(+), 22 deletions(-) diff --git a/docs/extensions/storage/apps.md b/docs/extensions/storage/apps.md index eeed7955a9..aa7247828c 100644 --- a/docs/extensions/storage/apps.md +++ b/docs/extensions/storage/apps.md @@ -101,6 +101,8 @@ curl 'https://ocis.test/app/list' **Response example**: +HTTP status code: 200 + ```json { "mime-types": [ @@ -218,6 +220,8 @@ All apps are expected to be opened in an iframe and the response will give some There are apps, which need to be opened in the iframe with a form post. The form post must include all form parameters included in the response. For these apps the response will look like this: +HTTP status code: 200 + ```json { "app_url": "https://.....", @@ -232,6 +236,8 @@ There are apps, which need to be opened in the iframe with a form post. The form There are apps, which need to be opened in the iframe with a GET request. The GET request must have set all headers included in the response. For these apps the response will look like this: +HTTP status code: 200 + ```json { "app_url": "https://...", @@ -246,30 +252,69 @@ There are apps, which need to be opened in the iframe with a GET request. The GE **Example responses (error case)**: -- wrong `view_mode` +- missing `file_id` + + HTTP status code: 400 ```json { - "code": "SERVER_ERROR", - "message": "Missing or invalid viewmode argument" + "code": "INVALID_PARAMETER", + "message": "missing file ID" + } + ``` + +- wrong `view_mode` + + HTTP status code: 400 + + ```json + { + "code": "INVALID_PARAMETER", + "message": "invalid view mode" } ``` - unknown `app_name` + HTTP status code: 404 + ```json { - "code": "SERVER_ERROR", - "message": "error searching for app provider" + "code": "RESOURCE_NOT_FOUND", + "message": "error: not found: app 'Collabor' not found" } ``` -- wrong / invalid file id / unauthorized to open the file +- wrong / invalid file id + + HTTP status code: 400 ```json { - "code": "SERVER_ERROR", - "message": "error statting file" + "code": "INVALID_PARAMETER", + "message": "invalid file ID" + } + ``` + +- file id does not point to a file + + HTTP status code: 400 + + ```json + { + "code": "INVALID_PARAMETER", + "message": "the given file id does not point to a file" + } + ``` + +- file does not exist / unauthorized to open the file + + HTTP status code: 404 + + ```json + { + "code": "RESOURCE_NOT_FOUND", + "message": "file does not exist" } ``` @@ -309,37 +354,69 @@ You will receive a file id of the freshly created file, which you can use to ope **Example responses (error case)**: -- missing parent folder ID +- missing `parent_container_id` + + HTTP status code: 400 + ```json { "code": "INVALID_PARAMETER", - "message": "Missing parent container ID" + "message": "missing parent container ID" } ``` -- missing file name +- missing `filename` + + HTTP status code: 400 + ```json { "code": "INVALID_PARAMETER", - "message": "Missing filename" + "message": "missing filename" + } + ``` + +- parent container not found + + HTTP status code: 404 + + ```json + { + "code": "RESOURCE_NOT_FOUND", + "message": "the parent container is not accessible or does not exist" + } + ``` + +- `parent_container_id` does not point to a container + + HTTP status code: 400 + + ```json + { + "code": "INVALID_PARAMETER", + "message": "the parent container id does not point to a container" + } + ``` + +- `filename` is invalid (eg. includes a path segment) + + HTTP status code: 400 + + ```json + { + "code": "INVALID_PARAMETER", + "message": "the filename must not contain a path segment" } ``` - file already exists - ```json - { - "code": "SERVER_ERROR", - "message": "The file already exists" - } - ``` - -- unauthorized / failed to find the parent folder + HTTP status code: 403 ```json { - "code": "RESOURCE_NOT_FOUND", - "message": "The parent container is not accessible or does not exist" + "code": "RESOURCE_ALREADY_EXISTS", + "message": "the file already exists" } ``` From a2d5de07b8c44c817eba4272f54af87bf4c02891 Mon Sep 17 00:00:00 2001 From: Phil Davis Date: Thu, 9 Dec 2021 17:13:58 +0545 Subject: [PATCH 19/56] [tests-only] wait for selenium and middleware docker images to start --- .drone.star | 29 ++++++++++++++++++++++++----- 1 file changed, 24 insertions(+), 5 deletions(-) diff --git a/.drone.star b/.drone.star index b50e5611e4..7d03e5571a 100644 --- a/.drone.star +++ b/.drone.star @@ -1,4 +1,4 @@ -"""oCIS CI defintion +"""oCIS CI definition """ # images @@ -6,6 +6,7 @@ OC_CI_ALPINE = "owncloudci/alpine:latest" OC_CI_GOLANG = "owncloudci/golang:1.17" OC_CI_NODEJS = "owncloudci/nodejs:14" OC_CI_PHP = "owncloudci/php:7.4" +OC_CI_WAIT_FOR = "owncloudci/wait-for:latest" MINIO_MC = "minio/mc:RELEASE.2021-10-07T04-19-58Z" # configuration @@ -577,7 +578,7 @@ def uiTestPipeline(ctx, filterTags, early_fail, runPart = 1, numberOfParts = 1, "arch": "amd64", }, "steps": skipIfUnchanged(ctx, "acceptance-tests") + restoreBuildArtifactCache(ctx, "ocis-binary-amd64", "ocis/bin/ocis") + - ocisServer(storage, accounts_hash_difficulty, [stepVolumeOC10Tests]) + [ + ocisServer(storage, accounts_hash_difficulty, [stepVolumeOC10Tests]) + waitForSeleniumService() + waitForMiddlewareService() + [ { "name": "webUITests", "image": OC_CI_NODEJS, @@ -644,7 +645,7 @@ def accountsUITests(ctx, storage = "ocis", accounts_hash_difficulty = 4): "arch": "amd64", }, "steps": skipIfUnchanged(ctx, "acceptance-tests") + restoreBuildArtifactCache(ctx, "ocis-binary-amd64", "ocis/bin/ocis") + - ocisServer(storage, accounts_hash_difficulty, [stepVolumeOC10Tests]) + [ + ocisServer(storage, accounts_hash_difficulty, [stepVolumeOC10Tests]) + waitForSeleniumService() + waitForMiddlewareService() + [ { "name": "WebUIAcceptanceTests", "image": OC_CI_NODEJS, @@ -708,7 +709,7 @@ def settingsUITests(ctx, storage = "ocis", accounts_hash_difficulty = 4): "arch": "amd64", }, "steps": skipIfUnchanged(ctx, "acceptance-tests") + restoreBuildArtifactCache(ctx, "ocis-binary-amd64", "ocis/bin/ocis") + - ocisServer(storage, accounts_hash_difficulty, [stepVolumeOC10Tests]) + [ + ocisServer(storage, accounts_hash_difficulty, [stepVolumeOC10Tests]) + waitForSeleniumService() + waitForMiddlewareService() + [ { "name": "WebUIAcceptanceTests", "image": OC_CI_NODEJS, @@ -785,7 +786,7 @@ def failEarly(ctx, early_fail): "image": "thegeeklab/drone-github-comment:1", "settings": { "message": ":boom: Acceptance test [${DRONE_STAGE_NAME}](${DRONE_BUILD_LINK}/${DRONE_STAGE_NUMBER}/1) failed. Further test are cancelled...", - "key": "pr-${DRONE_PULL_REQUEST}", #TODO: we could delete the comment after a successfull CI run + "key": "pr-${DRONE_PULL_REQUEST}", #TODO: we could delete the comment after a successful CI run "update": "true", "api_key": { "from_secret": "github_token", @@ -1443,6 +1444,15 @@ def middlewareService(): }], }] +def waitForMiddlewareService(): + return [{ + "name": "wait-for-middleware-service", + "image": OC_CI_WAIT_FOR, + "commands": [ + "wait-for -it middleware:3000 -t 300", + ], + }] + def cloneCoreRepos(): return [ { @@ -1485,6 +1495,15 @@ def selenium(): }, ] +def waitForSeleniumService(): + return [{ + "name": "wait-for-selenium-service", + "image": OC_CI_WAIT_FOR, + "commands": [ + "wait-for -it selenium:4444 -t 300", + ], + }] + def build(): return [ { From ef8726b5e92f0cc275515244e6aff6fa00605f0e Mon Sep 17 00:00:00 2001 From: Willy Kloucek Date: Thu, 9 Dec 2021 16:13:11 +0100 Subject: [PATCH 20/56] update expected failures --- tests/acceptance/expected-failures-webUI-on-OCIS-storage.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/tests/acceptance/expected-failures-webUI-on-OCIS-storage.md b/tests/acceptance/expected-failures-webUI-on-OCIS-storage.md index 6f471f4716..e187d962b3 100644 --- a/tests/acceptance/expected-failures-webUI-on-OCIS-storage.md +++ b/tests/acceptance/expected-failures-webUI-on-OCIS-storage.md @@ -114,8 +114,6 @@ Other free text and markdown formatting can be used elsewhere in the document if ### [share indicator in files are not shown until sharing sidebar is opened](https://github.com/owncloud/web/issues/4167) - [webUISharingInternalUsersSharingIndicator/shareWithUsers.feature:100](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingInternalUsersSharingIndicator/shareWithUsers.feature#L100) - [webUISharingInternalUsersSharingIndicator/shareWithUsers.feature:121](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingInternalUsersSharingIndicator/shareWithUsers.feature#L121) -- [webUISharingPublicManagement/publicLinkIndicator.feature:12](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingPublicManagement/publicLinkIndicator.feature#L12) -- [webUISharingPublicManagement/publicLinkIndicator.feature:47](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingPublicManagement/publicLinkIndicator.feature#L47) - [webUISharingPublicManagement/publicLinkIndicator.feature:64](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingPublicManagement/publicLinkIndicator.feature#L64) - [webUISharingPublicManagement/publicLinkIndicator.feature:81](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingPublicManagement/publicLinkIndicator.feature#L81) - [webUISharingPublicManagement/publicLinkIndicator.feature:98](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingPublicManagement/publicLinkIndicator.feature#L98) From f6c02716adeda437011339880097c1943f2ffa5f Mon Sep 17 00:00:00 2001 From: Willy Kloucek <34452982+wkloucek@users.noreply.github.com> Date: Thu, 9 Dec 2021 15:52:50 +0000 Subject: [PATCH 21/56] Automated changelog update [skip ci] --- CHANGELOG.md | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 48d57b187f..d52f3624c5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,7 +15,7 @@ The following sections list the changes for unreleased. * Change - OIDC: fallback if IDP doesn't provide "preferred_username" claim: [#2644](https://github.com/owncloud/ocis/issues/2644) * Enhancement - Cleanup ocis-pkg config: [#2813](https://github.com/owncloud/ocis/pull/2813) * Enhancement - Correct shutdown of services under runtime: [#2843](https://github.com/owncloud/ocis/pull/2843) -* Enhancement - Update REVA to v1.17.0: [#2835](https://github.com/owncloud/ocis/pull/2835) +* Enhancement - Update REVA to v1.17.0: [#2849](https://github.com/owncloud/ocis/pull/2849) * Enhancement - Update ownCloud Web to v4.6.0: [#2846](https://github.com/owncloud/ocis/pull/2846) ## Details @@ -93,12 +93,26 @@ The following sections list the changes for unreleased. https://github.com/owncloud/ocis/pull/2843 -* Enhancement - Update REVA to v1.17.0: [#2835](https://github.com/owncloud/ocis/pull/2835) +* Enhancement - Update REVA to v1.17.0: [#2849](https://github.com/owncloud/ocis/pull/2849) Updated REVA to v1.17.0 This update includes: - * #TODO: update this changelog before the next oCIS release + * Fix [cs3org/reva#2305](https://github.com/cs3org/reva/pull/2305): Make sure /app/new takes `target` as absolute path + * Fix [cs3org/reva#2303](https://github.com/cs3org/reva/pull/2303): Fix content disposition header for public links files + * Fix [cs3org/reva#2316](https://github.com/cs3org/reva/pull/2316): Fix the share types in propfinds + * Fix [cs3org/reva#2803](https://github.com/cs3org/reva/pull/2803): Fix app provider for editor public links + * Fix [cs3org/reva#2298](https://github.com/cs3org/reva/pull/2298): Remove share refs from trashbin + * Fix [cs3org/reva#2309](https://github.com/cs3org/reva/pull/2309): Remove early finish for zero byte file uploads + * Fix [cs3org/reva#1941](https://github.com/cs3org/reva/pull/1941): Fix TUS uploads with transfer token only + * Chg [cs3org/reva#2210](https://github.com/cs3org/reva/pull/2210): Fix app provider new file creation and improved error codes + * Enh [cs3org/reva#2217](https://github.com/cs3org/reva/pull/2217): OIDC auth driver for ESCAPE IAM + * Enh [cs3org/reva#2256](https://github.com/cs3org/reva/pull/2256): Return user type in the response of the ocs GET user call + * Enh [cs3org/reva#2315](https://github.com/cs3org/reva/pull/2315): Add new attributes to public link propfinds + * Enh [cs3org/reva#2740](https://github.com/cs3org/reva/pull/2740): Implement space membership endpoints + * Enh [cs3org/reva#2252](https://github.com/cs3org/reva/pull/2252): Add the xattr sys.acl to SysACL (eosgrpc) + * Enh [cs3org/reva#2314](https://github.com/cs3org/reva/pull/2314): OIDC: fallback if IDP doesn't provide "preferred_username" claim + https://github.com/owncloud/ocis/pull/2849 https://github.com/owncloud/ocis/pull/2835 https://github.com/owncloud/ocis/pull/2837 From e2d21edaa432883cddbbe0ea67f4e74d4dfc6308 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 9 Dec 2021 16:07:57 +0000 Subject: [PATCH 22/56] Bump github.com/golang-jwt/jwt/v4 from 4.1.0 to 4.2.0 Bumps [github.com/golang-jwt/jwt/v4](https://github.com/golang-jwt/jwt) from 4.1.0 to 4.2.0. - [Release notes](https://github.com/golang-jwt/jwt/releases) - [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md) - [Commits](https://github.com/golang-jwt/jwt/compare/v4.1.0...v4.2.0) --- updated-dependencies: - dependency-name: github.com/golang-jwt/jwt/v4 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- go.mod | 10 +++++----- go.sum | 19 ++++++++++--------- 2 files changed, 15 insertions(+), 14 deletions(-) diff --git a/go.mod b/go.mod index 4d989478e7..a2cd56ad20 100644 --- a/go.mod +++ b/go.mod @@ -19,7 +19,7 @@ require ( github.com/blevesearch/bleve/v2 v2.2.2 github.com/coreos/go-oidc/v3 v3.1.0 github.com/cs3org/go-cs3apis v0.0.0-20211104090126-8e972dca8304 - github.com/cs3org/reva v1.16.1-0.20211203225713-939768a1af06 + github.com/cs3org/reva v1.17.0 github.com/disintegration/imaging v1.6.2 github.com/glauth/glauth/v2 v2.0.0-20211021011345-ef3151c28733 github.com/go-chi/chi/v5 v5.0.7 @@ -29,7 +29,7 @@ require ( github.com/go-logr/logr v0.4.0 github.com/go-ozzo/ozzo-validation/v4 v4.3.0 github.com/gofrs/uuid v4.2.0+incompatible - github.com/golang-jwt/jwt/v4 v4.1.0 + github.com/golang-jwt/jwt/v4 v4.2.0 github.com/golang/protobuf v1.5.2 github.com/gookit/config/v2 v2.0.27 github.com/gorilla/mux v1.8.0 @@ -38,7 +38,7 @@ require ( github.com/justinas/alice v1.2.0 github.com/libregraph/lico v0.53.0 github.com/mennanov/fieldmask-utils v0.5.0 - github.com/mitchellh/mapstructure v1.4.2 + github.com/mitchellh/mapstructure v1.4.3 github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 github.com/nmcclain/asn1-ber v0.0.0-20170104154839-2661553a0484 github.com/nmcclain/ldap v0.0.0-20210720162743-7f8d1e44eeba @@ -87,7 +87,7 @@ require ( github.com/RoaringBitmap/roaring v0.9.4 // indirect github.com/acomagu/bufpipe v1.0.3 // indirect github.com/asaskevich/govalidator v0.0.0-20210307081110-f21760c49a8d // indirect - github.com/aws/aws-sdk-go v1.42.9 // indirect + github.com/aws/aws-sdk-go v1.42.19 // indirect github.com/beevik/etree v1.1.0 // indirect github.com/beorn7/perks v1.0.1 // indirect github.com/bitly/go-simplejson v0.5.0 // indirect @@ -140,7 +140,7 @@ require ( github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect github.com/golang/mock v1.6.0 // indirect github.com/golang/snappy v0.0.4 // indirect - github.com/gomodule/redigo v1.8.5 // indirect + github.com/gomodule/redigo v1.8.6 // indirect github.com/google/go-cmp v0.5.6 // indirect github.com/google/go-querystring v1.1.0 // indirect github.com/google/uuid v1.3.0 // indirect diff --git a/go.sum b/go.sum index 28a54a77ad..e8cb72438e 100644 --- a/go.sum +++ b/go.sum @@ -169,8 +169,8 @@ github.com/aws/aws-sdk-go v1.34.28/go.mod h1:H7NKnBqNVzoTJpGfLrQkkD+ytBA93eiDYi/ github.com/aws/aws-sdk-go v1.37.27/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro= github.com/aws/aws-sdk-go v1.38.35/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro= github.com/aws/aws-sdk-go v1.40.11/go.mod h1:585smgzpB/KqRA+K3y/NL/oYRqQvpNJYvLm+LY1U59Q= -github.com/aws/aws-sdk-go v1.42.9 h1:8ptAGgA+uC2TUbdvUeOVSfBocIZvGE2NKiLxkAcn1GA= -github.com/aws/aws-sdk-go v1.42.9/go.mod h1:585smgzpB/KqRA+K3y/NL/oYRqQvpNJYvLm+LY1U59Q= +github.com/aws/aws-sdk-go v1.42.19 h1:L/aM1QwsqVia9qIqexTHwYN+lgLYuOtf11VDgz0YIyw= +github.com/aws/aws-sdk-go v1.42.19/go.mod h1:585smgzpB/KqRA+K3y/NL/oYRqQvpNJYvLm+LY1U59Q= github.com/beevik/etree v1.1.0 h1:T0xke/WvNtMoCqgzPhkX2r4rjY3GDZFi+FjpRZY2Jbs= github.com/beevik/etree v1.1.0/go.mod h1:r8Aw8JqVegEf0w2fDnATrX9VpkMcyFeM0FhwO62wh+A= github.com/benbjohnson/clock v1.1.0 h1:Q92kusRqC1XV2MjkWETPvjJVqKetz1OzxZB7mHJLju8= @@ -299,8 +299,8 @@ github.com/crewjam/saml v0.4.5/go.mod h1:qCJQpUtZte9R1ZjUBcW8qtCNlinbO363ooNl02S github.com/cs3org/cato v0.0.0-20200828125504-e418fc54dd5e/go.mod h1:XJEZ3/EQuI3BXTp/6DUzFr850vlxq11I6satRtz0YQ4= github.com/cs3org/go-cs3apis v0.0.0-20211104090126-8e972dca8304 h1:e/nIPR518vyvrulo9goAZTtYD6gFfu/2/9MDe6mTGcw= github.com/cs3org/go-cs3apis v0.0.0-20211104090126-8e972dca8304/go.mod h1:UXha4TguuB52H14EMoSsCqDj7k8a/t7g4gVP+bgY5LY= -github.com/cs3org/reva v1.16.1-0.20211203225713-939768a1af06 h1:NP+Zvli7+9USaDss/+Ywk4KJ0H7n82UHZiU4V+x25I0= -github.com/cs3org/reva v1.16.1-0.20211203225713-939768a1af06/go.mod h1:3n/zVKsKTCL10Mwn2Nhtvn50gP5mA+933lQ2IYNAJso= +github.com/cs3org/reva v1.17.0 h1:UVdiiK6gUF2pb7qN9TKhMuf55sUKVnCcVjiVSzodluw= +github.com/cs3org/reva v1.17.0/go.mod h1:gtsVzMfDrUiUjH6qlHx+QqiRKsSYjVO6wEcCzANiqUg= github.com/cubewise-code/go-mime v0.0.0-20200519001935-8c5762b177d8 h1:Z9lwXumT5ACSmJ7WGnFl+OMLLjpz5uR2fyz7dC255FI= github.com/cubewise-code/go-mime v0.0.0-20200519001935-8c5762b177d8/go.mod h1:4abs/jPXcmJzYoYGF91JF9Uq9s/KL5n1jvFDix8KcqY= github.com/cyberdelia/templates v0.0.0-20141128023046-ca7fffd4298c/go.mod h1:GyV+0YP4qX0UQ7r2MoYZ+AvYDp12OF5yg4q8rGnyNh4= @@ -555,8 +555,9 @@ github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69 github.com/goji/httpauth v0.0.0-20160601135302-2da839ab0f4d/go.mod h1:nnjvkQ9ptGaCkuDUx6wNykzzlUixGxvkme+H/lnzb+A= github.com/golang-jwt/jwt v3.2.2+incompatible h1:IfV12K8xAKAnZqdXVzCZ+TOjboZ2keLg81eXfW3O+oY= github.com/golang-jwt/jwt v3.2.2+incompatible/go.mod h1:8pz2t5EyA70fFQQSrl6XZXzqecmYZeUEB8OUGHkxJ+I= -github.com/golang-jwt/jwt/v4 v4.1.0 h1:XUgk2Ex5veyVFVeLm0xhusUTQybEbexJXrvPNOKkSY0= github.com/golang-jwt/jwt/v4 v4.1.0/go.mod h1:/xlHOz8bRuivTWchD4jCa+NbatV+wEUSzwAxVc6locg= +github.com/golang-jwt/jwt/v4 v4.2.0 h1:besgBTC8w8HjP6NzQdxwKH9Z5oQMZ24ThTrHp3cZ8eU= +github.com/golang-jwt/jwt/v4 v4.2.0/go.mod h1:/xlHOz8bRuivTWchD4jCa+NbatV+wEUSzwAxVc6locg= github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0/go.mod h1:E/TSTwGwJL78qG/PmXZO1EjYhfJinVAhrmmHX6Z8B9k= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= github.com/golang/glog v1.0.0/go.mod h1:EWib/APOK0SL3dFbYqvxE3UYd8E6s1ouQ7iEp/0LWV4= @@ -602,8 +603,8 @@ github.com/golang/snappy v0.0.3/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEW github.com/golang/snappy v0.0.4 h1:yAGX7huGHXlcLOEtBnF4w7FQwA26wojNCwOYAEhLjQM= github.com/golang/snappy v0.0.4/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= github.com/golangci/lint-1 v0.0.0-20181222135242-d2cdd8c08219/go.mod h1:/X8TswGSh1pIozq4ZwCfxS0WA5JGXguxk94ar/4c87Y= -github.com/gomodule/redigo v1.8.5 h1:nRAxCa+SVsyjSBrtZmG/cqb6VbTmuRzpg/PoTFlpumc= -github.com/gomodule/redigo v1.8.5/go.mod h1:P9dn9mFrCBvWhGE1wpxx6fgq7BAeLBk+UUUzlpkBYO0= +github.com/gomodule/redigo v1.8.6 h1:h7kHSqUl2kxeaQtVslsfUCPJ1oz2pxcyzLy4zezIzPw= +github.com/gomodule/redigo v1.8.6/go.mod h1:P9dn9mFrCBvWhGE1wpxx6fgq7BAeLBk+UUUzlpkBYO0= github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= @@ -922,8 +923,8 @@ github.com/mitchellh/mapstructure v1.3.2/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RR github.com/mitchellh/mapstructure v1.3.3/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo= github.com/mitchellh/mapstructure v1.4.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo= github.com/mitchellh/mapstructure v1.4.1/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo= -github.com/mitchellh/mapstructure v1.4.2 h1:6h7AQ0yhTcIsmFmnAwQls75jp2Gzs4iB8W7pjMO+rqo= -github.com/mitchellh/mapstructure v1.4.2/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo= +github.com/mitchellh/mapstructure v1.4.3 h1:OVowDSCllw/YjdLkam3/sm7wEtOy59d8ndGgCcyj8cs= +github.com/mitchellh/mapstructure v1.4.3/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo= github.com/mitchellh/reflectwalk v1.0.2 h1:G2LzWKi524PWgd3mLHV8Y5k7s6XUvT0Gef6zxSIeXaQ= github.com/mitchellh/reflectwalk v1.0.2/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw= github.com/moby/sys/mount v0.2.0/go.mod h1:aAivFE2LB3W4bACsUXChRHQ0qKWsetY4Y9V7sxOougM= From d071a307141c486d165aee92f3c88cdda5db49e5 Mon Sep 17 00:00:00 2001 From: Willy Kloucek Date: Thu, 9 Dec 2021 17:42:08 +0100 Subject: [PATCH 23/56] release 1.16.0 --- .../change-rename-app-provider-env.md | 0 .../fix-basic-auth-route-claim-selector.md | 0 .../fix-create-group-without-name.md | 0 .../fix-s3ng-metadata-storage.md | 0 .../fix-use-cs3apis-up-and-download-workflow-accounts.md | 0 .../{unreleased => 1.16.0_2021-12-09}/ocis-pkg-config-cleanup.md | 0 changelog/{unreleased => 1.16.0_2021-12-09}/revamp-config.md | 0 .../{unreleased => 1.16.0_2021-12-09}/supervisor-stop-routine.md | 0 changelog/{unreleased => 1.16.0_2021-12-09}/update-reva.md | 0 changelog/{unreleased => 1.16.0_2021-12-09}/update-web-4.6.0.md | 0 .../{unreleased => 1.16.0_2021-12-09}/user-claim-fallback.md | 0 11 files changed, 0 insertions(+), 0 deletions(-) rename changelog/{unreleased => 1.16.0_2021-12-09}/change-rename-app-provider-env.md (100%) rename changelog/{unreleased => 1.16.0_2021-12-09}/fix-basic-auth-route-claim-selector.md (100%) rename changelog/{unreleased => 1.16.0_2021-12-09}/fix-create-group-without-name.md (100%) rename changelog/{unreleased => 1.16.0_2021-12-09}/fix-s3ng-metadata-storage.md (100%) rename changelog/{unreleased => 1.16.0_2021-12-09}/fix-use-cs3apis-up-and-download-workflow-accounts.md (100%) rename changelog/{unreleased => 1.16.0_2021-12-09}/ocis-pkg-config-cleanup.md (100%) rename changelog/{unreleased => 1.16.0_2021-12-09}/revamp-config.md (100%) rename changelog/{unreleased => 1.16.0_2021-12-09}/supervisor-stop-routine.md (100%) rename changelog/{unreleased => 1.16.0_2021-12-09}/update-reva.md (100%) rename changelog/{unreleased => 1.16.0_2021-12-09}/update-web-4.6.0.md (100%) rename changelog/{unreleased => 1.16.0_2021-12-09}/user-claim-fallback.md (100%) diff --git a/changelog/unreleased/change-rename-app-provider-env.md b/changelog/1.16.0_2021-12-09/change-rename-app-provider-env.md similarity index 100% rename from changelog/unreleased/change-rename-app-provider-env.md rename to changelog/1.16.0_2021-12-09/change-rename-app-provider-env.md diff --git a/changelog/unreleased/fix-basic-auth-route-claim-selector.md b/changelog/1.16.0_2021-12-09/fix-basic-auth-route-claim-selector.md similarity index 100% rename from changelog/unreleased/fix-basic-auth-route-claim-selector.md rename to changelog/1.16.0_2021-12-09/fix-basic-auth-route-claim-selector.md diff --git a/changelog/unreleased/fix-create-group-without-name.md b/changelog/1.16.0_2021-12-09/fix-create-group-without-name.md similarity index 100% rename from changelog/unreleased/fix-create-group-without-name.md rename to changelog/1.16.0_2021-12-09/fix-create-group-without-name.md diff --git a/changelog/unreleased/fix-s3ng-metadata-storage.md b/changelog/1.16.0_2021-12-09/fix-s3ng-metadata-storage.md similarity index 100% rename from changelog/unreleased/fix-s3ng-metadata-storage.md rename to changelog/1.16.0_2021-12-09/fix-s3ng-metadata-storage.md diff --git a/changelog/unreleased/fix-use-cs3apis-up-and-download-workflow-accounts.md b/changelog/1.16.0_2021-12-09/fix-use-cs3apis-up-and-download-workflow-accounts.md similarity index 100% rename from changelog/unreleased/fix-use-cs3apis-up-and-download-workflow-accounts.md rename to changelog/1.16.0_2021-12-09/fix-use-cs3apis-up-and-download-workflow-accounts.md diff --git a/changelog/unreleased/ocis-pkg-config-cleanup.md b/changelog/1.16.0_2021-12-09/ocis-pkg-config-cleanup.md similarity index 100% rename from changelog/unreleased/ocis-pkg-config-cleanup.md rename to changelog/1.16.0_2021-12-09/ocis-pkg-config-cleanup.md diff --git a/changelog/unreleased/revamp-config.md b/changelog/1.16.0_2021-12-09/revamp-config.md similarity index 100% rename from changelog/unreleased/revamp-config.md rename to changelog/1.16.0_2021-12-09/revamp-config.md diff --git a/changelog/unreleased/supervisor-stop-routine.md b/changelog/1.16.0_2021-12-09/supervisor-stop-routine.md similarity index 100% rename from changelog/unreleased/supervisor-stop-routine.md rename to changelog/1.16.0_2021-12-09/supervisor-stop-routine.md diff --git a/changelog/unreleased/update-reva.md b/changelog/1.16.0_2021-12-09/update-reva.md similarity index 100% rename from changelog/unreleased/update-reva.md rename to changelog/1.16.0_2021-12-09/update-reva.md diff --git a/changelog/unreleased/update-web-4.6.0.md b/changelog/1.16.0_2021-12-09/update-web-4.6.0.md similarity index 100% rename from changelog/unreleased/update-web-4.6.0.md rename to changelog/1.16.0_2021-12-09/update-web-4.6.0.md diff --git a/changelog/unreleased/user-claim-fallback.md b/changelog/1.16.0_2021-12-09/user-claim-fallback.md similarity index 100% rename from changelog/unreleased/user-claim-fallback.md rename to changelog/1.16.0_2021-12-09/user-claim-fallback.md From a85644cddd6c7428ecebe0636e53170b736bab99 Mon Sep 17 00:00:00 2001 From: David Christofas Date: Tue, 30 Nov 2021 16:46:29 +0100 Subject: [PATCH 24/56] enable signature auth in public share auth middleware --- .../unreleased/public-link-signature-auth.md | 5 +++++ proxy/pkg/middleware/public_share_auth.go | 22 ++++++++++++++----- 2 files changed, 22 insertions(+), 5 deletions(-) create mode 100644 changelog/unreleased/public-link-signature-auth.md diff --git a/changelog/unreleased/public-link-signature-auth.md b/changelog/unreleased/public-link-signature-auth.md new file mode 100644 index 0000000000..fc50ca8657 --- /dev/null +++ b/changelog/unreleased/public-link-signature-auth.md @@ -0,0 +1,5 @@ +Enhancement: Support signature auth in the public share auth middleware + +Enabled public share requests to be authenticated using the public share signature. + +https://github.com/owncloud/ocis/pull/2831 diff --git a/proxy/pkg/middleware/public_share_auth.go b/proxy/pkg/middleware/public_share_auth.go index 62e8d4206d..9b3ddcd5ff 100644 --- a/proxy/pkg/middleware/public_share_auth.go +++ b/proxy/pkg/middleware/public_share_auth.go @@ -2,6 +2,7 @@ package middleware import ( "net/http" + "strings" gateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1" ) @@ -32,12 +33,23 @@ func PublicShareAuth(opts ...Option) func(next http.Handler) http.Handler { return } - // We can ignore the username since it is always set to "public" in public shares. - _, password, ok := r.BasicAuth() + var sharePassword string + if signature := r.URL.Query().Get("signature"); signature != "" { + expiration := r.URL.Query().Get("expiration") + if expiration == "" { + logger.Warn().Str("signature", signature).Msg("cannot do signature auth without the expiration") + next.ServeHTTP(w, r) + return + } + sharePassword = strings.Join([]string{"signature", signature, expiration}, "|") + } else { + // We can ignore the username since it is always set to "public" in public shares. + _, password, ok := r.BasicAuth() - sharePassword := basicAuthPasswordPrefix - if ok { - sharePassword += password + sharePassword = basicAuthPasswordPrefix + if ok { + sharePassword += password + } } authResp, err := options.RevaGatewayClient.Authenticate(r.Context(), &gateway.AuthenticateRequest{ From ff1067a6a2394f30f8dd5e444313362cd21b50b1 Mon Sep 17 00:00:00 2001 From: Willy Kloucek Date: Thu, 9 Dec 2021 17:58:48 +0100 Subject: [PATCH 25/56] fix reva pr links --- changelog/1.16.0_2021-12-09/update-reva.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/changelog/1.16.0_2021-12-09/update-reva.md b/changelog/1.16.0_2021-12-09/update-reva.md index 0f3bd354a3..7f5b4753bf 100644 --- a/changelog/1.16.0_2021-12-09/update-reva.md +++ b/changelog/1.16.0_2021-12-09/update-reva.md @@ -6,7 +6,7 @@ This update includes: * Fix [cs3org/reva#2305](https://github.com/cs3org/reva/pull/2305): Make sure /app/new takes `target` as absolute path * Fix [cs3org/reva#2303](https://github.com/cs3org/reva/pull/2303): Fix content disposition header for public links files * Fix [cs3org/reva#2316](https://github.com/cs3org/reva/pull/2316): Fix the share types in propfinds - * Fix [cs3org/reva#2803](https://github.com/cs3org/reva/pull/2803): Fix app provider for editor public links + * Fix [cs3org/reva#2803](https://github.com/cs3org/reva/pull/2310): Fix app provider for editor public links * Fix [cs3org/reva#2298](https://github.com/cs3org/reva/pull/2298): Remove share refs from trashbin * Fix [cs3org/reva#2309](https://github.com/cs3org/reva/pull/2309): Remove early finish for zero byte file uploads * Fix [cs3org/reva#1941](https://github.com/cs3org/reva/pull/1941): Fix TUS uploads with transfer token only @@ -14,7 +14,7 @@ This update includes: * Enh [cs3org/reva#2217](https://github.com/cs3org/reva/pull/2217): OIDC auth driver for ESCAPE IAM * Enh [cs3org/reva#2256](https://github.com/cs3org/reva/pull/2256): Return user type in the response of the ocs GET user call * Enh [cs3org/reva#2315](https://github.com/cs3org/reva/pull/2315): Add new attributes to public link propfinds - * Enh [cs3org/reva#2740](https://github.com/cs3org/reva/pull/2740): Implement space membership endpoints + * Enh [cs3org/reva#2740](https://github.com/cs3org/reva/pull/2250): Implement space membership endpoints * Enh [cs3org/reva#2252](https://github.com/cs3org/reva/pull/2252): Add the xattr sys.acl to SysACL (eosgrpc) * Enh [cs3org/reva#2314](https://github.com/cs3org/reva/pull/2314): OIDC: fallback if IDP doesn't provide "preferred_username" claim From bad8b828eeb066a801fb1a2b4adce53db38ac970 Mon Sep 17 00:00:00 2001 From: Patrick Maier Date: Thu, 9 Dec 2021 18:18:44 +0100 Subject: [PATCH 26/56] [docs-only] 1.16 Release Notes --- docs/ocis/release_notes.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/docs/ocis/release_notes.md b/docs/ocis/release_notes.md index e475a7bb4b..523104047e 100644 --- a/docs/ocis/release_notes.md +++ b/docs/ocis/release_notes.md @@ -7,6 +7,29 @@ geekdocEditPath: edit/master/docs/ocis geekdocFilePath: release_notes.md --- +## ownCloud Infinite Scale 1.16.0 Technology Preview + +Version 1.16.0 brings bug fixes, new features and progress for ongoing feature implementations like 'Spaces' and application integrations. ownCloud Web comes with a couple of usability improvements (e.g., breadcrumb context menu, right-click menu for multi-select). Infinite Scale has got a revamped config handling that makes deployments easier and more flexible. Additionally, it enables easy and fast collaboration via public links. + +The most prominent changes in ownCloud Infinite Scale 1.16.0 and ownCloud Web 4.6.0 comprise: + +- ownCloud Web now provides a context menu in the navigation breadcrumb that allows users to conduct actions for the parent folder (e.g., sharing). [web#6044](https://github.com/owncloud/web/pull/6044) +- It is now possible to edit files with integrated applications in public links. [cs3org/reva#2310](https://github.com/cs3org/reva/pull/2310) +- Infinite Scale now provides the API endpoints to manage Spaces (e.g., add/remove users, manage their roles). [ocis#2740](https://github.com/owncloud/ocis/issues/2740) [cs3org/reva#2250](https://github.com/cs3org/reva/pull/2250) +- The config handling in Infinite Scale has received a huge rework to better enable different deployment and configuration models (environment variables, single config file, service-specific config files). More information can be found in the [documentation](https://owncloud.dev/ocis/config/). [#2708](https://github.com/owncloud/ocis/pull/2708) +- The right-click context menu in ownCloud Web now works when multiple files have been selected. [web#5973](https://github.com/owncloud/web/pull/5973) +- ownCloud Web now shows accessibility-optimized tooltips with absolute dates on relative dates. [web#6037](https://github.com/owncloud/web/pull/6037) +- Pagination in folders with many files now works properly again. [#6056](https://github.com/owncloud/web/pull/6056) +- The s3ng metadata storage backend works again. [#2807](https://github.com/owncloud/ocis/pull/2807) +- Improvements have been added to support more identity providers (e.g., Authelia). [cs3org/reva#2314](https://github.com/cs3org/reva/pull/2314) + +You can also read the full [ownCloud Infinite Scale changelog](https://github.com/owncloud/ocis/releases/tag/v1.16.0) and [ownCloud Web changelog](https://github.com/owncloud/web/releases/tag/v4.6.0) for further details on what has changed. + +### Breaking changes +{{< hint warning >}} +We are currently in a Tech Preview state and breaking changes may occur at any time. For more information see our [release roadmap]({{< ref "./release_roadmap" >}}) +{{< /hint >}} + ## ownCloud Infinite Scale 1.15.0 Technology Preview Version 1.15.0 brings improvements for the app provider (external application integrations) and more progress on the 'Spaces' feature. Public links now support multi-file and folder downloads as well as all other external application integrations. ownCloud Web 4.5.0 furthermore comes with improvements for use with the ownCloud Classic backend. From 427d0c0214f06d7914e7b9500bc47b3f20303e00 Mon Sep 17 00:00:00 2001 From: Artur Neumann Date: Fri, 10 Dec 2021 13:15:51 +0545 Subject: [PATCH 27/56] fix formatting --- docs/extensions/graph/spaces.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/extensions/graph/spaces.md b/docs/extensions/graph/spaces.md index 19ffa3f4c5..f20db0e2e5 100644 --- a/docs/extensions/graph/spaces.md +++ b/docs/extensions/graph/spaces.md @@ -130,6 +130,7 @@ curl -k 'https://localhost:9200/graph/v1.0/me/drives' -u einstein:relativity -v } } ] +``` As we can see the response already contains a space-aware dav endpoint, which we can use to upload files to the space: From 2856bcdef2f66297b548797cc31cc3c34c47cce2 Mon Sep 17 00:00:00 2001 From: David Christofas Date: Fri, 10 Dec 2021 08:34:19 +0000 Subject: [PATCH 28/56] Automated changelog update [skip ci] --- CHANGELOG.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index d52f3624c5..5d211f00a3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ The following sections list the changes for unreleased. * Change - Restructure Configuration Parsing: [#2708](https://github.com/owncloud/ocis/pull/2708) * Change - OIDC: fallback if IDP doesn't provide "preferred_username" claim: [#2644](https://github.com/owncloud/ocis/issues/2644) * Enhancement - Cleanup ocis-pkg config: [#2813](https://github.com/owncloud/ocis/pull/2813) +* Enhancement - Support signature auth in the public share auth middleware: [#2831](https://github.com/owncloud/ocis/pull/2831) * Enhancement - Correct shutdown of services under runtime: [#2843](https://github.com/owncloud/ocis/pull/2843) * Enhancement - Update REVA to v1.17.0: [#2849](https://github.com/owncloud/ocis/pull/2849) * Enhancement - Update ownCloud Web to v4.6.0: [#2846](https://github.com/owncloud/ocis/pull/2846) @@ -87,6 +88,12 @@ The following sections list the changes for unreleased. https://github.com/owncloud/ocis/pull/2813 +* Enhancement - Support signature auth in the public share auth middleware: [#2831](https://github.com/owncloud/ocis/pull/2831) + + Enabled public share requests to be authenticated using the public share signature. + + https://github.com/owncloud/ocis/pull/2831 + * Enhancement - Correct shutdown of services under runtime: [#2843](https://github.com/owncloud/ocis/pull/2843) Supervised goroutines now shut themselves down on context cancellation propagation. From 9dca6c32866b3f874d0a2198b284054245d76a99 Mon Sep 17 00:00:00 2001 From: pwengerter Date: Fri, 10 Dec 2021 10:20:10 +0100 Subject: [PATCH 29/56] Bump web to v4.6.1 --- changelog/1.16.0_2021-12-09/update-web-4.6.0.md | 8 -------- .../change-rename-app-provider-env.md | 0 .../fix-basic-auth-route-claim-selector.md | 0 .../fix-create-group-without-name.md | 0 .../fix-s3ng-metadata-storage.md | 0 .../fix-use-cs3apis-up-and-download-workflow-accounts.md | 0 .../ocis-pkg-config-cleanup.md | 0 .../revamp-config.md | 0 .../supervisor-stop-routine.md | 0 .../update-reva.md | 0 changelog/1.16.0_2021-12-10/update-web-4.6.1.md | 8 ++++++++ .../user-claim-fallback.md | 0 web/Makefile | 2 +- 13 files changed, 9 insertions(+), 9 deletions(-) delete mode 100644 changelog/1.16.0_2021-12-09/update-web-4.6.0.md rename changelog/{1.16.0_2021-12-09 => 1.16.0_2021-12-10}/change-rename-app-provider-env.md (100%) rename changelog/{1.16.0_2021-12-09 => 1.16.0_2021-12-10}/fix-basic-auth-route-claim-selector.md (100%) rename changelog/{1.16.0_2021-12-09 => 1.16.0_2021-12-10}/fix-create-group-without-name.md (100%) rename changelog/{1.16.0_2021-12-09 => 1.16.0_2021-12-10}/fix-s3ng-metadata-storage.md (100%) rename changelog/{1.16.0_2021-12-09 => 1.16.0_2021-12-10}/fix-use-cs3apis-up-and-download-workflow-accounts.md (100%) rename changelog/{1.16.0_2021-12-09 => 1.16.0_2021-12-10}/ocis-pkg-config-cleanup.md (100%) rename changelog/{1.16.0_2021-12-09 => 1.16.0_2021-12-10}/revamp-config.md (100%) rename changelog/{1.16.0_2021-12-09 => 1.16.0_2021-12-10}/supervisor-stop-routine.md (100%) rename changelog/{1.16.0_2021-12-09 => 1.16.0_2021-12-10}/update-reva.md (100%) create mode 100644 changelog/1.16.0_2021-12-10/update-web-4.6.1.md rename changelog/{1.16.0_2021-12-09 => 1.16.0_2021-12-10}/user-claim-fallback.md (100%) diff --git a/changelog/1.16.0_2021-12-09/update-web-4.6.0.md b/changelog/1.16.0_2021-12-09/update-web-4.6.0.md deleted file mode 100644 index 05cb15e9bb..0000000000 --- a/changelog/1.16.0_2021-12-09/update-web-4.6.0.md +++ /dev/null @@ -1,8 +0,0 @@ -Enhancement: Update ownCloud Web to v4.6.0 - -Tags: web - -We updated ownCloud Web to v4.6.0. Please refer to the changelog (linked) for details on the web release. - -https://github.com/owncloud/ocis/pull/2846 -https://github.com/owncloud/web/releases/tag/v4.6.0 diff --git a/changelog/1.16.0_2021-12-09/change-rename-app-provider-env.md b/changelog/1.16.0_2021-12-10/change-rename-app-provider-env.md similarity index 100% rename from changelog/1.16.0_2021-12-09/change-rename-app-provider-env.md rename to changelog/1.16.0_2021-12-10/change-rename-app-provider-env.md diff --git a/changelog/1.16.0_2021-12-09/fix-basic-auth-route-claim-selector.md b/changelog/1.16.0_2021-12-10/fix-basic-auth-route-claim-selector.md similarity index 100% rename from changelog/1.16.0_2021-12-09/fix-basic-auth-route-claim-selector.md rename to changelog/1.16.0_2021-12-10/fix-basic-auth-route-claim-selector.md diff --git a/changelog/1.16.0_2021-12-09/fix-create-group-without-name.md b/changelog/1.16.0_2021-12-10/fix-create-group-without-name.md similarity index 100% rename from changelog/1.16.0_2021-12-09/fix-create-group-without-name.md rename to changelog/1.16.0_2021-12-10/fix-create-group-without-name.md diff --git a/changelog/1.16.0_2021-12-09/fix-s3ng-metadata-storage.md b/changelog/1.16.0_2021-12-10/fix-s3ng-metadata-storage.md similarity index 100% rename from changelog/1.16.0_2021-12-09/fix-s3ng-metadata-storage.md rename to changelog/1.16.0_2021-12-10/fix-s3ng-metadata-storage.md diff --git a/changelog/1.16.0_2021-12-09/fix-use-cs3apis-up-and-download-workflow-accounts.md b/changelog/1.16.0_2021-12-10/fix-use-cs3apis-up-and-download-workflow-accounts.md similarity index 100% rename from changelog/1.16.0_2021-12-09/fix-use-cs3apis-up-and-download-workflow-accounts.md rename to changelog/1.16.0_2021-12-10/fix-use-cs3apis-up-and-download-workflow-accounts.md diff --git a/changelog/1.16.0_2021-12-09/ocis-pkg-config-cleanup.md b/changelog/1.16.0_2021-12-10/ocis-pkg-config-cleanup.md similarity index 100% rename from changelog/1.16.0_2021-12-09/ocis-pkg-config-cleanup.md rename to changelog/1.16.0_2021-12-10/ocis-pkg-config-cleanup.md diff --git a/changelog/1.16.0_2021-12-09/revamp-config.md b/changelog/1.16.0_2021-12-10/revamp-config.md similarity index 100% rename from changelog/1.16.0_2021-12-09/revamp-config.md rename to changelog/1.16.0_2021-12-10/revamp-config.md diff --git a/changelog/1.16.0_2021-12-09/supervisor-stop-routine.md b/changelog/1.16.0_2021-12-10/supervisor-stop-routine.md similarity index 100% rename from changelog/1.16.0_2021-12-09/supervisor-stop-routine.md rename to changelog/1.16.0_2021-12-10/supervisor-stop-routine.md diff --git a/changelog/1.16.0_2021-12-09/update-reva.md b/changelog/1.16.0_2021-12-10/update-reva.md similarity index 100% rename from changelog/1.16.0_2021-12-09/update-reva.md rename to changelog/1.16.0_2021-12-10/update-reva.md diff --git a/changelog/1.16.0_2021-12-10/update-web-4.6.1.md b/changelog/1.16.0_2021-12-10/update-web-4.6.1.md new file mode 100644 index 0000000000..55b909510c --- /dev/null +++ b/changelog/1.16.0_2021-12-10/update-web-4.6.1.md @@ -0,0 +1,8 @@ +Enhancement: Update ownCloud Web to v4.6.1 + +Tags: web + +We updated ownCloud Web to v4.6.1. Please refer to the changelog (linked) for details on the web release. + +https://github.com/owncloud/ocis/pull/2846 +https://github.com/owncloud/web/releases/tag/v4.6.1 diff --git a/changelog/1.16.0_2021-12-09/user-claim-fallback.md b/changelog/1.16.0_2021-12-10/user-claim-fallback.md similarity index 100% rename from changelog/1.16.0_2021-12-09/user-claim-fallback.md rename to changelog/1.16.0_2021-12-10/user-claim-fallback.md diff --git a/web/Makefile b/web/Makefile index 2d9c78e628..905fe383db 100644 --- a/web/Makefile +++ b/web/Makefile @@ -29,7 +29,7 @@ ci-go-generate: # CI runs ci-node-generate automatically before this target .PHONY: ci-node-generate ci-node-generate: pull-assets -WEB_ASSETS_VERSION = v4.6.0 +WEB_ASSETS_VERSION = v4.6.1 .PHONY: pull-assets pull-assets: From 052d219465d0471b9c5daec7479b2bf79255baac Mon Sep 17 00:00:00 2001 From: Michael Barz Date: Fri, 10 Dec 2021 14:55:58 +0100 Subject: [PATCH 30/56] cleanup docs --- docs/extensions/storage/spaces.md | 8 ++--- docs/extensions/storage/terminology.md | 47 -------------------------- 2 files changed, 4 insertions(+), 51 deletions(-) diff --git a/docs/extensions/storage/spaces.md b/docs/extensions/storage/spaces.md index 3624508753..e877171727 100644 --- a/docs/extensions/storage/spaces.md +++ b/docs/extensions/storage/spaces.md @@ -140,13 +140,13 @@ Upload a 6 bytes file: Query the quota again: ```json -... -"quota": { +{ + "quota": { "remaining": 4, "total": 10, "used": 6 -}, -... + } +} ``` Now attempt to upload 5 bytes to the space: diff --git a/docs/extensions/storage/terminology.md b/docs/extensions/storage/terminology.md index f4522fb727..b13ba0194e 100644 --- a/docs/extensions/storage/terminology.md +++ b/docs/extensions/storage/terminology.md @@ -36,48 +36,6 @@ A *reference* is a logical concept that identifies a [*resource*]({{< ref "#reso The `/` is important because currently the static [*storage registry*]({{< ref "#storage-space-registries" >}}) uses a map to look up which [*storage provider*]({{< ref "#storage-providers" >}}) is responsible for the resource. Paths must be prefixed with `/` so there can be no collisions between paths and storage provider ids in the same map. {{< /hint >}} - -{{< hint warning >}} -### Alternative: reference triple #### -A *reference* is a logical concept. It identifies a [*resource*]({{< ref "#resources" >}}) and consists of -a `storage_space`, a `` and a `` -``` -!: -``` -While all components are optional, only three cases are used: -| format | example | description | -|-|-|-| -| `!:` | `!:/absolute/path/to/file.ext` | absolute path | -| `!:` | `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!:path/to/file.ext` | path relative to the root of the storage space | -| `!:` | `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!c3cf23bb-8f47-4719-a150-1d25a1f6fb56:to/file.ext` | path relative to the specified node in the storage space, used to reference resources without disclosing parent paths | - -`` should be a UUID to prevent references from breaking when a *user* or [*storage space*]({{< ref "#storage-spaces" >}}) gets renamed. But it can also be derived from a migration of an oc10 instance by concatenating an instance identifier and the numeric storage id from oc10, e.g. `oc10-instance-a$1234`. - -A reference will often start as an absolute/global path, e.g. `!:/home/Projects/Foo`. The gateway will look up the storage provider that is responsible for the path - -| Name | Description | Who resolves it? | -|------|-------------|-| -| `!:/home/Projects/Foo` | the absolute path a client like davfs will use. | The gateway uses the storage registry to look up the responsible storage provider | -| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!:/Projects/Foo` | the `storage_space` is the same as the `root`, the path becomes relative to the root | the storage provider can use this reference to identify this resource | - -Now, the same file is accessed as a share -| Name | Description | -|------|-------------| -| `!:/users/Einstein/Projects/Foo` | `Foo` is the shared folder | -| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:` | `56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a` is the id of `Foo`, the path is empty | - - -The `:`, `!` and `$` are chosen from the set of [RFC3986 sub delimiters](https://tools.ietf.org/html/rfc3986#section-2.2) on purpose. They can be used in URLs without having to be encoded. In some cases, a delimiter can be left out if a component is not set: -| reference | interpretation | -|-|-| -| `/absolute/path/to/file.ext` | absolute path, all delimiters omitted | -| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!path/to/file.ext` | relative path in the given storage space, root delimiter `:` omitted | -| `56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:to/file.ext` | relative path in the given root node, storage space delimiter `!` omitted | -| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:` | node id in the given storage space, `:` must be present | -| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62` | root of the storage space, all delimiters omitted, can be distinguished by the `/` | - -{{< /hint >}} - ## Storage Drivers A *storage driver* implements access to a [*storage system*]({{< ref "#storage-systems" >}}): @@ -108,16 +66,11 @@ Technically, this means that every storage driver needs to have a map of a `uuid ### Storage Systems {{< svg src="extensions/storage/static/storageprovider.drawio.svg" >}} -{{< hint warning >}} -**Proposed Change** A *storage provider* manages multiple [*storage spaces*]({{< ref "#storage-space" >}}) by accessing a [*storage system*]({{< ref "#storage-systems" >}}) with a [*storage driver*]({{< ref "#storage-drivers" >}}). {{< svg src="extensions/storage/static/storageprovider-spaces.drawio.svg" >}} -By making [*storage providers*]({{< ref "#storage-providers" >}}) aware of [*storage spaces*]({{< ref "#storage-spaces" >}}) we can get rid of the current `enablehome` flag / hack in reva, which lead to the [spawn of `*home` drivers](https://github.com/cs3org/reva/tree/master/pkg/storage/fs). Furthermore, provisioning a new [*storage space*]({{< ref "#storage-space" >}}) becomes a generic operation, regardless of the need of provisioning a new user home or a new project space. -{{< /hint >}} - ## Storage Space Registries A *storage registry* manages the [*CS3 global namespace*]({{< ref "./namespaces.md#cs3-global-namespaces" >}}): From 5afb837d2390d97946e94f0435bed2e188c16056 Mon Sep 17 00:00:00 2001 From: Michael Barz Date: Fri, 10 Dec 2021 15:49:31 +0100 Subject: [PATCH 31/56] remove unneeded duplicates in docs --- docs/extensions/storage/terminology.md | 21 ++------------------- 1 file changed, 2 insertions(+), 19 deletions(-) diff --git a/docs/extensions/storage/terminology.md b/docs/extensions/storage/terminology.md index b13ba0194e..e40e2ad04f 100644 --- a/docs/extensions/storage/terminology.md +++ b/docs/extensions/storage/terminology.md @@ -73,24 +73,7 @@ by accessing a [*storage system*]({{< ref "#storage-systems" >}}) with a [*stora ## Storage Space Registries -A *storage registry* manages the [*CS3 global namespace*]({{< ref "./namespaces.md#cs3-global-namespaces" >}}): -It is used by the *gateway* -to look up `address` and `port` of the [*storage provider*]({{< ref "#storage-providers" >}}) -that should handle a [*reference*]({{< ref "#references" >}}). - -{{< svg src="extensions/storage/static/storageregistry.drawio.svg" >}} - -{{< hint warning >}} -**Proposed Change** -A *storage space registry* manages the [*namespace*]({{< ref "./namespaces.md" >}}) for a *user*: -It is used by the *gateway* -to look up `address` and `port` of the [*storage provider*]({{< ref "#storage-providers" >}}) -that is currently serving a [*storage space*]({{< ref "#storage-space" >}}). - -{{< svg src="extensions/storage/static/storageregistry-spaces.drawio.svg" >}} - -By making *storage registries* aware of [*storage spaces*]({{< ref "#storage-spaces" >}}) we can query them for a listing of all [*storage spaces*]({{< ref "#storage-spaces" >}}) a user has access to. Including his home, received shares, project folders or group drives. See [a WIP PR for spaces in the oCIS repo (#1827)](https://github.com/owncloud/ocis/pull/1827) for more info. -{{< /hint >}} +A [*storage spaces registry*]({{< ref "./spacesregistry.md" >}}) manages the [*namespace*]({{< ref "./namespaces.md" >}}) for a *user* ## Storage Spaces A *storage space* is a logical concept: @@ -105,7 +88,7 @@ Examples would be every user's home storage space, project storage spaces or gro Finally, a logical `storage space id` is not tied to a specific [*storage provider*]({{< ref "#storage-providers" >}}). If the [*storage driver*]({{< ref "#storage-drivers" >}}) supports it, we can import existing files including their `file id`, which makes it possible to move [*storage spaces*]({{< ref "#storage-spaces" >}}) between [*storage providers*]({{< ref "#storage-providers" >}}) to implement storage classes, e.g. with or without archival, workflows, on SSDs or HDDs. ## Shares -*To be clarified: we are aware that [*storage spaces*]({{< ref "#storage-spaces" >}}) may be too 'heavyweight' for ad hoc sharing with groups. That being said, there is no technical reason why group shares should not be treated like [*storage spaces*]({{< ref "#storage-spaces" >}}) that users can provision themselves. They would share the quota with the users home [*storage space*]({{< ref "#storage-spaces" >}}) and the share initiator would be the sole owner. Technically, the mechanism of treating a share like a new [*storage space*]({{< ref "#storage-spaces" >}}) would be the same. This obviously also extends to user shares and even file individual shares that would be wrapped in a virtual collection. It would also become possible to share collections of arbitrary files in a single storage space, e.g. the ten best pictures from a large album.* +*To be clarified: we are aware that [*storage spaces*]({{< ref "#storage-spaces" >}}) may be too 'heavyweight' for ad hoc sharEsing with groups. That being said, there is no technical reason why group shares should not be treated like [*storage spaces*]({{< ref "#storage-spaces" >}}) that users can provision themselves. They would share the quota with the users home [*storage space*]({{< ref "#storage-spaces" >}}) and the share initiator would be the sole owner. Technically, the mechanism of treating a share like a new [*storage space*]({{< ref "#storage-spaces" >}}) would be the same. This obviously also extends to user shares and even file individual shares that would be wrapped in a virtual collection. It would also become possible to share collections of arbitrary files in a single storage space, e.g. the ten best pictures from a large album.* ## Storage Systems From e628750dc27d743b442097505e5ec58e004f4c13 Mon Sep 17 00:00:00 2001 From: Michael Barz Date: Fri, 3 Dec 2021 09:14:22 +0100 Subject: [PATCH 32/56] graph update --- changelog/unreleased/update-libregraph-api.md | 5 ++ go.mod | 6 +- go.sum | 12 +-- graph/pkg/service/v0/drives.go | 79 ++++++++----------- graph/pkg/service/v0/errorcode/errorcode.go | 6 +- 5 files changed, 48 insertions(+), 60 deletions(-) create mode 100644 changelog/unreleased/update-libregraph-api.md diff --git a/changelog/unreleased/update-libregraph-api.md b/changelog/unreleased/update-libregraph-api.md new file mode 100644 index 0000000000..f8da54c775 --- /dev/null +++ b/changelog/unreleased/update-libregraph-api.md @@ -0,0 +1,5 @@ +Change: Update libre-graph-api to v0.3.0 + +This updates the libre-graph-api to use the latest spec and types. + +https://github.com/owncloud/ocis/pull/2858 diff --git a/go.mod b/go.mod index 55aed36748..6e118c61bd 100644 --- a/go.mod +++ b/go.mod @@ -46,7 +46,7 @@ require ( github.com/olekukonko/tablewriter v0.0.5 github.com/onsi/ginkgo v1.16.5 github.com/onsi/gomega v1.17.0 - github.com/owncloud/open-graph-api-go v0.0.0-20211018134924-bea0e2a6ec3e + github.com/owncloud/libre-graph-api-go v0.3.0 github.com/pkg/errors v0.9.1 github.com/prometheus/client_golang v1.11.0 github.com/rs/zerolog v1.26.0 @@ -199,8 +199,8 @@ require ( github.com/prometheus/common v0.32.1 // indirect github.com/prometheus/procfs v0.7.3 // indirect github.com/prometheus/statsd_exporter v0.22.2 // indirect - github.com/rickb777/date v1.16.1 // indirect - github.com/rickb777/plural v1.4.1 // indirect + github.com/rickb777/date v1.12.4 // indirect + github.com/rickb777/plural v1.2.0 // indirect github.com/rivo/uniseg v0.2.0 // indirect github.com/rs/cors v1.8.0 // indirect github.com/rs/xid v1.3.0 // indirect diff --git a/go.sum b/go.sum index 3f125c24d7..c94be2021e 100644 --- a/go.sum +++ b/go.sum @@ -992,7 +992,6 @@ github.com/onsi/ginkgo v1.16.5/go.mod h1:+E8gABHa3K6zRBolWtd+ROzc/U5bkGt0FwiG042 github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY= github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo= -github.com/onsi/gomega v1.10.5/go.mod h1:gza4q3jKQJijlu05nKWRCW/GavJumGt8aNRxWg7mt48= github.com/onsi/gomega v1.17.0 h1:9Luw4uT5HTjHTN8+aNcSThgH1vdXnmdJ8xIfZ4wyTRE= github.com/onsi/gomega v1.17.0/go.mod h1:HnhC7FXeEQY45zxNK3PPoIUhzk/80Xly9PcubAlGdZY= github.com/op/go-logging v0.0.0-20160315200505-970db520ece7/go.mod h1:HzydrMdWErDVzsI23lYNej1Htcns9BCg93Dk0bBINWk= @@ -1009,8 +1008,8 @@ github.com/oracle/oci-go-sdk v24.3.0+incompatible/go.mod h1:VQb79nF8Z2cwLkLS35uk github.com/orcaman/concurrent-map v0.0.0-20210501183033-44dafcb38ecc h1:Ak86L+yDSOzKFa7WM5bf5itSOo1e3Xh8bm5YCMUXIjQ= github.com/orcaman/concurrent-map v0.0.0-20210501183033-44dafcb38ecc/go.mod h1:Lu3tH6HLW3feq74c2GC+jIMS/K2CFcDWnWD9XkenwhI= github.com/ovh/go-ovh v1.1.0/go.mod h1:AxitLZ5HBRPyUd+Zl60Ajaag+rNTdVXWIkzfrVuTXWA= -github.com/owncloud/open-graph-api-go v0.0.0-20211018134924-bea0e2a6ec3e h1:Aww53qP8t9WmFq88NZk5BIcVz6zH4z4QqE98JEpsLRs= -github.com/owncloud/open-graph-api-go v0.0.0-20211018134924-bea0e2a6ec3e/go.mod h1:1Z3JQaJg9KjZ9dW5fxA0hfqdgL9Ma/9wmU3Uk+7rIx4= +github.com/owncloud/libre-graph-api-go v0.3.0 h1:CjSWuL6Gd+HgIlZfBLjYTloH1F/8naxaL5xZZz4N+ms= +github.com/owncloud/libre-graph-api-go v0.3.0/go.mod h1:dOt7+kXldpyUGBFdfFBs6PbWElqqa2/Q5+pMF9izL3w= github.com/oxtoacart/bpool v0.0.0-20190530202638-03653db5a59c h1:rp5dCmg/yLR3mgFuSOe4oEnDDmGLROTvMragMUXpTQw= github.com/oxtoacart/bpool v0.0.0-20190530202638-03653db5a59c/go.mod h1:X07ZCGwUbLaax7L0S3Tw4hpejzu63ZrrQiUe6W0hcy0= github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc= @@ -1091,13 +1090,10 @@ github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40T github.com/rainycape/memcache v0.0.0-20150622160815-1031fa0ce2f2/go.mod h1:7tZKcyumwBO6qip7RNQ5r77yrssm9bfCowcLEBcU5IA= github.com/rcrowley/go-metrics v0.0.0-20181016184325-3113b8401b8a/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4= github.com/rcrowley/go-metrics v0.0.0-20190826022208-cac0b30c2563/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4= +github.com/rickb777/date v1.12.4 h1:+6IzcCCS/1t17DrmnEvrznyq7nM8vPwir6/UhlyohKw= github.com/rickb777/date v1.12.4/go.mod h1:xP0eo/I5qmUt97yRGClHZfyLZ3ikMw6v6SU5MOGZTE0= -github.com/rickb777/date v1.16.1 h1:nUx7FrnRLxwj4QpbuHOz7RRcnEyFOiXnZxdC2lx0f8c= -github.com/rickb777/date v1.16.1/go.mod h1:QwU+l0bIHSFsMQH12voxZbC531J+lM3A/ZFq5gku8F8= +github.com/rickb777/plural v1.2.0 h1:5tvEc7UBCZ7l8h/2UeybSkt/uu1DQsZFOFdNevmUhlE= github.com/rickb777/plural v1.2.0/go.mod h1:UdpyWFCGbo3mvK3f/PfZOAOrkjzJlYN/sD46XNWJ+Es= -github.com/rickb777/plural v1.3.0/go.mod h1:xyHbelv4YvJE51gjMnHvk+U2e9zIysg6lTnSQK8XUYA= -github.com/rickb777/plural v1.4.1 h1:5MMLcbIaapLFmvDGRT5iPk8877hpTPt8Y9cdSKRw9sU= -github.com/rickb777/plural v1.4.1/go.mod h1:kdmXUpmKBJTS0FtG/TFumd//VBWsNTD7zOw7x4umxNw= github.com/rivo/uniseg v0.2.0 h1:S1pD9weZBuJdFmowNwbpi7BJ8TNftyUImj/0WQi72jY= github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc= github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg= diff --git a/graph/pkg/service/v0/drives.go b/graph/pkg/service/v0/drives.go index aa370cbdfd..065998ac33 100644 --- a/graph/pkg/service/v0/drives.go +++ b/graph/pkg/service/v0/drives.go @@ -22,11 +22,11 @@ import ( types "github.com/cs3org/go-cs3apis/cs3/types/v1beta1" ctxpkg "github.com/cs3org/reva/pkg/ctx" "github.com/go-chi/render" + libregraph "github.com/owncloud/libre-graph-api-go" "github.com/owncloud/ocis/graph/pkg/service/v0/errorcode" "github.com/owncloud/ocis/ocis-pkg/service/grpc" sproto "github.com/owncloud/ocis/settings/pkg/proto/v0" settingsSvc "github.com/owncloud/ocis/settings/pkg/service/v0" - msgraph "github.com/owncloud/open-graph-api-go" merrors "go-micro.dev/v4/errors" ) @@ -190,7 +190,7 @@ func (g Graph) CreateDrive(w http.ResponseWriter, r *http.Request) { errorcode.GeneralException.Render(w, r, http.StatusInternalServerError, err.Error()) return } - drive := msgraph.Drive{} + drive := libregraph.Drive{} if err := json.NewDecoder(r.Body).Decode(&drive); err != nil { errorcode.GeneralException.Render(w, r, http.StatusBadRequest, "invalid schema definition") return @@ -268,7 +268,7 @@ func (g Graph) UpdateDrive(w http.ResponseWriter, r *http.Request) { return } - drive := msgraph.Drive{} + drive := libregraph.Drive{} if err = json.NewDecoder(r.Body).Decode(&drive); err != nil { errorcode.GeneralException.Render(w, r, http.StatusBadRequest, fmt.Sprintf("invalid request body: %v", r.Body)) return @@ -341,38 +341,34 @@ func cs3TimestampToTime(t *types.Timestamp) time.Time { return time.Unix(int64(t.Seconds), int64(t.Nanos)) } -func cs3ResourceToDriveItem(res *storageprovider.ResourceInfo) (*msgraph.DriveItem, error) { +func cs3ResourceToDriveItem(res *storageprovider.ResourceInfo) (*libregraph.DriveItem, error) { size := new(int64) *size = int64(res.Size) // uint64 -> int :boom: name := path.Base(res.Path) - driveItem := &msgraph.DriveItem{ - BaseItem: msgraph.BaseItem{ - Entity: msgraph.Entity{ - Id: &res.Id.OpaqueId, - }, - Name: &name, - ETag: &res.Etag, - }, + driveItem := &libregraph.DriveItem{ + Id: &res.Id.OpaqueId, + Name: &name, + ETag: &res.Etag, Size: size, } if res.Mtime != nil { lastModified := cs3TimestampToTime(res.Mtime) - driveItem.BaseItem.LastModifiedDateTime = &lastModified + driveItem.LastModifiedDateTime = &lastModified } if res.Type == storageprovider.ResourceType_RESOURCE_TYPE_FILE { - driveItem.File = &msgraph.OpenGraphFile{ // FIXME We cannot use msgraph.File here because the openapi codegenerator autodetects 'File' as a go type ... + driveItem.File = &libregraph.OpenGraphFile{ // FIXME We cannot use libregraph.File here because the openapi codegenerator autodetects 'File' as a go type ... MimeType: &res.MimeType, } } if res.Type == storageprovider.ResourceType_RESOURCE_TYPE_CONTAINER { - driveItem.Folder = &msgraph.Folder{} + driveItem.Folder = &libregraph.Folder{} } return driveItem, nil } -func formatDriveItems(mds []*storageprovider.ResourceInfo) ([]*msgraph.DriveItem, error) { - responses := make([]*msgraph.DriveItem, 0, len(mds)) +func formatDriveItems(mds []*storageprovider.ResourceInfo) ([]*libregraph.DriveItem, error) { + responses := make([]*libregraph.DriveItem, 0, len(mds)) for i := range mds { res, err := cs3ResourceToDriveItem(mds[i]) if err != nil { @@ -384,31 +380,22 @@ func formatDriveItems(mds []*storageprovider.ResourceInfo) ([]*msgraph.DriveItem return responses, nil } -func cs3StorageSpaceToDrive(baseURL *url.URL, space *storageprovider.StorageSpace) (*msgraph.Drive, error) { +func cs3StorageSpaceToDrive(baseURL *url.URL, space *storageprovider.StorageSpace) (*libregraph.Drive, error) { rootID := space.Root.StorageId + "!" + space.Root.OpaqueId - drive := &msgraph.Drive{ - BaseItem: msgraph.BaseItem{ - Entity: msgraph.Entity{ - Id: &space.Id.OpaqueId, - }, - Name: &space.Name, - //"createdDateTime": "string (timestamp)", // TODO read from StorageSpace ... needs Opaque for now - //"description": "string", // TODO read from StorageSpace ... needs Opaque for now - }, - Owner: &msgraph.IdentitySet{ - User: &msgraph.Identity{ + drive := &libregraph.Drive{ + Id: &space.Id.OpaqueId, + Name: &space.Name, + //"createdDateTime": "string (timestamp)", // TODO read from StorageSpace ... needs Opaque for now + //"description": "string", // TODO read from StorageSpace ... needs Opaque for now + Owner: &libregraph.IdentitySet{ + User: &libregraph.Identity{ Id: &space.Owner.Id.OpaqueId, // DisplayName: , TODO read and cache from users provider }, }, - DriveType: &space.SpaceType, - Root: &msgraph.DriveItem{ - BaseItem: msgraph.BaseItem{ - Entity: msgraph.Entity{ - Id: &rootID, - }, - }, + Root: &libregraph.DriveItem{ + Id: &rootID, }, } @@ -422,7 +409,7 @@ func cs3StorageSpaceToDrive(baseURL *url.URL, space *storageprovider.StorageSpac if space.Mtime != nil { lastModified := cs3TimestampToTime(space.Mtime) - drive.BaseItem.LastModifiedDateTime = &lastModified + drive.LastModifiedDateTime = &lastModified } if space.Quota != nil { var t int64 @@ -431,7 +418,7 @@ func cs3StorageSpaceToDrive(baseURL *url.URL, space *storageprovider.StorageSpac } else { t = int64(space.Quota.QuotaMaxBytes) } - drive.Quota = &msgraph.Quota{ + drive.Quota = &libregraph.Quota{ Total: &t, } } @@ -440,8 +427,8 @@ func cs3StorageSpaceToDrive(baseURL *url.URL, space *storageprovider.StorageSpac return drive, nil } -func (g Graph) formatDrives(ctx context.Context, baseURL *url.URL, mds []*storageprovider.StorageSpace) ([]*msgraph.Drive, error) { - responses := make([]*msgraph.Drive, 0, len(mds)) +func (g Graph) formatDrives(ctx context.Context, baseURL *url.URL, mds []*storageprovider.StorageSpace) ([]*libregraph.Drive, error) { + responses := make([]*libregraph.Drive, 0, len(mds)) for i := range mds { res, err := cs3StorageSpaceToDrive(baseURL, mds[i]) if err != nil { @@ -458,11 +445,11 @@ func (g Graph) formatDrives(ctx context.Context, baseURL *url.URL, mds []*storag return responses, nil } -func (g Graph) getDriveQuota(ctx context.Context, space *storageprovider.StorageSpace) (msgraph.Quota, error) { +func (g Graph) getDriveQuota(ctx context.Context, space *storageprovider.StorageSpace) (libregraph.Quota, error) { client, err := g.GetClient() if err != nil { g.logger.Error().Err(err).Msg("error creating grpc client") - return msgraph.Quota{}, err + return libregraph.Quota{}, err } req := &gateway.GetQuotaRequest{ @@ -478,17 +465,17 @@ func (g Graph) getDriveQuota(ctx context.Context, space *storageprovider.Storage switch { case err != nil: g.logger.Error().Err(err).Msg("error sending get quota grpc request") - return msgraph.Quota{}, err + return libregraph.Quota{}, err case res.Status.Code != cs3rpc.Code_CODE_OK: g.logger.Error().Err(err).Msg("error sending sending get quota grpc request") - return msgraph.Quota{}, err + return libregraph.Quota{}, err } total := int64(res.TotalBytes) used := int64(res.UsedBytes) remaining := total - used - qta := msgraph.Quota{ + qta := libregraph.Quota{ Remaining: &remaining, Total: &total, Used: &used, @@ -514,7 +501,7 @@ func calculateQuotaState(total int64, used int64) (state string) { } } -func getQuota(quota *msgraph.Quota, defaultQuota string) *provider.Quota { +func getQuota(quota *libregraph.Quota, defaultQuota string) *provider.Quota { switch { case quota != nil && quota.Total != nil: if q := *quota.Total; q >= 0 { diff --git a/graph/pkg/service/v0/errorcode/errorcode.go b/graph/pkg/service/v0/errorcode/errorcode.go index 23dc333e58..0f540fa7ce 100644 --- a/graph/pkg/service/v0/errorcode/errorcode.go +++ b/graph/pkg/service/v0/errorcode/errorcode.go @@ -6,7 +6,7 @@ import ( "github.com/go-chi/chi/v5/middleware" "github.com/go-chi/render" - msgraph "github.com/owncloud/open-graph-api-go" + libregraph "github.com/owncloud/libre-graph-api-go" ) // ErrorCode defines code as used in MS Graph - see https://docs.microsoft.com/en-us/graph/errors?context=graph%2Fapi%2F1.0&view=graph-rest-1.0 @@ -86,8 +86,8 @@ func (e ErrorCode) Render(w http.ResponseWriter, r *http.Request, status int, ms } innererror["request-id"] = middleware.GetReqID(r.Context()) - resp := &msgraph.OdataError{ - Error: msgraph.OdataErrorMain{ + resp := &libregraph.OdataError{ + Error: libregraph.OdataErrorMain{ Code: e.String(), Message: msg, Innererror: &innererror, From c2f648d6b172458abd5cbff4d4b208b5fdd87aaf Mon Sep 17 00:00:00 2001 From: Michael Barz Date: Sun, 12 Dec 2021 16:26:45 +0000 Subject: [PATCH 33/56] Automated changelog update [skip ci] --- CHANGELOG.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5d211f00a3..bfe5f03436 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,7 @@ The following sections list the changes for unreleased. * Bugfix - Use the CS3api up- and download workflow for the accounts service: [#2837](https://github.com/owncloud/ocis/pull/2837) * Change - Rename `APP_PROVIDER_BASIC_*` environment variables: [#2812](https://github.com/owncloud/ocis/pull/2812) * Change - Restructure Configuration Parsing: [#2708](https://github.com/owncloud/ocis/pull/2708) +* Change - Update libre-graph-api to v0.3.0: [#2858](https://github.com/owncloud/ocis/pull/2858) * Change - OIDC: fallback if IDP doesn't provide "preferred_username" claim: [#2644](https://github.com/owncloud/ocis/issues/2644) * Enhancement - Cleanup ocis-pkg config: [#2813](https://github.com/owncloud/ocis/pull/2813) * Enhancement - Support signature auth in the public share auth middleware: [#2831](https://github.com/owncloud/ocis/pull/2831) @@ -76,6 +77,12 @@ The following sections list the changes for unreleased. https://github.com/owncloud/ocis/pull/2708 +* Change - Update libre-graph-api to v0.3.0: [#2858](https://github.com/owncloud/ocis/pull/2858) + + This updates the libre-graph-api to use the latest spec and types. + + https://github.com/owncloud/ocis/pull/2858 + * Change - OIDC: fallback if IDP doesn't provide "preferred_username" claim: [#2644](https://github.com/owncloud/ocis/issues/2644) Some IDPs don't add the "preferred_username" claim. Fallback to the "email" claim in that case From efd7f76f783bf92a9dee29c17ee8b68da54e2c36 Mon Sep 17 00:00:00 2001 From: Michael Barz Date: Fri, 10 Dec 2021 14:24:59 +0100 Subject: [PATCH 34/56] change update drives --- graph/pkg/service/v0/drives.go | 28 ++++++++++------------------ graph/pkg/service/v0/service.go | 6 +++--- 2 files changed, 13 insertions(+), 21 deletions(-) diff --git a/graph/pkg/service/v0/drives.go b/graph/pkg/service/v0/drives.go index aa370cbdfd..b4ca4b85bf 100644 --- a/graph/pkg/service/v0/drives.go +++ b/graph/pkg/service/v0/drives.go @@ -12,7 +12,6 @@ import ( "strings" "time" - "github.com/CiscoM31/godata" gateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1" userv1beta1 "github.com/cs3org/go-cs3apis/cs3/identity/user/v1beta1" cs3rpc "github.com/cs3org/go-cs3apis/cs3/rpc/v1beta1" @@ -21,6 +20,7 @@ import ( storageprovider "github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1" types "github.com/cs3org/go-cs3apis/cs3/types/v1beta1" ctxpkg "github.com/cs3org/reva/pkg/ctx" + "github.com/go-chi/chi/v5" "github.com/go-chi/render" "github.com/owncloud/ocis/graph/pkg/service/v0/errorcode" "github.com/owncloud/ocis/ocis-pkg/service/grpc" @@ -249,22 +249,14 @@ func (g Graph) CreateDrive(w http.ResponseWriter, r *http.Request) { } func (g Graph) UpdateDrive(w http.ResponseWriter, r *http.Request) { - // wildcards however addressed here is not yet supported. We want to address drives by their unique - // identifiers. Any open queries need to be implemented. Same applies for sub-entities. - // For further reading: http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part2-url-conventions.html#sec_AddressingaSubsetofaCollection - - // strip "/graph/v1.0/" out and parse the rest. This is how godata input is expected. - //https://github.com/CiscoM31/godata/blob/d70e191d2908191623be84401fecc40d6af4afde/url_parser_test.go#L10 - sanitized := strings.TrimPrefix(r.URL.Path, "/graph/v1.0/") - - req, err := godata.ParseRequest(r.Context(), sanitized, r.URL.Query()) + driveID := chi.URLParam(r, "driveID") + driveID, err := url.PathUnescape(driveID) if err != nil { - errorcode.GeneralException.Render(w, r, http.StatusBadRequest, err.Error()) - return + errorcode.InvalidRequest.Render(w, r, http.StatusBadRequest, "unescaping drive id failed") } - if req.FirstSegment.Identifier.Get() == "" { - errorcode.GeneralException.Render(w, r, http.StatusBadRequest, "identifier cannot be empty") + if driveID == "" { + errorcode.InvalidRequest.Render(w, r, http.StatusBadRequest, "missing drive id") return } @@ -274,9 +266,9 @@ func (g Graph) UpdateDrive(w http.ResponseWriter, r *http.Request) { return } - identifierParts := strings.Split(req.FirstSegment.Identifier.Get(), "!") + identifierParts := strings.Split(driveID, "!") if len(identifierParts) != 2 { - errorcode.GeneralException.Render(w, r, http.StatusBadRequest, fmt.Sprintf("invalid resource id: %v", req.FirstSegment.Identifier.Get())) + errorcode.GeneralException.Render(w, r, http.StatusBadRequest, fmt.Sprintf("invalid resource id: %v", driveID)) w.WriteHeader(http.StatusInternalServerError) return } @@ -294,7 +286,7 @@ func (g Graph) UpdateDrive(w http.ResponseWriter, r *http.Request) { // the original storage space. StorageSpace: &provider.StorageSpace{ Id: &storageprovider.StorageSpaceId{ - OpaqueId: req.FirstSegment.Identifier.Get(), + OpaqueId: driveID, }, Root: &provider.ResourceId{ StorageId: storageID, @@ -330,7 +322,7 @@ func (g Graph) UpdateDrive(w http.ResponseWriter, r *http.Request) { } if resp.GetStatus().GetCode() != v1beta11.Code_CODE_OK { - errorcode.GeneralException.Render(w, r, http.StatusInternalServerError, "") + errorcode.GeneralException.Render(w, r, http.StatusInternalServerError, resp.GetStatus().GetMessage()) return } diff --git a/graph/pkg/service/v0/service.go b/graph/pkg/service/v0/service.go index e722d160b9..36d0cf451b 100644 --- a/graph/pkg/service/v0/service.go +++ b/graph/pkg/service/v0/service.go @@ -85,9 +85,9 @@ func NewService(opts ...Option) Service { r.Route("/drives", func(r chi.Router) { r.Get("/", svc.GetDrives) r.Post("/", svc.CreateDrive) - }) - r.Route("/Drive({firstSegmentIdentifier})", func(r chi.Router) { - r.Patch("/*", svc.UpdateDrive) + r.Route("/{driveID}", func(r chi.Router) { + r.Patch("/", svc.UpdateDrive) + }) }) }) }) From 6e73ee7466ac19032e4a8dc05de81e56596326f2 Mon Sep 17 00:00:00 2001 From: Michael Barz Date: Sat, 11 Dec 2021 21:40:58 +0100 Subject: [PATCH 35/56] handle non existent spaces gracefully --- .../update-spaces-handle-non-existent.md | 5 ++++ go.mod | 2 ++ go.sum | 4 +-- graph/pkg/service/v0/drives.go | 27 ++++++++++++++++--- 4 files changed, 33 insertions(+), 5 deletions(-) create mode 100644 changelog/unreleased/update-spaces-handle-non-existent.md diff --git a/changelog/unreleased/update-spaces-handle-non-existent.md b/changelog/unreleased/update-spaces-handle-non-existent.md new file mode 100644 index 0000000000..c69fbc8d25 --- /dev/null +++ b/changelog/unreleased/update-spaces-handle-non-existent.md @@ -0,0 +1,5 @@ +Change: Return not found when updating non existent space + +If a spaceid of a space which is updated doesn't exist, handle it as a not found error. + +https://github.com/cs3org/reva/pull/2869 diff --git a/go.mod b/go.mod index a2cd56ad20..350a659b41 100644 --- a/go.mod +++ b/go.mod @@ -241,3 +241,5 @@ require ( stash.kopano.io/kgol/kcc-go/v5 v5.0.1 // indirect stash.kopano.io/kgol/oidc-go v0.3.2 // indirect ) + +replace github.com/cs3org/reva => github.com/micbar/reva v1.9.1-0.20211211204557-ace6314aca5d diff --git a/go.sum b/go.sum index e8cb72438e..be2e4c97f3 100644 --- a/go.sum +++ b/go.sum @@ -299,8 +299,6 @@ github.com/crewjam/saml v0.4.5/go.mod h1:qCJQpUtZte9R1ZjUBcW8qtCNlinbO363ooNl02S github.com/cs3org/cato v0.0.0-20200828125504-e418fc54dd5e/go.mod h1:XJEZ3/EQuI3BXTp/6DUzFr850vlxq11I6satRtz0YQ4= github.com/cs3org/go-cs3apis v0.0.0-20211104090126-8e972dca8304 h1:e/nIPR518vyvrulo9goAZTtYD6gFfu/2/9MDe6mTGcw= github.com/cs3org/go-cs3apis v0.0.0-20211104090126-8e972dca8304/go.mod h1:UXha4TguuB52H14EMoSsCqDj7k8a/t7g4gVP+bgY5LY= -github.com/cs3org/reva v1.17.0 h1:UVdiiK6gUF2pb7qN9TKhMuf55sUKVnCcVjiVSzodluw= -github.com/cs3org/reva v1.17.0/go.mod h1:gtsVzMfDrUiUjH6qlHx+QqiRKsSYjVO6wEcCzANiqUg= github.com/cubewise-code/go-mime v0.0.0-20200519001935-8c5762b177d8 h1:Z9lwXumT5ACSmJ7WGnFl+OMLLjpz5uR2fyz7dC255FI= github.com/cubewise-code/go-mime v0.0.0-20200519001935-8c5762b177d8/go.mod h1:4abs/jPXcmJzYoYGF91JF9Uq9s/KL5n1jvFDix8KcqY= github.com/cyberdelia/templates v0.0.0-20141128023046-ca7fffd4298c/go.mod h1:GyV+0YP4qX0UQ7r2MoYZ+AvYDp12OF5yg4q8rGnyNh4= @@ -884,6 +882,8 @@ github.com/mendsley/gojwk v0.0.0-20141217222730-4d5ec6e58103 h1:Z/i1e+gTZrmcGeZy github.com/mendsley/gojwk v0.0.0-20141217222730-4d5ec6e58103/go.mod h1:o9YPB5aGP8ob35Vy6+vyq3P3bWe7NQWzf+JLiXCiMaE= github.com/mennanov/fieldmask-utils v0.5.0 h1:8em4akN0NM3hmmrg8VbvOPfdS4SSBdbFd53m9VtfOg0= github.com/mennanov/fieldmask-utils v0.5.0/go.mod h1:lah2lHczE2ff+7SqnNKpB+YzaO7M3h5iNO4LgPTJheM= +github.com/micbar/reva v1.9.1-0.20211211204557-ace6314aca5d h1:YjTtmls7nPPIa/kAdzlebZlk8bf/E3gA13RophdAFbs= +github.com/micbar/reva v1.9.1-0.20211211204557-ace6314aca5d/go.mod h1:gtsVzMfDrUiUjH6qlHx+QqiRKsSYjVO6wEcCzANiqUg= github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg= github.com/miekg/dns v1.1.26/go.mod h1:bPDLeHnStXmXAq1m/Ch/hvfNHr14JKNPMBo3VZKjuso= github.com/miekg/dns v1.1.40/go.mod h1:KNUDUusw/aVsxyTYZM1oqvCicbwhgbNgztCETuNZ7xM= diff --git a/graph/pkg/service/v0/drives.go b/graph/pkg/service/v0/drives.go index b4ca4b85bf..7a75629463 100644 --- a/graph/pkg/service/v0/drives.go +++ b/graph/pkg/service/v0/drives.go @@ -240,7 +240,7 @@ func (g Graph) CreateDrive(w http.ResponseWriter, r *http.Request) { newDrive, err := cs3StorageSpaceToDrive(wdu, resp.StorageSpace) if err != nil { - g.logger.Error().Err(err).Msg("error parsing url") + g.logger.Error().Err(err).Msg("error parsing space") errorcode.GeneralException.Render(w, r, http.StatusInternalServerError, err.Error()) return } @@ -322,11 +322,32 @@ func (g Graph) UpdateDrive(w http.ResponseWriter, r *http.Request) { } if resp.GetStatus().GetCode() != v1beta11.Code_CODE_OK { - errorcode.GeneralException.Render(w, r, http.StatusInternalServerError, resp.GetStatus().GetMessage()) + switch resp.Status.GetCode() { + case v1beta11.Code_CODE_NOT_FOUND: + errorcode.ItemNotFound.Render(w, r, http.StatusNotFound, resp.GetStatus().GetMessage()) + return + default: + errorcode.GeneralException.Render(w, r, http.StatusInternalServerError, resp.GetStatus().GetMessage()) + return + } + } + + wdu, err := url.Parse(g.config.Spaces.WebDavBase + g.config.Spaces.WebDavPath) + if err != nil { + g.logger.Error().Err(err).Msg("error parsing url") + errorcode.GeneralException.Render(w, r, http.StatusInternalServerError, err.Error()) return } - w.WriteHeader(http.StatusNoContent) + updatedDrive, err := cs3StorageSpaceToDrive(wdu, resp.StorageSpace) + if err != nil { + g.logger.Error().Err(err).Msg("error parsing space") + errorcode.GeneralException.Render(w, r, http.StatusInternalServerError, err.Error()) + return + } + + render.Status(r, http.StatusOK) + render.JSON(w, r, updatedDrive) } func cs3TimestampToTime(t *types.Timestamp) time.Time { From 7c2e70f8b670d7f8ae9e10dd01bcd97e2060a313 Mon Sep 17 00:00:00 2001 From: Michael Barz Date: Sun, 12 Dec 2021 17:34:04 +0100 Subject: [PATCH 36/56] fix api tests --- go.mod | 2 +- .../features/apiSpaces/changeSpaces.feature | 9 ++++----- .../acceptance/features/bootstrap/SpacesContext.php | 12 ++++++------ 3 files changed, 11 insertions(+), 12 deletions(-) diff --git a/go.mod b/go.mod index 350a659b41..d395cdd174 100644 --- a/go.mod +++ b/go.mod @@ -19,7 +19,7 @@ require ( github.com/blevesearch/bleve/v2 v2.2.2 github.com/coreos/go-oidc/v3 v3.1.0 github.com/cs3org/go-cs3apis v0.0.0-20211104090126-8e972dca8304 - github.com/cs3org/reva v1.17.0 + github.com/cs3org/reva v1.17.1-0.20211212151213-778de37266ff github.com/disintegration/imaging v1.6.2 github.com/glauth/glauth/v2 v2.0.0-20211021011345-ef3151c28733 github.com/go-chi/chi/v5 v5.0.7 diff --git a/tests/acceptance/features/apiSpaces/changeSpaces.feature b/tests/acceptance/features/apiSpaces/changeSpaces.feature index 8a00e08ef8..1397ace3f9 100644 --- a/tests/acceptance/features/apiSpaces/changeSpaces.feature +++ b/tests/acceptance/features/apiSpaces/changeSpaces.feature @@ -11,9 +11,9 @@ Feature: Change data of space And the administrator has given "Alice" the role "Admin" using the settings api Scenario: Alice changes a name of the space via the Graph api, she expects a 204 code and checks that the space name has changed - Given user "Alice" has created a space "Project Jupiter" of type "project" with quota "20" + Given user "Alice" has created a space "Project Jupiter" of type "project" with quota "20" When user "Alice" changes the name of the "Project Jupiter" space to "Project Death Star" - Then the HTTP status code should be "204" + Then the HTTP status code should be "200" When user "Alice" lists all available spaces via the GraphApi Then the json responded should contain a space "Project Death Star" with these key and value pairs: | key | value | @@ -23,12 +23,11 @@ Feature: Change data of space | root@@@webDavUrl | %base_url%/dav/spaces/%space_id% | Scenario: Alice increases quota of the space via the Graph api, she expects a 204 code and checks that the quota has changed - Given user "Alice" has created a space "Project Earth" of type "project" with quota "20" + Given user "Alice" has created a space "Project Earth" of type "project" with quota "20" When user "Alice" changes the quota of the "Project Earth" space to "100" - Then the HTTP status code should be "204" + Then the HTTP status code should be "200" When user "Alice" lists all available spaces via the GraphApi Then the json responded should contain a space "Project Earth" with these key and value pairs: | key | value | | name | Project Earth | | quota@@@total | 100 | - \ No newline at end of file diff --git a/tests/acceptance/features/bootstrap/SpacesContext.php b/tests/acceptance/features/bootstrap/SpacesContext.php index c58954105a..bfe573f16a 100644 --- a/tests/acceptance/features/bootstrap/SpacesContext.php +++ b/tests/acceptance/features/bootstrap/SpacesContext.php @@ -187,11 +187,11 @@ class SpacesContext implements Context { */ public function getSpaceByName(string $user, string $name): array { $this->theUserListsAllHisAvailableSpacesUsingTheGraphApi($user); - + $spaces = $this->getAvailableSpaces(); Assert::assertIsArray($spaces[$name], "Space with name $name for user $user not found"); Assert::assertNotEmpty($spaces[$name]["root"]["webDavUrl"], "WebDavUrl for space with name $name for user $user not found"); - + return $spaces[$name]; } @@ -802,7 +802,7 @@ class SpacesContext implements Context { string $spaceName ): void { $space = $this->getSpaceByName($user, $spaceName); - + $baseUrl = $this->featureContext->getBaseUrl(); if (!str_ends_with($baseUrl, '/')) { $baseUrl .= '/'; @@ -838,7 +838,7 @@ class SpacesContext implements Context { string $ownerUser ): void { $space = $this->getSpaceByName($ownerUser, $spaceName); - + $baseUrl = $this->featureContext->getBaseUrl(); if (!str_ends_with($baseUrl, '/')) { $baseUrl .= '/'; @@ -1044,7 +1044,7 @@ class SpacesContext implements Context { if (!str_ends_with($fullUrl, '/')) { $fullUrl .= '/'; } - $fullUrl .= "graph/v1.0/Drive($spaceId)"; + $fullUrl .= "graph/v1.0/drives/$spaceId"; $method = 'PATCH'; return HttpRequestHelper::sendRequest($fullUrl, $xRequestId, $method, $user, $password, $headers, $body); @@ -1090,7 +1090,7 @@ class SpacesContext implements Context { */ public function userHasUploadedFile(string $user, string $spaceName, string $fileContent, string $destination):void { $this->theUserListsAllHisAvailableSpacesUsingTheGraphApi($user); - + $space = $this->getSpaceByName($user, $spaceName); Assert::assertIsArray($space, "Space with name $spaceName not found"); Assert::assertNotEmpty($space["root"]["webDavUrl"], "WebDavUrl for space with name $spaceName not found"); From 3e693495fb33f36507b6ccc66bb0ed831dd1cc27 Mon Sep 17 00:00:00 2001 From: Michael Barz Date: Sun, 12 Dec 2021 21:57:38 +0100 Subject: [PATCH 37/56] adapt the docs --- docs/extensions/storage/spaces.md | 6 +++--- docs/ocis/adr/0007-api-for-spaces.md | 6 ++---- 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/docs/extensions/storage/spaces.md b/docs/extensions/storage/spaces.md index e877171727..45b445c79c 100644 --- a/docs/extensions/storage/spaces.md +++ b/docs/extensions/storage/spaces.md @@ -47,16 +47,16 @@ This is an extract of an element of the list spaces response. An entire object h Having introduced the above, one can refer to a Drive with the following URL format: ```console -'https://localhost:9200/graph/v1.0/Drive(1284d238-aa92-42ce-bdc4-0b0000009157!07c26b3a-9944-4f2b-ab33-b0b326fc7570") +'https://localhost:9200/graph/v1.0/drives/1284d238-aa92-42ce-bdc4-0b0000009157!07c26b3a-9944-4f2b-ab33-b0b326fc7570 ``` Updating an entity attribute: ```console -curl -X PATCH 'https://localhost:9200/graph/v1.0/Drive("1284d238-aa92-42ce-bdc4-0b0000009157!07c26b3a-9944-4f2b-ab33-b0b326fc7570)' -d '{"name":"42"}' -v +curl -X PATCH 'https://localhost:9200/graph/v1.0/drives/1284d238-aa92-42ce-bdc4-0b0000009157!07c26b3a-9944-4f2b-ab33-b0b326fc7570' -d '{"name":"42"}' -v ``` -The previous URL resource path segment (`Drive(1284d238-aa92-42ce-bdc4-0b0000009157!07c26b3a-9944-4f2b-ab33-b0b326fc7570)`) is parsed and handed over to the storage registry in order to apply the patch changes in the body, in this case update the space name attribute to `42`. Since space names are not unique we only support addressing them by their unique identifiers, any other query would render too ambiguous and explode in complexity. +The previous URL resource path segment (`1284d238-aa92-42ce-bdc4-0b0000009157!07c26b3a-9944-4f2b-ab33-b0b326fc7570`) is parsed and handed over to the storage registry in order to apply the patch changes in the body, in this case update the space name attribute to `42`. Since space names are not unique we only support addressing them by their unique identifiers, any other query would render too ambiguous and explode in complexity. ### Updating a space description diff --git a/docs/ocis/adr/0007-api-for-spaces.md b/docs/ocis/adr/0007-api-for-spaces.md index 0058a8bd48..7d4d42f4a5 100644 --- a/docs/ocis/adr/0007-api-for-spaces.md +++ b/docs/ocis/adr/0007-api-for-spaces.md @@ -45,7 +45,7 @@ This the DRAFT for the API. ownCloud servers provide an API to query for available spaces of an user. -See the openAPI Specification for the [open Graph API](https://owncloud.dev/open-graph-api/). +See the openAPI Specification for the [Libre Graph API](https://owncloud.dev/libre-graph-api/). Most important, the API returns the WebDAV endpoint for each space. With that, clients do not have to make assumptions about WebDAV routes any more. @@ -80,8 +80,7 @@ The reply to both calls is either one or a list of [Drive representation objects "owner": { "@odata.type": "microsoft.graph.identitySet" }, "quota": { "@odata.type": "microsoft.graph.quota" }, "root": { "@odata.type": "microsoft.graph.driveItem" }, - "webUrl": "url", - "ocCoOwner": [ { "@odata.type": "microsoft.graph.identitySet" } ] + "webUrl": "url" } ``` @@ -93,7 +92,6 @@ The meaning of the objects in Open Graph API context are: 4. **quota** - quota information about this space 5. **root** - the root driveItem object. 6. **webUrl** - The URL to make this space visible in the browser. -7. **ocCoOwner** - optional array owner objects of the co-owners of a space (*) The following *driveType* values are available in the first step, but might be enhanced later: From 59ed6dc0741b6f4d56afa33e43e3d4ce39e9112d Mon Sep 17 00:00:00 2001 From: Swikriti Tripathi Date: Mon, 13 Dec 2021 14:39:41 +0545 Subject: [PATCH 38/56] Bump commit id for tests --- .drone.env | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.drone.env b/.drone.env index 04c78573c2..c9b25b5748 100644 --- a/.drone.env +++ b/.drone.env @@ -1,5 +1,5 @@ # The test runner source for API tests -CORE_COMMITID=4bbd91de68aefdf94c03d2920a3353702f856280 +CORE_COMMITID=5f98837bd95d75660fc9fbc9a8802f7db8793a89 CORE_BRANCH=acceptance-test-changes-waiting-2021-11 # The test runner source for UI tests From b3eb8e1d86b8a9000758f7b22d8b882279b90a97 Mon Sep 17 00:00:00 2001 From: Willy Kloucek <34452982+wkloucek@users.noreply.github.com> Date: Mon, 13 Dec 2021 09:51:03 +0000 Subject: [PATCH 39/56] Automated changelog update [skip ci] --- CHANGELOG.md | 52 +++++++++++++++++++++++++++++++--------------------- 1 file changed, 31 insertions(+), 21 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bfe5f03436..6177572b13 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,31 @@ The following sections list the changes for unreleased. -[unreleased]: https://github.com/owncloud/ocis/compare/v1.15.0...master +[unreleased]: https://github.com/owncloud/ocis/compare/v1.16.0...master + +## Summary + +* Change - Update libre-graph-api to v0.3.0: [#2858](https://github.com/owncloud/ocis/pull/2858) +* Enhancement - Support signature auth in the public share auth middleware: [#2831](https://github.com/owncloud/ocis/pull/2831) + +## Details + +* Change - Update libre-graph-api to v0.3.0: [#2858](https://github.com/owncloud/ocis/pull/2858) + + This updates the libre-graph-api to use the latest spec and types. + + https://github.com/owncloud/ocis/pull/2858 + +* Enhancement - Support signature auth in the public share auth middleware: [#2831](https://github.com/owncloud/ocis/pull/2831) + + Enabled public share requests to be authenticated using the public share signature. + + https://github.com/owncloud/ocis/pull/2831 +# Changelog for [1.16.0] (2021-12-10) + +The following sections list the changes for 1.16.0. + +[1.16.0]: https://github.com/owncloud/ocis/compare/v1.15.0...v1.16.0 ## Summary @@ -12,13 +36,11 @@ The following sections list the changes for unreleased. * Bugfix - Use the CS3api up- and download workflow for the accounts service: [#2837](https://github.com/owncloud/ocis/pull/2837) * Change - Rename `APP_PROVIDER_BASIC_*` environment variables: [#2812](https://github.com/owncloud/ocis/pull/2812) * Change - Restructure Configuration Parsing: [#2708](https://github.com/owncloud/ocis/pull/2708) -* Change - Update libre-graph-api to v0.3.0: [#2858](https://github.com/owncloud/ocis/pull/2858) * Change - OIDC: fallback if IDP doesn't provide "preferred_username" claim: [#2644](https://github.com/owncloud/ocis/issues/2644) * Enhancement - Cleanup ocis-pkg config: [#2813](https://github.com/owncloud/ocis/pull/2813) -* Enhancement - Support signature auth in the public share auth middleware: [#2831](https://github.com/owncloud/ocis/pull/2831) * Enhancement - Correct shutdown of services under runtime: [#2843](https://github.com/owncloud/ocis/pull/2843) * Enhancement - Update REVA to v1.17.0: [#2849](https://github.com/owncloud/ocis/pull/2849) -* Enhancement - Update ownCloud Web to v4.6.0: [#2846](https://github.com/owncloud/ocis/pull/2846) +* Enhancement - Update ownCloud Web to v4.6.1: [#2846](https://github.com/owncloud/ocis/pull/2846) ## Details @@ -77,12 +99,6 @@ The following sections list the changes for unreleased. https://github.com/owncloud/ocis/pull/2708 -* Change - Update libre-graph-api to v0.3.0: [#2858](https://github.com/owncloud/ocis/pull/2858) - - This updates the libre-graph-api to use the latest spec and types. - - https://github.com/owncloud/ocis/pull/2858 - * Change - OIDC: fallback if IDP doesn't provide "preferred_username" claim: [#2644](https://github.com/owncloud/ocis/issues/2644) Some IDPs don't add the "preferred_username" claim. Fallback to the "email" claim in that case @@ -95,12 +111,6 @@ The following sections list the changes for unreleased. https://github.com/owncloud/ocis/pull/2813 -* Enhancement - Support signature auth in the public share auth middleware: [#2831](https://github.com/owncloud/ocis/pull/2831) - - Enabled public share requests to be authenticated using the public share signature. - - https://github.com/owncloud/ocis/pull/2831 - * Enhancement - Correct shutdown of services under runtime: [#2843](https://github.com/owncloud/ocis/pull/2843) Supervised goroutines now shut themselves down on context cancellation propagation. @@ -114,7 +124,7 @@ The following sections list the changes for unreleased. * Fix [cs3org/reva#2305](https://github.com/cs3org/reva/pull/2305): Make sure /app/new takes `target` as absolute path * Fix [cs3org/reva#2303](https://github.com/cs3org/reva/pull/2303): Fix content disposition header for public links files * Fix [cs3org/reva#2316](https://github.com/cs3org/reva/pull/2316): Fix the share types in propfinds - * Fix [cs3org/reva#2803](https://github.com/cs3org/reva/pull/2803): Fix app provider for editor public links + * Fix [cs3org/reva#2803](https://github.com/cs3org/reva/pull/2310): Fix app provider for editor public links * Fix [cs3org/reva#2298](https://github.com/cs3org/reva/pull/2298): Remove share refs from trashbin * Fix [cs3org/reva#2309](https://github.com/cs3org/reva/pull/2309): Remove early finish for zero byte file uploads * Fix [cs3org/reva#1941](https://github.com/cs3org/reva/pull/1941): Fix TUS uploads with transfer token only @@ -122,7 +132,7 @@ The following sections list the changes for unreleased. * Enh [cs3org/reva#2217](https://github.com/cs3org/reva/pull/2217): OIDC auth driver for ESCAPE IAM * Enh [cs3org/reva#2256](https://github.com/cs3org/reva/pull/2256): Return user type in the response of the ocs GET user call * Enh [cs3org/reva#2315](https://github.com/cs3org/reva/pull/2315): Add new attributes to public link propfinds - * Enh [cs3org/reva#2740](https://github.com/cs3org/reva/pull/2740): Implement space membership endpoints + * Enh [cs3org/reva#2740](https://github.com/cs3org/reva/pull/2250): Implement space membership endpoints * Enh [cs3org/reva#2252](https://github.com/cs3org/reva/pull/2252): Add the xattr sys.acl to SysACL (eosgrpc) * Enh [cs3org/reva#2314](https://github.com/cs3org/reva/pull/2314): OIDC: fallback if IDP doesn't provide "preferred_username" claim @@ -130,15 +140,15 @@ The following sections list the changes for unreleased. https://github.com/owncloud/ocis/pull/2835 https://github.com/owncloud/ocis/pull/2837 -* Enhancement - Update ownCloud Web to v4.6.0: [#2846](https://github.com/owncloud/ocis/pull/2846) +* Enhancement - Update ownCloud Web to v4.6.1: [#2846](https://github.com/owncloud/ocis/pull/2846) Tags: web - We updated ownCloud Web to v4.6.0. Please refer to the changelog (linked) for details on the web + We updated ownCloud Web to v4.6.1. Please refer to the changelog (linked) for details on the web release. https://github.com/owncloud/ocis/pull/2846 - https://github.com/owncloud/web/releases/tag/v4.6.0 + https://github.com/owncloud/web/releases/tag/v4.6.1 # Changelog for [1.15.0] (2021-11-19) The following sections list the changes for 1.15.0. From 23458ba4b9c13663f2e77c11d0782c7661beed12 Mon Sep 17 00:00:00 2001 From: Michael Barz Date: Mon, 13 Dec 2021 16:22:28 +0100 Subject: [PATCH 40/56] remove unecessary replace --- go.mod | 2 -- go.sum | 4 ++-- graph/pkg/service/v0/drives.go | 4 ++-- 3 files changed, 4 insertions(+), 6 deletions(-) diff --git a/go.mod b/go.mod index d395cdd174..b961c53233 100644 --- a/go.mod +++ b/go.mod @@ -241,5 +241,3 @@ require ( stash.kopano.io/kgol/kcc-go/v5 v5.0.1 // indirect stash.kopano.io/kgol/oidc-go v0.3.2 // indirect ) - -replace github.com/cs3org/reva => github.com/micbar/reva v1.9.1-0.20211211204557-ace6314aca5d diff --git a/go.sum b/go.sum index be2e4c97f3..0754b535f5 100644 --- a/go.sum +++ b/go.sum @@ -299,6 +299,8 @@ github.com/crewjam/saml v0.4.5/go.mod h1:qCJQpUtZte9R1ZjUBcW8qtCNlinbO363ooNl02S github.com/cs3org/cato v0.0.0-20200828125504-e418fc54dd5e/go.mod h1:XJEZ3/EQuI3BXTp/6DUzFr850vlxq11I6satRtz0YQ4= github.com/cs3org/go-cs3apis v0.0.0-20211104090126-8e972dca8304 h1:e/nIPR518vyvrulo9goAZTtYD6gFfu/2/9MDe6mTGcw= github.com/cs3org/go-cs3apis v0.0.0-20211104090126-8e972dca8304/go.mod h1:UXha4TguuB52H14EMoSsCqDj7k8a/t7g4gVP+bgY5LY= +github.com/cs3org/reva v1.17.1-0.20211212151213-778de37266ff h1:IUj5GfxDa8WgNzeH+5MnJREzjoneejygmnMgU2hAWCg= +github.com/cs3org/reva v1.17.1-0.20211212151213-778de37266ff/go.mod h1:gtsVzMfDrUiUjH6qlHx+QqiRKsSYjVO6wEcCzANiqUg= github.com/cubewise-code/go-mime v0.0.0-20200519001935-8c5762b177d8 h1:Z9lwXumT5ACSmJ7WGnFl+OMLLjpz5uR2fyz7dC255FI= github.com/cubewise-code/go-mime v0.0.0-20200519001935-8c5762b177d8/go.mod h1:4abs/jPXcmJzYoYGF91JF9Uq9s/KL5n1jvFDix8KcqY= github.com/cyberdelia/templates v0.0.0-20141128023046-ca7fffd4298c/go.mod h1:GyV+0YP4qX0UQ7r2MoYZ+AvYDp12OF5yg4q8rGnyNh4= @@ -882,8 +884,6 @@ github.com/mendsley/gojwk v0.0.0-20141217222730-4d5ec6e58103 h1:Z/i1e+gTZrmcGeZy github.com/mendsley/gojwk v0.0.0-20141217222730-4d5ec6e58103/go.mod h1:o9YPB5aGP8ob35Vy6+vyq3P3bWe7NQWzf+JLiXCiMaE= github.com/mennanov/fieldmask-utils v0.5.0 h1:8em4akN0NM3hmmrg8VbvOPfdS4SSBdbFd53m9VtfOg0= github.com/mennanov/fieldmask-utils v0.5.0/go.mod h1:lah2lHczE2ff+7SqnNKpB+YzaO7M3h5iNO4LgPTJheM= -github.com/micbar/reva v1.9.1-0.20211211204557-ace6314aca5d h1:YjTtmls7nPPIa/kAdzlebZlk8bf/E3gA13RophdAFbs= -github.com/micbar/reva v1.9.1-0.20211211204557-ace6314aca5d/go.mod h1:gtsVzMfDrUiUjH6qlHx+QqiRKsSYjVO6wEcCzANiqUg= github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg= github.com/miekg/dns v1.1.26/go.mod h1:bPDLeHnStXmXAq1m/Ch/hvfNHr14JKNPMBo3VZKjuso= github.com/miekg/dns v1.1.40/go.mod h1:KNUDUusw/aVsxyTYZM1oqvCicbwhgbNgztCETuNZ7xM= diff --git a/graph/pkg/service/v0/drives.go b/graph/pkg/service/v0/drives.go index 7a75629463..d46a6139ec 100644 --- a/graph/pkg/service/v0/drives.go +++ b/graph/pkg/service/v0/drives.go @@ -249,10 +249,10 @@ func (g Graph) CreateDrive(w http.ResponseWriter, r *http.Request) { } func (g Graph) UpdateDrive(w http.ResponseWriter, r *http.Request) { - driveID := chi.URLParam(r, "driveID") - driveID, err := url.PathUnescape(driveID) + driveID, err := url.PathUnescape(chi.URLParam(r, "driveID")) if err != nil { errorcode.InvalidRequest.Render(w, r, http.StatusBadRequest, "unescaping drive id failed") + return } if driveID == "" { From cd732fcbbca51b75c786c93e21a607b6d02142b6 Mon Sep 17 00:00:00 2001 From: Michael Barz Date: Mon, 13 Dec 2021 19:04:08 +0000 Subject: [PATCH 41/56] Automated changelog update [skip ci] --- CHANGELOG.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6177572b13..a3afca8451 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,7 @@ The following sections list the changes for unreleased. ## Summary * Change - Update libre-graph-api to v0.3.0: [#2858](https://github.com/owncloud/ocis/pull/2858) +* Change - Return not found when updating non existent space: [#2869](https://github.com/cs3org/reva/pull/2869) * Enhancement - Support signature auth in the public share auth middleware: [#2831](https://github.com/owncloud/ocis/pull/2831) ## Details @@ -17,6 +18,12 @@ The following sections list the changes for unreleased. https://github.com/owncloud/ocis/pull/2858 +* Change - Return not found when updating non existent space: [#2869](https://github.com/cs3org/reva/pull/2869) + + If a spaceid of a space which is updated doesn't exist, handle it as a not found error. + + https://github.com/cs3org/reva/pull/2869 + * Enhancement - Support signature auth in the public share auth middleware: [#2831](https://github.com/owncloud/ocis/pull/2831) Enabled public share requests to be authenticated using the public share signature. From 5af08c516bf5e2d582245b7c052d1a1b8ea80d99 Mon Sep 17 00:00:00 2001 From: Phil Davis Date: Tue, 14 Dec 2021 10:10:13 +0545 Subject: [PATCH 42/56] [tests-only] Minor local API test changes --- tests/acceptance/config/behat.yml | 16 +++------------- .../features/bootstrap/ArchiverContext.php | 3 +-- .../features/bootstrap/RevaContext.php | 2 +- .../features/bootstrap/SpacesContext.php | 10 +++++----- 4 files changed, 10 insertions(+), 21 deletions(-) diff --git a/tests/acceptance/config/behat.yml b/tests/acceptance/config/behat.yml index 016de39b49..02d11fb3bb 100644 --- a/tests/acceptance/config/behat.yml +++ b/tests/acceptance/config/behat.yml @@ -6,7 +6,7 @@ default: apiAccountsHashDifficulty: paths: - '%paths.base%/../features/apiAccountsHashDifficulty' - context: &common_ldap_suite_context + context: parameters: ldapAdminPassword: admin ldapUsersOU: TestUsers @@ -34,12 +34,7 @@ default: contexts: - SpacesContext: - OccContext: - - FeatureContext: &common_feature_context_params - baseUrl: http://localhost:8080 - adminUsername: admin - adminPassword: admin - regularUserPassword: 123456 - ocPath: apps/testing/api/v1/occ + - FeatureContext: *common_feature_context_params - CapabilitiesContext: - ChecksumContext: - FavoritesContext: @@ -54,12 +49,7 @@ default: contexts: - ArchiverContext: - OccContext: - - FeatureContext: &common_feature_context_params - baseUrl: http://localhost:8080 - adminUsername: admin - adminPassword: admin - regularUserPassword: 123456 - ocPath: apps/testing/api/v1/occ + - FeatureContext: *common_feature_context_params - CapabilitiesContext: - ChecksumContext: - FilesVersionsContext: diff --git a/tests/acceptance/features/bootstrap/ArchiverContext.php b/tests/acceptance/features/bootstrap/ArchiverContext.php index 4ebd96b267..3be54893ad 100644 --- a/tests/acceptance/features/bootstrap/ArchiverContext.php +++ b/tests/acceptance/features/bootstrap/ArchiverContext.php @@ -38,7 +38,7 @@ class ArchiverContext implements Context { /** * @var FeatureContext */ - private $featureContext; + private FeatureContext $featureContext; /** * @BeforeScenario @@ -80,7 +80,6 @@ class ArchiverContext implements Context { case 'id': case 'ids': return 'id=' . $this->featureContext->getFileIdForPath($user, $resource); - break; case 'path': case 'paths': return 'path=' . $resource; diff --git a/tests/acceptance/features/bootstrap/RevaContext.php b/tests/acceptance/features/bootstrap/RevaContext.php index b74ff023d2..fbf77172fc 100644 --- a/tests/acceptance/features/bootstrap/RevaContext.php +++ b/tests/acceptance/features/bootstrap/RevaContext.php @@ -34,7 +34,7 @@ class RevaContext implements Context { /** * @var FeatureContext */ - private $featureContext; + private FeatureContext $featureContext; /** * @BeforeScenario diff --git a/tests/acceptance/features/bootstrap/SpacesContext.php b/tests/acceptance/features/bootstrap/SpacesContext.php index bfe573f16a..5dec1726a2 100644 --- a/tests/acceptance/features/bootstrap/SpacesContext.php +++ b/tests/acceptance/features/bootstrap/SpacesContext.php @@ -185,14 +185,14 @@ class SpacesContext implements Context { * * @return array */ - public function getSpaceByName(string $user, string $name): array { + public function getSpaceByName(string $user, string $spaceName): array { $this->theUserListsAllHisAvailableSpacesUsingTheGraphApi($user); $spaces = $this->getAvailableSpaces(); - Assert::assertIsArray($spaces[$name], "Space with name $name for user $user not found"); - Assert::assertNotEmpty($spaces[$name]["root"]["webDavUrl"], "WebDavUrl for space with name $name for user $user not found"); + Assert::assertIsArray($spaces[$spaceName], "Space with name $spaceName for user $user not found"); + Assert::assertNotEmpty($spaces[$spaceName]["root"]["webDavUrl"], "WebDavUrl for space with name $spaceName for user $user not found"); - return $spaces[$name]; + return $spaces[$spaceName]; } /** @@ -604,7 +604,7 @@ class SpacesContext implements Context { * * @throws Exception|GuzzleException */ - public function usertheSpaceShouldContainEntries( + public function userTheSpaceShouldContainEntries( string $user, string $spaceName, string $shouldOrNot, From 74c51afa88cffab7ac7728555c381e092edc0c5b Mon Sep 17 00:00:00 2001 From: Phil Davis Date: Tue, 14 Dec 2021 12:34:35 +0545 Subject: [PATCH 43/56] Bump core commit id to include new group name tests --- .drone.env | 2 +- tests/acceptance/expected-failures-API-on-OCIS-storage.md | 8 ++++++++ 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/.drone.env b/.drone.env index c9b25b5748..fd230a574b 100644 --- a/.drone.env +++ b/.drone.env @@ -1,5 +1,5 @@ # The test runner source for API tests -CORE_COMMITID=5f98837bd95d75660fc9fbc9a8802f7db8793a89 +CORE_COMMITID=2987544bfd6f6823df68f6ea392ba41c1ea4f07c CORE_BRANCH=acceptance-test-changes-waiting-2021-11 # The test runner source for UI tests diff --git a/tests/acceptance/expected-failures-API-on-OCIS-storage.md b/tests/acceptance/expected-failures-API-on-OCIS-storage.md index b548cdd78a..1f3c7b46e5 100644 --- a/tests/acceptance/expected-failures-API-on-OCIS-storage.md +++ b/tests/acceptance/expected-failures-API-on-OCIS-storage.md @@ -983,6 +983,14 @@ _ocs: api compatibility, return correct status code_ - [apiProvisioningGroups-v1/addGroup.feature:134](https://github.com/owncloud/core/blob/master/tests/acceptance/features/apiProvisioningGroups-v1/addGroup.feature#L134) - [apiProvisioningGroups-v2/addGroup.feature:129](https://github.com/owncloud/core/blob/master/tests/acceptance/features/apiProvisioningGroups-v2/addGroup.feature#L129) +#### [group names with space at the start or end should not be allowed](https://github.com/owncloud/ocis/issues/2876) +- [apiProvisioningGroups-v1/addGroup.feature:158](https://github.com/owncloud/core/blob/master/tests/acceptance/features/apiProvisioningGroups-v1/addGroup.feature#L158) +- [apiProvisioningGroups-v1/addGroup.feature:166](https://github.com/owncloud/core/blob/master/tests/acceptance/features/apiProvisioningGroups-v1/addGroup.feature#L166) +- [apiProvisioningGroups-v1/addGroup.feature:174](https://github.com/owncloud/core/blob/master/tests/acceptance/features/apiProvisioningGroups-v1/addGroup.feature#L174) +- [apiProvisioningGroups-v2/addGroup.feature:154](https://github.com/owncloud/core/blob/master/tests/acceptance/features/apiProvisioningGroups-v2/addGroup.feature#L154) +- [apiProvisioningGroups-v2/addGroup.feature:162](https://github.com/owncloud/core/blob/master/tests/acceptance/features/apiProvisioningGroups-v2/addGroup.feature#L162) +- [apiProvisioningGroups-v2/addGroup.feature:170](https://github.com/owncloud/core/blob/master/tests/acceptance/features/apiProvisioningGroups-v2/addGroup.feature#L170) + #### [cannot create group with '/'](https://github.com/owncloud/product/issues/285) - [apiProvisioningGroups-v1/addToGroup.feature:82](https://github.com/owncloud/core/blob/master/tests/acceptance/features/apiProvisioningGroups-v1/addToGroup.feature#L82) - [apiProvisioningGroups-v1/deleteGroup.feature:85](https://github.com/owncloud/core/blob/master/tests/acceptance/features/apiProvisioningGroups-v1/deleteGroup.feature#L85) From c1d3867239a4b6766f29bfa531efdf03d6a1e6d2 Mon Sep 17 00:00:00 2001 From: Willy Kloucek Date: Tue, 14 Dec 2021 12:54:22 +0100 Subject: [PATCH 44/56] update gologr --- glauth/pkg/mlogr/mlogr.go | 40 ++++++++++++++++++--------------------- go.mod | 2 +- go.sum | 2 ++ 3 files changed, 21 insertions(+), 23 deletions(-) diff --git a/glauth/pkg/mlogr/mlogr.go b/glauth/pkg/mlogr/mlogr.go index fd1cbfb554..61cdc8de39 100644 --- a/glauth/pkg/mlogr/mlogr.go +++ b/glauth/pkg/mlogr/mlogr.go @@ -14,23 +14,28 @@ const traceVerbosity = 8 // New returns a logr.Logger which is implemented by the log. func New(l *plog.Logger) logr.Logger { - return logger{ + sink := logSink{ l: l, verbosity: 0, prefix: "glauth", values: nil, } + + return logr.New(sink) } -// logger is a logr.Logger that uses the ocis-pkg log. -type logger struct { +func (l logSink) Init(info logr.RuntimeInfo) { +} + +// logSink is a logr.LogSink that uses the ocis-pkg log. +type logSink struct { l *plog.Logger verbosity int prefix string values []interface{} } -func (l logger) clone() logger { +func (l logSink) clone() logSink { out := l out.values = copySlice(l.values) return out @@ -71,8 +76,8 @@ func add(e *zerolog.Event, keysAndVals []interface{}) { } } -func (l logger) Info(msg string, keysAndVals ...interface{}) { - if l.Enabled() { +func (l logSink) Info(level int, msg string, kvList ...interface{}) { + if l.Enabled(level) { var e *zerolog.Event if l.verbosity < debugVerbosity { e = l.l.Info() @@ -86,16 +91,16 @@ func (l logger) Info(msg string, keysAndVals ...interface{}) { e.Str("name", l.prefix) } add(e, l.values) - add(e, keysAndVals) + add(e, kvList) e.Msg(msg) } } -func (l logger) Enabled() bool { +func (l logSink) Enabled(level int) bool { return true } -func (l logger) Error(err error, msg string, keysAndVals ...interface{}) { +func (l logSink) Error(err error, msg string, keysAndVals ...interface{}) { e := l.l.Error().Err(err) if l.prefix != "" { e.Str("name", l.prefix) @@ -105,18 +110,10 @@ func (l logger) Error(err error, msg string, keysAndVals ...interface{}) { e.Msg(msg) } -func (l logger) V(verbosity int) logr.InfoLogger { - //new := l.clone() - //new.level = level - //return new - l.verbosity = verbosity - return l -} - -// WithName returns a new logr.Logger with the specified name appended. zerologr +// WithName returns a new logr.LogSink with the specified name appended. zerologr // uses '/' characters to separate name elements. Callers should not pass '/' // in the provided name string, but this library does not actually enforce that. -func (l logger) WithName(name string) logr.Logger { +func (l logSink) WithName(name string) logr.LogSink { nl := l.clone() if len(l.prefix) > 0 { nl.prefix = l.prefix + "/" @@ -124,11 +121,10 @@ func (l logger) WithName(name string) logr.Logger { nl.prefix += name return nl } -func (l logger) WithValues(kvList ...interface{}) logr.Logger { +func (l logSink) WithValues(kvList ...interface{}) logr.LogSink { nl := l.clone() nl.values = append(nl.values, kvList...) return nl } -var _ logr.Logger = logger{} -var _ logr.InfoLogger = logger{} +var _ logr.LogSink = logSink{} diff --git a/go.mod b/go.mod index 4805f9e916..31a8774708 100644 --- a/go.mod +++ b/go.mod @@ -26,7 +26,7 @@ require ( github.com/go-chi/cors v1.2.0 github.com/go-chi/render v1.0.1 github.com/go-ldap/ldap/v3 v3.4.1 - github.com/go-logr/logr v0.4.0 + github.com/go-logr/logr v1.2.2 github.com/go-ozzo/ozzo-validation/v4 v4.3.0 github.com/gofrs/uuid v4.2.0+incompatible github.com/golang-jwt/jwt/v4 v4.2.0 diff --git a/go.sum b/go.sum index 514b84a0ad..b5ce5c5b65 100644 --- a/go.sum +++ b/go.sum @@ -415,6 +415,8 @@ github.com/go-logfmt/logfmt v0.5.1 h1:otpy5pqBCBZ1ng9RQ0dPu4PN7ba75Y/aA+UpowDyNV github.com/go-logfmt/logfmt v0.5.1/go.mod h1:WYhtIu8zTZfxdn5+rREduYbwxfcBr/Vr6KEVveWlfTs= github.com/go-logr/logr v0.4.0 h1:K7/B1jt6fIBQVd4Owv2MqGQClcgf0R266+7C/QjRcLc= github.com/go-logr/logr v0.4.0/go.mod h1:z6/tIYblkpsD+a4lm/fGIIU9mZ+XfAiaFtq7xTgseGU= +github.com/go-logr/logr v1.2.2 h1:ahHml/yUpnlb96Rp8HCvtYVPY8ZYpxq3g7UYchIYwbs= +github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= github.com/go-openapi/analysis v0.0.0-20180825180245-b006789cd277/go.mod h1:k70tL6pCuVxPJOHXQ+wIac1FUrvNkHolPie/cLEU6hI= github.com/go-openapi/analysis v0.17.0/go.mod h1:IowGgpVeD0vNm45So8nr+IcQ3pxVtpRoBWb8PVZO0ik= github.com/go-openapi/analysis v0.18.0/go.mod h1:IowGgpVeD0vNm45So8nr+IcQ3pxVtpRoBWb8PVZO0ik= From 5859be13f21e29e2b03d31a936fd18508b278e80 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 14 Dec 2021 14:03:51 +0000 Subject: [PATCH 45/56] Bump go.opentelemetry.io/otel from 1.2.0 to 1.3.0 Bumps [go.opentelemetry.io/otel](https://github.com/open-telemetry/opentelemetry-go) from 1.2.0 to 1.3.0. - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.2.0...v1.3.0) --- updated-dependencies: - dependency-name: go.opentelemetry.io/otel dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- go.mod | 5 +++-- go.sum | 11 ++++++++--- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/go.mod b/go.mod index 31a8774708..45f4a67bfa 100644 --- a/go.mod +++ b/go.mod @@ -59,10 +59,10 @@ require ( go-micro.dev/v4 v4.4.0 go.opencensus.io v0.23.0 go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.27.0 - go.opentelemetry.io/otel v1.2.0 + go.opentelemetry.io/otel v1.3.0 go.opentelemetry.io/otel/exporters/jaeger v1.2.0 go.opentelemetry.io/otel/sdk v1.2.0 - go.opentelemetry.io/otel/trace v1.2.0 + go.opentelemetry.io/otel/trace v1.3.0 golang.org/x/crypto v0.0.0-20211108221036-ceb1ce70b4fa golang.org/x/image v0.0.0-20210628002857-a66eb6448b8d golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8 @@ -133,6 +133,7 @@ require ( github.com/go-git/go-git/v5 v5.4.2 // indirect github.com/go-kit/log v0.2.0 // indirect github.com/go-logfmt/logfmt v0.5.1 // indirect + github.com/go-logr/stdr v1.2.0 // indirect github.com/go-sql-driver/mysql v1.6.0 // indirect github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0 // indirect github.com/gogo/protobuf v1.3.2 // indirect diff --git a/go.sum b/go.sum index b5ce5c5b65..6425f6c4f8 100644 --- a/go.sum +++ b/go.sum @@ -413,10 +413,13 @@ github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG1KdI/P7A= github.com/go-logfmt/logfmt v0.5.1 h1:otpy5pqBCBZ1ng9RQ0dPu4PN7ba75Y/aA+UpowDyNVA= github.com/go-logfmt/logfmt v0.5.1/go.mod h1:WYhtIu8zTZfxdn5+rREduYbwxfcBr/Vr6KEVveWlfTs= -github.com/go-logr/logr v0.4.0 h1:K7/B1jt6fIBQVd4Owv2MqGQClcgf0R266+7C/QjRcLc= github.com/go-logr/logr v0.4.0/go.mod h1:z6/tIYblkpsD+a4lm/fGIIU9mZ+XfAiaFtq7xTgseGU= +github.com/go-logr/logr v1.2.0/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= +github.com/go-logr/logr v1.2.1/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= github.com/go-logr/logr v1.2.2 h1:ahHml/yUpnlb96Rp8HCvtYVPY8ZYpxq3g7UYchIYwbs= github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= +github.com/go-logr/stdr v1.2.0 h1:j4LrlVXgrbIWO83mmQUnK0Hi+YnbD+vzrE1z/EphbFE= +github.com/go-logr/stdr v1.2.0/go.mod h1:YkVgnZu1ZjjL7xTxrfm/LLZBfkhTqSR1ydtm6jTKKwI= github.com/go-openapi/analysis v0.0.0-20180825180245-b006789cd277/go.mod h1:k70tL6pCuVxPJOHXQ+wIac1FUrvNkHolPie/cLEU6hI= github.com/go-openapi/analysis v0.17.0/go.mod h1:IowGgpVeD0vNm45So8nr+IcQ3pxVtpRoBWb8PVZO0ik= github.com/go-openapi/analysis v0.18.0/go.mod h1:IowGgpVeD0vNm45So8nr+IcQ3pxVtpRoBWb8PVZO0ik= @@ -1312,14 +1315,16 @@ go.opencensus.io v0.23.0 h1:gqCw0LfLxScz8irSi8exQc7fyQ0fKQU/qnC/X8+V/1M= go.opencensus.io v0.23.0/go.mod h1:XItmlyltB5F7CS4xOC1DcqMoFqwtC6OG2xF7mCv7P7E= go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.27.0 h1:TON1iU3Y5oIytGQHIejDYLam5uoSMsmA0UV9Yupb5gQ= go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.27.0/go.mod h1:T/zQwBldOpoAEpE3HMbLnI8ydESZVz4ggw6Is4FF9LI= -go.opentelemetry.io/otel v1.2.0 h1:YOQDvxO1FayUcT9MIhJhgMyNO1WqoduiyvQHzGN0kUQ= go.opentelemetry.io/otel v1.2.0/go.mod h1:aT17Fk0Z1Nor9e0uisf98LrntPGMnk4frBO9+dkf69I= +go.opentelemetry.io/otel v1.3.0 h1:APxLf0eiBwLl+SOXiJJCVYzA1OOJNyAoV8C5RNRyy7Y= +go.opentelemetry.io/otel v1.3.0/go.mod h1:PWIKzi6JCp7sM0k9yZ43VX+T345uNbAkDKwHVjb2PTs= go.opentelemetry.io/otel/exporters/jaeger v1.2.0 h1:C/5Egj3MJBXRJi22cSl07suqPqtZLnLFmH//OxETUEc= go.opentelemetry.io/otel/exporters/jaeger v1.2.0/go.mod h1:KJLFbEMKTNPIfOxcg/WikIozEoKcPgJRz3Ce1vLlM8E= go.opentelemetry.io/otel/sdk v1.2.0 h1:wKN260u4DesJYhyjxDa7LRFkuhH7ncEVKU37LWcyNIo= go.opentelemetry.io/otel/sdk v1.2.0/go.mod h1:jNN8QtpvbsKhgaC6V5lHiejMoKD+V8uadoSafgHPx1U= -go.opentelemetry.io/otel/trace v1.2.0 h1:Ys3iqbqZhcf28hHzrm5WAquMkDHNZTUkw7KHbuNjej0= go.opentelemetry.io/otel/trace v1.2.0/go.mod h1:N5FLswTubnxKxOJHM7XZC074qpeEdLy3CgAVsdMucK0= +go.opentelemetry.io/otel/trace v1.3.0 h1:doy8Hzb1RJ+I3yFhtDmwNc7tIyw1tNMOIsyPzp1NOGY= +go.opentelemetry.io/otel/trace v1.3.0/go.mod h1:c/VDhno8888bvQYmbYLqe41/Ldmr/KKunbvWM4/fEjk= go.opentelemetry.io/proto/otlp v0.7.0/go.mod h1:PqfVotwruBrMGOCsRd/89rSnXhoiJIqeYNgFYFoEGnI= go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= From a3a37a2ad54d4a27d40d1328e703818256453ddf Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 14 Dec 2021 14:44:27 +0000 Subject: [PATCH 46/56] Bump go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc Bumps [go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc](https://github.com/open-telemetry/opentelemetry-go-contrib) from 0.27.0 to 0.28.0. - [Release notes](https://github.com/open-telemetry/opentelemetry-go-contrib/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go-contrib/blob/main/CHANGELOG.md) - [Commits](https://github.com/open-telemetry/opentelemetry-go-contrib/compare/zpages/v0.27.0...zpages/v0.28.0) --- updated-dependencies: - dependency-name: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- go.mod | 2 +- go.sum | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/go.mod b/go.mod index 45f4a67bfa..798f76f217 100644 --- a/go.mod +++ b/go.mod @@ -58,7 +58,7 @@ require ( github.com/yaegashi/msgraph.go v0.1.4 go-micro.dev/v4 v4.4.0 go.opencensus.io v0.23.0 - go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.27.0 + go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.28.0 go.opentelemetry.io/otel v1.3.0 go.opentelemetry.io/otel/exporters/jaeger v1.2.0 go.opentelemetry.io/otel/sdk v1.2.0 diff --git a/go.sum b/go.sum index 6425f6c4f8..6edb730c57 100644 --- a/go.sum +++ b/go.sum @@ -1313,8 +1313,9 @@ go.opencensus.io v0.22.4/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= go.opencensus.io v0.22.5/go.mod h1:5pWMHQbX5EPX2/62yrJeAkowc+lfs/XD7Uxpq3pI6kk= go.opencensus.io v0.23.0 h1:gqCw0LfLxScz8irSi8exQc7fyQ0fKQU/qnC/X8+V/1M= go.opencensus.io v0.23.0/go.mod h1:XItmlyltB5F7CS4xOC1DcqMoFqwtC6OG2xF7mCv7P7E= -go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.27.0 h1:TON1iU3Y5oIytGQHIejDYLam5uoSMsmA0UV9Yupb5gQ= go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.27.0/go.mod h1:T/zQwBldOpoAEpE3HMbLnI8ydESZVz4ggw6Is4FF9LI= +go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.28.0 h1:Ky1MObd188aGbgb5OgNnwGuEEwI9MVIcc7rBW6zk5Ak= +go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.28.0/go.mod h1:vEhqr0m4eTc+DWxfsXoXue2GBgV2uUwVznkGIHW/e5w= go.opentelemetry.io/otel v1.2.0/go.mod h1:aT17Fk0Z1Nor9e0uisf98LrntPGMnk4frBO9+dkf69I= go.opentelemetry.io/otel v1.3.0 h1:APxLf0eiBwLl+SOXiJJCVYzA1OOJNyAoV8C5RNRyy7Y= go.opentelemetry.io/otel v1.3.0/go.mod h1:PWIKzi6JCp7sM0k9yZ43VX+T345uNbAkDKwHVjb2PTs= From b796a96d3b3bae13911ab93fc44ad3fc36271d9d Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 14 Dec 2021 15:12:39 +0000 Subject: [PATCH 47/56] Bump go.opentelemetry.io/otel/exporters/jaeger from 1.2.0 to 1.3.0 Bumps [go.opentelemetry.io/otel/exporters/jaeger](https://github.com/open-telemetry/opentelemetry-go) from 1.2.0 to 1.3.0. - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.2.0...v1.3.0) --- updated-dependencies: - dependency-name: go.opentelemetry.io/otel/exporters/jaeger dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- go.mod | 4 ++-- go.sum | 6 ++++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/go.mod b/go.mod index 45f4a67bfa..e5766a80ef 100644 --- a/go.mod +++ b/go.mod @@ -60,8 +60,8 @@ require ( go.opencensus.io v0.23.0 go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.27.0 go.opentelemetry.io/otel v1.3.0 - go.opentelemetry.io/otel/exporters/jaeger v1.2.0 - go.opentelemetry.io/otel/sdk v1.2.0 + go.opentelemetry.io/otel/exporters/jaeger v1.3.0 + go.opentelemetry.io/otel/sdk v1.3.0 go.opentelemetry.io/otel/trace v1.3.0 golang.org/x/crypto v0.0.0-20211108221036-ceb1ce70b4fa golang.org/x/image v0.0.0-20210628002857-a66eb6448b8d diff --git a/go.sum b/go.sum index 6425f6c4f8..d3bec35d48 100644 --- a/go.sum +++ b/go.sum @@ -1318,10 +1318,12 @@ go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.2 go.opentelemetry.io/otel v1.2.0/go.mod h1:aT17Fk0Z1Nor9e0uisf98LrntPGMnk4frBO9+dkf69I= go.opentelemetry.io/otel v1.3.0 h1:APxLf0eiBwLl+SOXiJJCVYzA1OOJNyAoV8C5RNRyy7Y= go.opentelemetry.io/otel v1.3.0/go.mod h1:PWIKzi6JCp7sM0k9yZ43VX+T345uNbAkDKwHVjb2PTs= -go.opentelemetry.io/otel/exporters/jaeger v1.2.0 h1:C/5Egj3MJBXRJi22cSl07suqPqtZLnLFmH//OxETUEc= go.opentelemetry.io/otel/exporters/jaeger v1.2.0/go.mod h1:KJLFbEMKTNPIfOxcg/WikIozEoKcPgJRz3Ce1vLlM8E= -go.opentelemetry.io/otel/sdk v1.2.0 h1:wKN260u4DesJYhyjxDa7LRFkuhH7ncEVKU37LWcyNIo= +go.opentelemetry.io/otel/exporters/jaeger v1.3.0 h1:HfydzioALdtcB26H5WHc4K47iTETJCdloL7VN579/L0= +go.opentelemetry.io/otel/exporters/jaeger v1.3.0/go.mod h1:KoYHi1BtkUPncGSRtCe/eh1ijsnePhSkxwzz07vU0Fc= go.opentelemetry.io/otel/sdk v1.2.0/go.mod h1:jNN8QtpvbsKhgaC6V5lHiejMoKD+V8uadoSafgHPx1U= +go.opentelemetry.io/otel/sdk v1.3.0 h1:3278edCoH89MEJ0Ky8WQXVmDQv3FX4ZJ3Pp+9fJreAI= +go.opentelemetry.io/otel/sdk v1.3.0/go.mod h1:rIo4suHNhQwBIPg9axF8V9CA72Wz2mKF1teNrup8yzs= go.opentelemetry.io/otel/trace v1.2.0/go.mod h1:N5FLswTubnxKxOJHM7XZC074qpeEdLy3CgAVsdMucK0= go.opentelemetry.io/otel/trace v1.3.0 h1:doy8Hzb1RJ+I3yFhtDmwNc7tIyw1tNMOIsyPzp1NOGY= go.opentelemetry.io/otel/trace v1.3.0/go.mod h1:c/VDhno8888bvQYmbYLqe41/Ldmr/KKunbvWM4/fEjk= From f322dfcfcaf5bae7d1474820aa279da3c103d9dd Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 15 Dec 2021 06:33:02 +0000 Subject: [PATCH 48/56] Bump google.golang.org/grpc from 1.42.0 to 1.43.0 Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.42.0 to 1.43.0. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](https://github.com/grpc/grpc-go/compare/v1.42.0...v1.43.0) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- go.mod | 2 +- go.sum | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/go.mod b/go.mod index 0d8b12dd25..8ce810a449 100644 --- a/go.mod +++ b/go.mod @@ -67,7 +67,7 @@ require ( golang.org/x/image v0.0.0-20210628002857-a66eb6448b8d golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8 google.golang.org/genproto v0.0.0-20211129164237-f09f9a12af12 - google.golang.org/grpc v1.42.0 + google.golang.org/grpc v1.43.0 google.golang.org/protobuf v1.27.1 gopkg.in/yaml.v2 v2.4.0 gotest.tools/v3 v3.0.3 diff --git a/go.sum b/go.sum index 31a006b842..8ac3a0b1c9 100644 --- a/go.sum +++ b/go.sum @@ -1860,8 +1860,9 @@ google.golang.org/grpc v1.39.0/go.mod h1:PImNr+rS9TWYb2O4/emRugxiyHZ5JyHW5F+RPnD google.golang.org/grpc v1.39.1/go.mod h1:PImNr+rS9TWYb2O4/emRugxiyHZ5JyHW5F+RPnDzfrE= google.golang.org/grpc v1.40.0/go.mod h1:ogyxbiOoUXAkP+4+xa6PZSE9DZgIHtSpzjDTB9KAK34= google.golang.org/grpc v1.41.0/go.mod h1:U3l9uK9J0sini8mHphKoXyaqDA/8VyGnDee1zzIUK6k= -google.golang.org/grpc v1.42.0 h1:XT2/MFpuPFsEX2fWh3YQtHkZ+WYZFQRfaUgLZYj/p6A= google.golang.org/grpc v1.42.0/go.mod h1:k+4IHHFw41K8+bbowsex27ge2rCb65oeWqe4jJ590SU= +google.golang.org/grpc v1.43.0 h1:Eeu7bZtDZ2DpRCsLhUlcrLnvYaMK1Gz86a+hMVvELmM= +google.golang.org/grpc v1.43.0/go.mod h1:k+4IHHFw41K8+bbowsex27ge2rCb65oeWqe4jJ590SU= google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.1.0/go.mod h1:6Kw0yEErY5E/yWrBtf03jp27GLLJujG4z/JK95pnjjw= google.golang.org/grpc/examples v0.0.0-20210902184326-c93e472777b9/go.mod h1:gID3PKrg7pWKntu9Ss6zTLJ0ttC0X9IHgREOCZwbCVU= google.golang.org/grpc/examples v0.0.0-20211020220737-f00baa6c3c84 h1:vTEaoYojw/smuQT/Fva/AX+2Bnla97/oRbY75XFhg40= From 32f7af851d3caa9a175c63af11837b3a3b252474 Mon Sep 17 00:00:00 2001 From: Willy Kloucek Date: Wed, 15 Dec 2021 10:43:53 +0100 Subject: [PATCH 49/56] add new file url of the app provider to the ocs capabilities --- .../unreleased/enhancement-add-new-file-capability.md | 8 ++++++++ docs/extensions/storage/apps.md | 3 ++- storage/pkg/config/config.go | 6 ++++++ 3 files changed, 16 insertions(+), 1 deletion(-) create mode 100644 changelog/unreleased/enhancement-add-new-file-capability.md diff --git a/changelog/unreleased/enhancement-add-new-file-capability.md b/changelog/unreleased/enhancement-add-new-file-capability.md new file mode 100644 index 0000000000..57fe8c5b5e --- /dev/null +++ b/changelog/unreleased/enhancement-add-new-file-capability.md @@ -0,0 +1,8 @@ +Enhancement: add new file url of the app provider to the ocs capabilities + +We've added the new file capability of the app provider to the ocs capabilities, so that +clients can discover this url analogous to the app list and file open urls. + +https://github.com/owncloud/ocis/pull/2884 +https://github.com/cs3org/reva/pull/2379 +https://github.com/owncloud/web/pull/5890#issuecomment-993905242 diff --git a/docs/extensions/storage/apps.md b/docs/extensions/storage/apps.md index aa7247828c..1ca902ba69 100644 --- a/docs/extensions/storage/apps.md +++ b/docs/extensions/storage/apps.md @@ -24,7 +24,8 @@ The capabilities endpoint (eg. `https://localhost:9200/ocs/v1.php/cloud/capabili "enabled": true, "version": "1.0.0", "apps_url": "/app/list", - "open_url": "/app/open" + "open_url": "/app/open", + "new_url": "/app/new" } ] } diff --git a/storage/pkg/config/config.go b/storage/pkg/config/config.go index 0687716b31..6691edda40 100644 --- a/storage/pkg/config/config.go +++ b/storage/pkg/config/config.go @@ -61,6 +61,7 @@ type AppProvider struct { WopiDriver WopiDriver `ocisConfig:"wopi_driver"` AppsURL string `ocisConfig:"apps_url"` OpenURL string `ocisConfig:"open_url"` + NewURL string `ocisConfig:"new_url"` } type WopiDriver struct { @@ -938,6 +939,7 @@ func DefaultConfig() *Config { WopiDriver: WopiDriver{}, AppsURL: "/app/list", OpenURL: "/app/open", + NewURL: "/app/new", }, Configs: nil, UploadMaxChunkSize: 1e+8, @@ -1317,6 +1319,10 @@ func structMappings(cfg *Config) []shared.EnvBinding { EnvVars: []string{"STORAGE_FRONTEND_APP_PROVIDER_OPEN_URL"}, Destination: &cfg.Reva.AppProvider.OpenURL, }, + { + EnvVars: []string{"STORAGE_FRONTEND_APP_PROVIDER_NEW_URL"}, + Destination: &cfg.Reva.AppProvider.NewURL, + }, // gateway { From 2536df7199fb723c0acbab494b8f1fcf2575e183 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 15 Dec 2021 10:06:00 +0000 Subject: [PATCH 50/56] Bump github.com/spf13/cobra from 1.2.1 to 1.3.0 Bumps [github.com/spf13/cobra](https://github.com/spf13/cobra) from 1.2.1 to 1.3.0. - [Release notes](https://github.com/spf13/cobra/releases) - [Changelog](https://github.com/spf13/cobra/blob/master/CHANGELOG.md) - [Commits](https://github.com/spf13/cobra/compare/v1.2.1...v1.3.0) --- updated-dependencies: - dependency-name: github.com/spf13/cobra dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- go.mod | 12 +++---- go.sum | 98 +++++++++++++++++++++++++++++++++++++++++++++++++++++----- 2 files changed, 95 insertions(+), 15 deletions(-) diff --git a/go.mod b/go.mod index 8ce810a449..2c969652e0 100644 --- a/go.mod +++ b/go.mod @@ -51,7 +51,7 @@ require ( github.com/prometheus/client_golang v1.11.0 github.com/rs/zerolog v1.26.0 github.com/sirupsen/logrus v1.8.1 - github.com/spf13/cobra v1.2.1 + github.com/spf13/cobra v1.3.0 github.com/stretchr/testify v1.7.0 github.com/thejerf/suture/v4 v4.0.1 github.com/urfave/cli/v2 v2.3.0 @@ -66,7 +66,7 @@ require ( golang.org/x/crypto v0.0.0-20211108221036-ceb1ce70b4fa golang.org/x/image v0.0.0-20210628002857-a66eb6448b8d golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8 - google.golang.org/genproto v0.0.0-20211129164237-f09f9a12af12 + google.golang.org/genproto v0.0.0-20211208223120-3a66f561d7aa google.golang.org/grpc v1.43.0 google.golang.org/protobuf v1.27.1 gopkg.in/yaml.v2 v2.4.0 @@ -75,7 +75,6 @@ require ( ) require ( - cloud.google.com/go v0.93.3 // indirect contrib.go.opencensus.io/exporter/prometheus v0.4.0 // indirect github.com/Azure/go-ntlmssp v0.0.0-20200615164410-66371956d46c // indirect github.com/Masterminds/goutils v1.1.1 // indirect @@ -139,7 +138,6 @@ require ( github.com/gogo/protobuf v1.3.2 // indirect github.com/golang-jwt/jwt v3.2.2+incompatible // indirect github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect - github.com/golang/mock v1.6.0 // indirect github.com/golang/snappy v0.0.4 // indirect github.com/gomodule/redigo v1.8.6 // indirect github.com/google/go-cmp v0.5.6 // indirect @@ -164,7 +162,7 @@ require ( github.com/klauspost/cpuid/v2 v2.0.9 // indirect github.com/longsleep/go-metrics v1.0.0 // indirect github.com/mattermost/xml-roundtrip-validator v0.1.0 // indirect - github.com/mattn/go-colorable v0.1.11 // indirect + github.com/mattn/go-colorable v0.1.12 // indirect github.com/mattn/go-isatty v0.0.14 // indirect github.com/mattn/go-runewidth v0.0.13 // indirect github.com/mattn/go-sqlite3 v1.14.9 // indirect @@ -229,12 +227,12 @@ require ( go.uber.org/zap v1.19.1 // indirect golang.org/x/net v0.0.0-20211111083644-e5c967477495 // indirect golang.org/x/sync v0.0.0-20210220032951-036812b2e83c // indirect - golang.org/x/sys v0.0.0-20211110154304-99a53858aa08 // indirect + golang.org/x/sys v0.0.0-20211205182925-97ca703d548d // indirect golang.org/x/text v0.3.7 // indirect golang.org/x/time v0.0.0-20210723032227-1f47c861a9ac // indirect golang.org/x/tools v0.1.7 // indirect google.golang.org/appengine v1.6.7 // indirect - gopkg.in/ini.v1 v1.63.2 // indirect + gopkg.in/ini.v1 v1.66.2 // indirect gopkg.in/square/go-jose.v2 v2.6.0 // indirect gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 // indirect gopkg.in/warnings.v0 v0.1.2 // indirect diff --git a/go.sum b/go.sum index 8ac3a0b1c9..83b3bd6054 100644 --- a/go.sum +++ b/go.sum @@ -26,8 +26,12 @@ cloud.google.com/go v0.83.0/go.mod h1:Z7MJUsANfY0pYPdw0lbnivPx4/vhy/e2FEkSkF7vAV cloud.google.com/go v0.84.0/go.mod h1:RazrYuxIK6Kb7YrzzhPoLmCVzl7Sup4NrbKPg8KHSUM= cloud.google.com/go v0.87.0/go.mod h1:TpDYlFy7vuLzZMMZ+B6iRiELaY7z/gJPaqbMx6mlWcY= cloud.google.com/go v0.90.0/go.mod h1:kRX0mNRHe0e2rC6oNakvwQqzyDmg57xJ+SZU1eT2aDQ= -cloud.google.com/go v0.93.3 h1:wPBktZFzYBcCZVARvwVKqH1uEj+aLXofJEtrb4oOsio= cloud.google.com/go v0.93.3/go.mod h1:8utlLll2EF5XMAV15woO4lSbWQlk8rer9aLOfLh7+YI= +cloud.google.com/go v0.94.1/go.mod h1:qAlAugsXlC+JWO+Bke5vCtc9ONxjQT3drlTTnAplMW4= +cloud.google.com/go v0.97.0/go.mod h1:GF7l59pYBVlXQIBLx3a761cZ41F9bBH3JUlihCt2Udc= +cloud.google.com/go v0.98.0/go.mod h1:ua6Ush4NALrHk5QXDWnjvZHN93OuF0HfuEPq9I1X0cM= +cloud.google.com/go v0.99.0 h1:y/cM2iqGgGi5D5DQZl6D9STN/3dR/Vx5Mp8s752oJTY= +cloud.google.com/go v0.99.0/go.mod h1:w0Xx2nLzqWJPuozYQX+hFfCSI8WioryfRDzkoI/Y2ZA= cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o= cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE= cloud.google.com/go/bigquery v1.4.0/go.mod h1:S8dzgnTigyfTmLBfrtrhyYhwRxG72rYxvftPBK2Dvzc= @@ -37,6 +41,7 @@ cloud.google.com/go/bigquery v1.8.0/go.mod h1:J5hqkt3O0uAFnINi6JXValWIb1v0goeZM7 cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE= cloud.google.com/go/datastore v1.1.0/go.mod h1:umbIZjpQpHh4hmRpGhH4tLFup+FVzqBi1b3c64qFpCk= cloud.google.com/go/firestore v1.1.0/go.mod h1:ulACoGHTpvq5r8rxGJ4ddJZBZqakUQqClKRT5SZwBmk= +cloud.google.com/go/firestore v1.6.1/go.mod h1:asNXNOzBdyVQmEU+ggO8UPodTkEVFW5Qx+rwHnAz+EY= cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I= cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw= cloud.google.com/go/pubsub v1.2.0/go.mod h1:jhfEVHT8odbXTkndysNHCcx0awwzvfOlguIAii9o8iA= @@ -73,6 +78,7 @@ github.com/BurntSushi/toml v0.4.1/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbi github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo= github.com/CiscoM31/godata v1.0.5 h1:AITXpa/5ybXEq59A0nqUGiS7ZXVJnQtFw5o09tyN/UA= github.com/CiscoM31/godata v1.0.5/go.mod h1:wcmFm66GMdOE316TgwFO1wo5ainCvTK26omd93oZf2M= +github.com/DataDog/datadog-go v3.2.0+incompatible/go.mod h1:LButxg5PwREeZtORoXG3tL4fMGNddJ+vMq1mwgfaqoQ= github.com/GeertJohan/yubigo v0.0.0-20190917122436-175bc097e60e h1:Bqtt5C+uVk+vH/t5dmB47uDCTwxw16EYHqvJnmY2aQc= github.com/GeertJohan/yubigo v0.0.0-20190917122436-175bc097e60e/go.mod h1:njRCDrl+1RQ/A/+KVU8Ho2EWAxUSkohOWczdW3dzDG0= github.com/HdrHistogram/hdrhistogram-go v1.1.2/go.mod h1:yDgFjdqOqDEKOvasDdhWNXYg9BVp4O+o5f6V/ehm6Oo= @@ -128,7 +134,9 @@ github.com/arl/statsviz v0.4.0/go.mod h1:+5inUy/dxy11x/KSmicG3ZrEEy0Yr81AFm3dn4Q github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hCbHZ8TKRvWD2dDTCfh9M9ya+I9JpbB7O8o= github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8= github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da/go.mod h1:Q73ZrmVTwzkszR9V5SSuryQ31EELlFMUz1kKyl939pY= +github.com/armon/go-metrics v0.3.10/go.mod h1:4O98XIr/9W0sxpJ8UaYkvjk10Iff7SnFrb4QAOwNTFc= github.com/armon/go-radix v0.0.0-20180808171621-7fddfc383310/go.mod h1:ufUuZ+zHj4x4TnLV4JWEpy2hxWSpsRywHrMgIH9cCH8= +github.com/armon/go-radix v1.0.0/go.mod h1:ufUuZ+zHj4x4TnLV4JWEpy2hxWSpsRywHrMgIH9cCH8= github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPdPJAN/hZIm0C4OItdklCFmMRWYpio= github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs= github.com/asaskevich/govalidator v0.0.0-20180720115003-f9ffefc3facf/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY= @@ -232,6 +240,7 @@ github.com/cenkalti/backoff/v4 v4.1.0/go.mod h1:scbssz8iZGpm3xbr14ovlUdkxfGXNInq github.com/cenkalti/backoff/v4 v4.1.1/go.mod h1:scbssz8iZGpm3xbr14ovlUdkxfGXNInqkPWOWmG2CLw= github.com/census-instrumentation/opencensus-proto v0.2.0/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= +github.com/census-instrumentation/opencensus-proto v0.3.0/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= github.com/cespare/xxhash v1.1.0 h1:a6HrQnmkObjyL+Gs60czilIUGqrzKutQD6XZog3p+ko= github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc= github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= @@ -243,6 +252,8 @@ github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWR github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI= github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU= github.com/cilium/ebpf v0.0.0-20200110133405-4032b1d8aae3/go.mod h1:MA5e5Lr8slmEg9bt0VpxxWqJlO4iwu3FBdHUzV7wQVg= +github.com/circonus-labs/circonus-gometrics v2.3.1+incompatible/go.mod h1:nmEj6Dob7S7YxXgwXpfOuvO54S+tGdZdw9fuRZt25Ag= +github.com/circonus-labs/circonusllhist v0.1.3/go.mod h1:kMXHVDlOchFAehlya5ePtbp5jckzBHf4XRpQvBOLI+I= github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= github.com/cloudflare/cloudflare-go v0.14.0/go.mod h1:EnwdgGMaFOruiPZRFSgn+TsQ3hQ7C/YWzIGLeu5c304= github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc= @@ -252,7 +263,9 @@ github.com/cncf/udpa/go v0.0.0-20210930031921-04548b0d99d4/go.mod h1:6pvJx4me5XP github.com/cncf/xds/go v0.0.0-20210312221358-fbca930ec8ed/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= github.com/cncf/xds/go v0.0.0-20210805033703-aa0b78936158/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= github.com/cncf/xds/go v0.0.0-20210922020428-25de7278fc84/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= +github.com/cncf/xds/go v0.0.0-20211001041855-01bcc9b48dfe/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= github.com/cncf/xds/go v0.0.0-20211011173535-cb28da3451f1/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= +github.com/cncf/xds/go v0.0.0-20211130200136-a8f946100490/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs= github.com/containerd/cgroups v0.0.0-20200531161412-0dbf7f05ba59/go.mod h1:pA0z1pT8KYB3TCXK/ocprsh7MAkoW8bZVzPdih9snmM= github.com/containerd/console v0.0.0-20180822173158-c12b1e7919c1/go.mod h1:Tj/on1eG8kiEhd0+fhSDzsPAFESxzBBvdyEgyryXffw= github.com/containerd/containerd v1.3.2/go.mod h1:bC6axHOhabU15QhwfG7w5PipXdVtMXFTttgp+kVtyUA= @@ -342,7 +355,9 @@ github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.m github.com/envoyproxy/go-control-plane v0.9.9-0.20210217033140-668b12f5399d/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk= github.com/envoyproxy/go-control-plane v0.9.9-0.20210512163311-63b5d3c536b0/go.mod h1:hliV/p42l8fGbc6Y9bQ70uLwIvmJyVE5k4iMKlh8wCQ= github.com/envoyproxy/go-control-plane v0.9.10-0.20210907150352-cf90f659a021/go.mod h1:AFq3mo9L8Lqqiid3OhADV3RfLJnjiw63cSpi+fDTRC0= +github.com/envoyproxy/go-control-plane v0.10.1/go.mod h1:AY7fTTXNdv/aJ2O5jwpxAPOWUZ7hQAEvzN5Pf27BkQQ= github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= +github.com/envoyproxy/protoc-gen-validate v0.6.2/go.mod h1:2t7qjJNvHPx8IjnBOzl9E9/baC+qXE/TeeyBRzgJDws= github.com/eternnoir/gncp v0.0.0-20170707042257-c70df2d0cd68 h1:DHBMBKJK69xBWnD/jNkTN0sOT7nT7I5If9VMsk9Jj5Y= github.com/eternnoir/gncp v0.0.0-20170707042257-c70df2d0cd68/go.mod h1:8FuQ7lU9ZvIJGvc04F/qblkjqIfBahAoEFV+XPxByGw= github.com/evanphx/json-patch/v5 v5.5.0/go.mod h1:G79N1coSVB93tBe7j6PhzjmR3/2VvlbKOFpnXhI9Bw4= @@ -658,6 +673,8 @@ github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I= github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg= github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk= +github.com/googleapis/gax-go/v2 v2.1.0/go.mod h1:Q3nei7sK6ybPYH7twZdmQpAd1MKb7pfu6SK+H1/DsU0= +github.com/googleapis/gax-go/v2 v2.1.1/go.mod h1:hddJymUZASv3XPyGkUpKj8pPO47Rmb0eJc8R6ouapiM= github.com/gookit/color v1.4.2 h1:tXy44JFSFkKnELV6WaMo/lLfu/meqITX3iAV52do7lk= github.com/gookit/color v1.4.2/go.mod h1:fqRyamkC1W8uxl+lxCQxOT09l/vYfZ+QeiX3rKQHCoQ= github.com/gookit/config/v2 v2.0.27 h1:jCQSAe3qN8gFABz6h4qA9MNhTk0D4ipn6+1jeAD7g9E= @@ -694,20 +711,29 @@ github.com/grpc-ecosystem/grpc-gateway/v2 v2.7.1 h1:p5m7GOEGXyoq6QWl4/RRMsQ6tWbT github.com/grpc-ecosystem/grpc-gateway/v2 v2.7.1/go.mod h1:8ZeZajTed/blCOHBbj8Fss8bPHiFKcmJJzuIbUtFCAo= github.com/h2non/parth v0.0.0-20190131123155-b4df798d6542/go.mod h1:Ow0tF8D4Kplbc8s8sSb3V2oUCygFHVp8gC3Dn6U4MNI= github.com/hashicorp/consul/api v1.1.0/go.mod h1:VmuI/Lkw1nC05EYQWNKwWGbkg+FbDBtguAZLlVdkD9Q= +github.com/hashicorp/consul/api v1.11.0/go.mod h1:XjsvQN+RJGWI2TWy1/kqaE16HrR2J/FWgkYjdZQsX9M= github.com/hashicorp/consul/sdk v0.1.1/go.mod h1:VKf9jXwCTEY1QZP2MOLRhb5i/I/ssyNV1vwHyQBF0x8= +github.com/hashicorp/consul/sdk v0.8.0/go.mod h1:GBvyrGALthsZObzUGsfgHZQDXjg4lOjagTIwIR1vPms= github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= +github.com/hashicorp/go-cleanhttp v0.5.0/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80= github.com/hashicorp/go-cleanhttp v0.5.1/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80= +github.com/hashicorp/go-cleanhttp v0.5.2/go.mod h1:kO/YDlP8L1346E6Sodw+PrpBSV4/SoxCXGY6BqNFT48= github.com/hashicorp/go-hclog v0.9.2/go.mod h1:5CU+agLiy3J7N7QjHK5d05KxGsuXiQLrjA0H7acj2lQ= +github.com/hashicorp/go-hclog v0.12.0/go.mod h1:whpDNt7SSdeAju8AWKIWsul05p54N/39EeqMAyrmvFQ= github.com/hashicorp/go-hclog v0.14.1/go.mod h1:whpDNt7SSdeAju8AWKIWsul05p54N/39EeqMAyrmvFQ= github.com/hashicorp/go-hclog v1.0.0 h1:bkKf0BeBXcSYa7f5Fyi9gMuQ8gNsxeiNpZjR6VxNZeo= github.com/hashicorp/go-hclog v1.0.0/go.mod h1:whpDNt7SSdeAju8AWKIWsul05p54N/39EeqMAyrmvFQ= github.com/hashicorp/go-immutable-radix v1.0.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60= +github.com/hashicorp/go-immutable-radix v1.3.1/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60= github.com/hashicorp/go-msgpack v0.5.3/go.mod h1:ahLV/dePpqEmjfWmKiqvPkv/twdG7iPBM1vqhUKIvfM= github.com/hashicorp/go-multierror v1.0.0/go.mod h1:dHtQlpGsu+cZNNAkkCN/P3hoUDHhCYQXV3UM06sGGrk= +github.com/hashicorp/go-multierror v1.1.0/go.mod h1:spPvp8C1qA32ftKqdAHm4hHTbPw+vmowP0z+KUhOZdA= github.com/hashicorp/go-plugin v1.4.3 h1:DXmvivbWD5qdiBts9TpBC7BYL1Aia5sxbRgQB+v6UZM= github.com/hashicorp/go-plugin v1.4.3/go.mod h1:5fGEH17QVwTTcR0zV7yhDPLLmFX9YSZ38b18Udy6vYQ= +github.com/hashicorp/go-retryablehttp v0.5.3/go.mod h1:9B5zBasrRhHXnJnui7y6sL7es7NDiJgTc6Er0maI1Xs= github.com/hashicorp/go-retryablehttp v0.6.6/go.mod h1:vAew36LZh98gCBJNLH42IQ1ER/9wtLZZ8meHqQvEYWY= github.com/hashicorp/go-rootcerts v1.0.0/go.mod h1:K6zTfqpRlCUIjkwsN4Z+hiSfzSTQa6eBIzfwKfwNnHU= +github.com/hashicorp/go-rootcerts v1.0.2/go.mod h1:pqUvnprVnM5bf7AOirdbb01K4ccR319Vf4pU3K5EGc8= github.com/hashicorp/go-sockaddr v1.0.0/go.mod h1:7Xibr9yA9JjQq1JpNB2Vw7kxv8xerXegt+ozgdvDeDU= github.com/hashicorp/go-sockaddr v1.0.2/go.mod h1:rB4wwRAUzs07qva3c5SdrY/NEtAUjGlgmH/UkBUC97A= github.com/hashicorp/go-syslog v1.0.0/go.mod h1:qPfqrKkXGihmCqbJM2mZgkZGvKG1dFdvsLplgctolz4= @@ -717,13 +743,20 @@ github.com/hashicorp/go-uuid v1.0.2/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/b github.com/hashicorp/go.net v0.0.1/go.mod h1:hjKkEWcCURg++eb33jQU7oqQcI9XDCnUzHA0oac0k90= github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= +github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4= github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ= github.com/hashicorp/hcl/v2 v2.10.1/go.mod h1:FwWsfWEjyV/CMj8s/gqAuiviY72rJ1/oayI9WftqcKg= github.com/hashicorp/logutils v1.0.0/go.mod h1:QIAnNjmIWmVIIkWDTG1z5v++HQmx9WQRO+LraFDTW64= github.com/hashicorp/mdns v1.0.0/go.mod h1:tL+uN++7HEJ6SQLQ2/p+z2pH24WQKWjBPkE0mNTz8vQ= +github.com/hashicorp/mdns v1.0.1/go.mod h1:4gW7WsVCke5TE7EPeYliwHlRUyBtfCwuFwuMg2DmyNY= +github.com/hashicorp/mdns v1.0.4/go.mod h1:mtBihi+LeNXGtG8L9dX59gAEa12BDtBQSp4v/YAJqrc= github.com/hashicorp/memberlist v0.1.3/go.mod h1:ajVTdAv/9Im8oMAAj5G31PhhMCZJV2pPBoIllUwCN7I= +github.com/hashicorp/memberlist v0.2.2/go.mod h1:MS2lj3INKhZjWNqd3N0m3J+Jxf3DAOnAH9VT3Sh9MUE= github.com/hashicorp/memberlist v0.2.4/go.mod h1:MS2lj3INKhZjWNqd3N0m3J+Jxf3DAOnAH9VT3Sh9MUE= +github.com/hashicorp/memberlist v0.3.0/go.mod h1:MS2lj3INKhZjWNqd3N0m3J+Jxf3DAOnAH9VT3Sh9MUE= github.com/hashicorp/serf v0.8.2/go.mod h1:6hOLApaqBFA1NXqRQAsxw9QxuDEvNxSQRwA/JwenrHc= +github.com/hashicorp/serf v0.9.5/go.mod h1:UWDWwZeL5cuWDJdl0C6wrvrUwEqtQ4ZKBKKENpqIUyk= +github.com/hashicorp/serf v0.9.6/go.mod h1:TXZNMjZQijwlDvp+r0b63xZ45H7JmCmgg4gpTwn9UV4= github.com/hashicorp/yamux v0.0.0-20180604194846-3520598351bb/go.mod h1:+NfK9FKeTrX5uv1uIXGdwYDTeHna2qgaIlx54MXqjAM= github.com/hashicorp/yamux v0.0.0-20210826001029-26ff87cf9493 h1:brI5vBRUlAlM34VFmnLPwjnCL/FxAJp9XvOdX6Zt+XE= github.com/hashicorp/yamux v0.0.0-20210826001029-26ff87cf9493/go.mod h1:CtWFDAQgb7dxtzFs4tWbplKIe2jSi3+5vKbgIO0SLnQ= @@ -766,6 +799,7 @@ github.com/jpillora/backoff v1.0.0/go.mod h1:J/6gKK9jxlEcS3zixgDgUAsiuZ7yrSoa/FX github.com/json-iterator/go v1.1.5/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU= github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU= github.com/json-iterator/go v1.1.7/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= +github.com/json-iterator/go v1.1.9/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= github.com/json-iterator/go v1.1.11/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= @@ -836,6 +870,7 @@ github.com/liquidweb/liquidweb-go v1.6.3/go.mod h1:SuXXp+thr28LnjEw18AYtWwIbWMHS github.com/longsleep/go-metrics v0.0.0-20170706183227-c1943bcf9047/go.mod h1:Eq9KjddJTZCHG0ja+SEJNp739Um4URrcBuccq3Ih/NI= github.com/longsleep/go-metrics v1.0.0 h1:o2A6Dbu4MhLpZuL444WFoZzM7X7igewrj2MouwTlmVM= github.com/longsleep/go-metrics v1.0.0/go.mod h1:w6QO1LBkVla70FZrrF6XcB0YN+jTEYugjkn3+6RYTSM= +github.com/lyft/protoc-gen-star v0.5.3/go.mod h1:V0xaHgaf5oCCqmcxYcWiDfTiKsZsRc87/1qhoTACD8w= github.com/m3o/m3o-go/client v0.0.0-20210421144725-8bfd7992ada3/go.mod h1:vmeaYrKYpgVNhny/l7iH8mXS88S7ijUiYni3gZUrCq0= github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= github.com/magiconair/properties v1.8.1/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= @@ -858,11 +893,12 @@ github.com/mattermost/xml-roundtrip-validator v0.1.0/go.mod h1:qccnGMcpgwcNaBnxq github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU= github.com/mattn/go-colorable v0.1.2/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE= github.com/mattn/go-colorable v0.1.4/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE= +github.com/mattn/go-colorable v0.1.6/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc= github.com/mattn/go-colorable v0.1.7/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc= github.com/mattn/go-colorable v0.1.8/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc= github.com/mattn/go-colorable v0.1.9/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc= -github.com/mattn/go-colorable v0.1.11 h1:nQ+aFkoE2TMGc0b68U2OKSexC+eq46+XwZzWXHRmPYs= -github.com/mattn/go-colorable v0.1.11/go.mod h1:u5H1YNBxpqRaxsYJYSkiCWKzEfiAb1Gb520KVy5xxl4= +github.com/mattn/go-colorable v0.1.12 h1:jF+Du6AlPIjs2BiUiQlKOX0rt3SujHxPnksPKZbaA40= +github.com/mattn/go-colorable v0.1.12/go.mod h1:u5H1YNBxpqRaxsYJYSkiCWKzEfiAb1Gb520KVy5xxl4= github.com/mattn/go-isatty v0.0.3/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4= github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= github.com/mattn/go-isatty v0.0.9/go.mod h1:YNRxwqDuOph6SZLI9vUUz6OYw3QyUt7WiY2yME+cCiQ= @@ -892,6 +928,7 @@ github.com/mennanov/fieldmask-utils v0.5.0/go.mod h1:lah2lHczE2ff+7SqnNKpB+YzaO7 github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg= github.com/miekg/dns v1.1.26/go.mod h1:bPDLeHnStXmXAq1m/Ch/hvfNHr14JKNPMBo3VZKjuso= github.com/miekg/dns v1.1.40/go.mod h1:KNUDUusw/aVsxyTYZM1oqvCicbwhgbNgztCETuNZ7xM= +github.com/miekg/dns v1.1.41/go.mod h1:p6aan82bvRIyn+zDIv9xYNUpwa73JcSh9BKwknJysuI= github.com/miekg/dns v1.1.43 h1:JKfpVSCB84vrAmHzyrsxB5NAr5kLoMXZArPSw7Qlgyg= github.com/miekg/dns v1.1.43/go.mod h1:+evo5L0630/F6ca/Z9+GAqzhjGyn8/c+TBaOyfEl0V4= github.com/mileusna/useragent v1.0.2 h1:DgVKtiPnjxlb73z9bCwgdUvU2nQNQ97uhgfO8l9uz/w= @@ -905,6 +942,7 @@ github.com/minio/sha256-simd v0.1.1/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g= github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM= github.com/mitchellh/cli v1.0.0/go.mod h1:hNIlj7HEI86fIcpObd7a0FcrxTWetlwJDGcceTlRvqc= +github.com/mitchellh/cli v1.1.0/go.mod h1:xcISNoH86gajksDmfB23e/pu+B+GeFRMYmoHXxx3xhI= github.com/mitchellh/copystructure v1.2.0 h1:vpKXTN4ewci03Vljg/q9QvCGUDttBOGBIa15WveJJGw= github.com/mitchellh/copystructure v1.2.0/go.mod h1:qLl+cE2AmVv+CoeAwDPye/v+N2HKCj9FbZEVFJRxO9s= github.com/mitchellh/go-homedir v1.0.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0= @@ -1019,6 +1057,7 @@ github.com/owncloud/libre-graph-api-go v0.3.0/go.mod h1:dOt7+kXldpyUGBFdfFBs6PbW github.com/oxtoacart/bpool v0.0.0-20190530202638-03653db5a59c h1:rp5dCmg/yLR3mgFuSOe4oEnDDmGLROTvMragMUXpTQw= github.com/oxtoacart/bpool v0.0.0-20190530202638-03653db5a59c/go.mod h1:X07ZCGwUbLaax7L0S3Tw4hpejzu63ZrrQiUe6W0hcy0= github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc= +github.com/pascaldekloe/goe v0.1.0/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc= github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaRUnok+kx1WdO15EQc= github.com/patrickmn/go-cache v2.1.0+incompatible/go.mod h1:3Qf8kWWT7OJRJbdiICTKqZju1ZixQ/KpMGzzAfe6+WQ= github.com/pborman/uuid v1.2.0/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtPdI/k= @@ -1027,6 +1066,7 @@ github.com/pelletier/go-toml v1.4.0/go.mod h1:PN7xzY2wHTK0K9p34ErDQMlFxa51Fk0OUr github.com/pelletier/go-toml v1.7.0/go.mod h1:vwGMzjaWMwyfHwgIBhI2YUM4fB6nL6lVAvS1LBMMhTE= github.com/pelletier/go-toml v1.8.1/go.mod h1:T2/BmBdy8dvIRq1a/8aqjN41wvWlN4lrapLU/GW4pbc= github.com/pelletier/go-toml v1.9.3/go.mod h1:u1nR/EPcESfeI/szUZKdtJ0xRNbUoANCkoOuaOx1Y+c= +github.com/pelletier/go-toml v1.9.4/go.mod h1:u1nR/EPcESfeI/szUZKdtJ0xRNbUoANCkoOuaOx1Y+c= github.com/pierrec/lz4 v2.0.5+incompatible/go.mod h1:pdkljMzZIN41W+lC3N2tnIh5sFi+IEE17M5jbnwPHcY= github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA= github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= @@ -1040,6 +1080,7 @@ github.com/pkg/xattr v0.4.4/go.mod h1:sBD3RAqlr8Q+RC3FutZcikpT8nyDrIEEBw2J744gVW github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/posener/complete v1.1.1/go.mod h1:em0nMJCgc9GFtwrmVmEMR/ZL6WyhyjMBndrE9hABlRI= +github.com/posener/complete v1.2.3/go.mod h1:WZIdtGGp+qx0sLrYKtIRAruyNpv6hFCicSgv7Sy7s/s= github.com/pquerna/cachecontrol v0.0.0-20180517163645-1555304b9b35/go.mod h1:prYjPmNq4d1NPVmpShWobRqXY3q7Vp+80DqgxxUrUIA= github.com/pquerna/cachecontrol v0.1.0 h1:yJMy84ti9h/+OEWa752kBTKv4XC30OtVVHYv/8cTqKc= github.com/pquerna/cachecontrol v0.1.0/go.mod h1:NrUG3Z7Rdu85UNR3vm7SOsl1nFIeSiQnrHV5K9mBcUI= @@ -1053,6 +1094,7 @@ github.com/prometheus/client_golang v0.9.3-0.20190127221311-3c4408c8b829/go.mod github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDft0ttaMvbicHlPoso= github.com/prometheus/client_golang v1.0.0/go.mod h1:db9x61etRT2tGnBNRi70OPL5FsnadC4Ky3P0J6CfImo= github.com/prometheus/client_golang v1.1.0/go.mod h1:I1FGZT9+L76gKKOs5djB6ezCbFQP1xR9D75/vuwEF3g= +github.com/prometheus/client_golang v1.4.0/go.mod h1:e9GMxYsXl05ICDXkRhurwBS4Q3OK1iX/F2sw+iXX5zU= github.com/prometheus/client_golang v1.7.1/go.mod h1:PY5Wy2awLA44sXw4AOSfFBetzPP4j5+D6mVACh+pe2M= github.com/prometheus/client_golang v1.11.0 h1:HNkLOAEQMIDv/K+04rukrLx6ch7msSRwf3/SASFAGtQ= github.com/prometheus/client_golang v1.11.0/go.mod h1:Z6t4BnS23TR94PD6BsDNk8yVqroYurpAkEiz0P2BEV0= @@ -1069,6 +1111,7 @@ github.com/prometheus/common v0.2.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y8 github.com/prometheus/common v0.4.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4= github.com/prometheus/common v0.4.1/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4= github.com/prometheus/common v0.6.0/go.mod h1:eBmuwkDJBwy6iBfxCBob6t6dR6ENT/y+J+Zk0j9GMYc= +github.com/prometheus/common v0.9.1/go.mod h1:yhUN8i9wzaXS3w1O07YhxHEBxD+W35wd8bs7vj7HSQ4= github.com/prometheus/common v0.10.0/go.mod h1:Tlit/dnDKsSWFlCLTWaA1cyBgKHSMdTB80sz/V91rCo= github.com/prometheus/common v0.26.0/go.mod h1:M7rCNAaPfAosfx8veZJCuw84e35h3Cfd9VFqTh1DIvc= github.com/prometheus/common v0.28.0/go.mod h1:vu+V0TpY+O6vW9J44gczi3Ap/oXXR10b+M/gUGO4Hls= @@ -1085,6 +1128,7 @@ github.com/prometheus/procfs v0.0.0-20190117184657-bf6a532e95b1/go.mod h1:c3At6R github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA= github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA= github.com/prometheus/procfs v0.0.3/go.mod h1:4A/X28fw3Fc593LaREMrKMqOKvUAntwMDaekg4FpcdQ= +github.com/prometheus/procfs v0.0.8/go.mod h1:7Qr8sr6344vo1JqZ6HhLceV9o3AJ1Ff+GxbHq6oeK9A= github.com/prometheus/procfs v0.1.3/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU= github.com/prometheus/procfs v0.6.0/go.mod h1:cz+aTbrPOrUb4q7XlbU9ygM+/jj0fzG6c1xBZuNvfVA= github.com/prometheus/procfs v0.7.3 h1:4jVXhlkAyzOScmCkXBTOLRLTz8EeU+eyjrwB/EPq0VU= @@ -1129,6 +1173,7 @@ github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQD github.com/ryanuber/columnize v0.0.0-20160712163229-9b3edd62028f/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts= github.com/ryanuber/columnize v2.1.0+incompatible/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts= github.com/sacloud/libsacloud v1.36.2/go.mod h1:P7YAOVmnIn3DKHqCZcUKYUXmSwGBm3yS7IBEjKVSrjg= +github.com/sagikazarmark/crypt v0.3.0/go.mod h1:uD/D+6UF4SrIR1uGEv7bBNkNqLGqUr43MRiaGWX1Nig= github.com/satori/go.uuid v1.2.0 h1:0uYX9dsZ2yD7q2RtLRtPSdGDWzjeM3TbMJP9utgA0ww= github.com/satori/go.uuid v1.2.0/go.mod h1:dA0hQrYB0VpLJoorglMZABFdXlWrHn1NEOzdhQKdks0= github.com/scaleway/scaleway-sdk-go v1.0.0-beta.7.0.20210127161313-bd30bebeac4f/go.mod h1:CJJ5VAbozOl0yEw7nHB9+7BXTJbIn6h7W+f6Gau5IP8= @@ -1168,17 +1213,20 @@ github.com/sony/gobreaker v0.4.1 h1:oMnRNZXX5j85zso6xCPRNPtmAycat+WcoKbklScLDgQ= github.com/sony/gobreaker v0.4.1/go.mod h1:ZKptC7FHNvhBz7dN2LGjPVBz2sZJmc0/PkyDJOjmxWY= github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ= +github.com/spf13/afero v1.3.3/go.mod h1:5KUK8ByomD5Ti5Artl0RtHeI5pTF7MIDuXL3yY520V4= github.com/spf13/afero v1.4.1/go.mod h1:Ai8FlHk4v/PARR026UzYexafAt9roJ7LcLMAmO6Z93I= github.com/spf13/afero v1.6.0/go.mod h1:Ai8FlHk4v/PARR026UzYexafAt9roJ7LcLMAmO6Z93I= github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE= github.com/spf13/cast v1.3.1/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE= +github.com/spf13/cast v1.4.1/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE= github.com/spf13/cobra v0.0.1/go.mod h1:1l0Ry5zgKvJasoi3XT1TypsSe7PqH0Sj9dhYf7v3XqQ= github.com/spf13/cobra v0.0.2-0.20171109065643-2da4a54c5cee/go.mod h1:1l0Ry5zgKvJasoi3XT1TypsSe7PqH0Sj9dhYf7v3XqQ= github.com/spf13/cobra v0.0.3/go.mod h1:1l0Ry5zgKvJasoi3XT1TypsSe7PqH0Sj9dhYf7v3XqQ= github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU= github.com/spf13/cobra v1.1.1/go.mod h1:WnodtKOvamDL/PwE2M4iKs8aMDBZ5Q5klgD3qfVJQMI= -github.com/spf13/cobra v1.2.1 h1:+KmjbUw1hriSNMF55oPrkZcb27aECyrj8V2ytv7kWDw= github.com/spf13/cobra v1.2.1/go.mod h1:ExllRjgxM/piMAM+3tAZvg8fsklGAf3tPfi+i8t68Nk= +github.com/spf13/cobra v1.3.0 h1:R7cSvGu+Vv+qX0gW5R/85dx2kmmJT5z5NM8ifdYjdn0= +github.com/spf13/cobra v1.3.0/go.mod h1:BrRVncBjOJa/eUcVVm9CE+oC6as8k+VYr4NY7WCi9V4= github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo= github.com/spf13/jwalterweatherman v1.1.0/go.mod h1:aNWZUN0dPAAO/Ljvb5BEdw96iTZ0EXowPYD95IqWIGo= github.com/spf13/pflag v1.0.0/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= @@ -1191,6 +1239,7 @@ github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DM github.com/spf13/viper v1.7.0/go.mod h1:8WkrPz2fc9jxqZNCJI/76HCieCp4Q8HaLFoCha5qpdg= github.com/spf13/viper v1.7.1/go.mod h1:8WkrPz2fc9jxqZNCJI/76HCieCp4Q8HaLFoCha5qpdg= github.com/spf13/viper v1.8.1/go.mod h1:o0Pch8wJ9BVSWGQMbra6iw0oQ5oktSIBaujf1rJH9Ns= +github.com/spf13/viper v1.10.0/go.mod h1:SoyBPwAtKDzypXNDFKN5kzH7ppppbGZtls1UpIy5AsM= github.com/steveyen/gtreap v0.1.0 h1:CjhzTa274PyJLJuMZwIzCO1PfC00oRa8d1Kc78bFXJM= github.com/steveyen/gtreap v0.1.0/go.mod h1:kl/5J7XbrOmlIbYIXdRHDDE5QxHqpk0cmkT7Z4dM9/Y= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= @@ -1220,6 +1269,7 @@ github.com/transip/gotransip/v6 v6.2.0/go.mod h1:pQZ36hWWRahCUXkFWlx9Hs711gLd8J4 github.com/tus/tusd v1.1.0/go.mod h1:3DWPOdeCnjBwKtv98y5dSws3itPqfce5TVa0s59LRiA= github.com/tus/tusd v1.6.0 h1:IU9Z2Z5FZfHIap6NPFbPItyx/eU6aN87z4ya/mPzS4g= github.com/tus/tusd v1.6.0/go.mod h1:ygrT4B9ZSb27dx3uTnobX5nOFDnutBL6iWKLH4+KpA0= +github.com/tv42/httpunix v0.0.0-20150427012821-b75d8614f926/go.mod h1:9ESjWnEqriFuLhtthL60Sar/7RFoluCcXsuvEwTV5KM= github.com/uber-go/atomic v1.3.2/go.mod h1:/Ct5t2lcmbJ4OSe/waGBoaVvVqtO0bmtfVNex1PFV8g= github.com/uber/jaeger-client-go v2.29.1+incompatible/go.mod h1:WVhlPFC8FDjOFMMWRy2pZqQJSXxYSwNYOkTr/Z6d3Kk= github.com/uber/jaeger-lib v2.4.1+incompatible/go.mod h1:ComeNDZlWwrWnDv8aPp0Ba6+uUTzImX/AauajbLI56U= @@ -1290,6 +1340,7 @@ go.etcd.io/etcd/client/pkg/v3 v3.5.0/go.mod h1:IJHfcCEKxYu1Os13ZdwCwIUTUVGYTSAM3 go.etcd.io/etcd/client/pkg/v3 v3.5.1 h1:XIQcHCFSG53bJETYeRJtIxdLv2EWRGxcfzR8lSnTH4E= go.etcd.io/etcd/client/pkg/v3 v3.5.1/go.mod h1:IJHfcCEKxYu1Os13ZdwCwIUTUVGYTSAM3YSwc9/Ac1g= go.etcd.io/etcd/client/v2 v2.305.0/go.mod h1:h9puh54ZTgAKtEbut2oe9P4L/oqKCVB6xsXlzd7alYQ= +go.etcd.io/etcd/client/v2 v2.305.1/go.mod h1:pMEacxZW7o8pg4CrFE7pquyCJJzZvkvdD2RibOCCCGs= go.etcd.io/etcd/client/v3 v3.5.0/go.mod h1:AIKXXVX/DQXtfTEqBryiLTUXwON+GuvO6Z7lLS/oTh0= go.etcd.io/etcd/client/v3 v3.5.1 h1:oImGuV5LGKjCqXdjkMHCyWa5OO1gYKCnC/1sgdfj1Uk= go.etcd.io/etcd/client/v3 v3.5.1/go.mod h1:OnjH4M8OnAotwaB2l9bVgZzRFKru7/ZMoS46OtKyd3Q= @@ -1381,6 +1432,7 @@ golang.org/x/crypto v0.0.0-20210421170649-83a5a9bb288b/go.mod h1:T9bdIzuCu7OtxOm golang.org/x/crypto v0.0.0-20210513164829-c07d793c2f9a/go.mod h1:P+XmwS30IXTQdn5tA2iutPOUgjI07+tq3H3K9MVA1s8= golang.org/x/crypto v0.0.0-20210616213533-5ff15b29337e/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= +golang.org/x/crypto v0.0.0-20210817164053-32db794688a5/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20211108221036-ceb1ce70b4fa h1:idItI2DDfCokpg0N51B2VtiLdJ4vAuXC9fnCb2gACo4= golang.org/x/crypto v0.0.0-20211108221036-ceb1ce70b4fa/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= @@ -1426,6 +1478,7 @@ golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.1/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/mod v0.5.0/go.mod h1:5OXOZSfqPIIbmVBIIKWRFfZjPR0E5r58TLhUjH0a2Ro= golang.org/x/net v0.0.0-20180530234432-1e491301e022/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180811021610-c39426892332/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -1482,6 +1535,7 @@ golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v golang.org/x/net v0.0.0-20210316092652-d523dce5a7f4/go.mod h1:RBQZq4jEuRlivfhVLdyRGr576XBO4/greRjx4P4O3yc= golang.org/x/net v0.0.0-20210326060303-6b1517762897/go.mod h1:uSPa2vr4CLtc/ILN5odXGNXS6mhrKVzTaCXzk9m6W3k= golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM= +golang.org/x/net v0.0.0-20210410081132-afb366fc7cd1/go.mod h1:9tjilg8BloeKEkVJvy7fQ90B1CfIiPueXVOjqfkSzI8= golang.org/x/net v0.0.0-20210428140749-89ef3d95e781/go.mod h1:OJAsFXCWl8Ukc7SiCT/9KSuxbyM7479/AVlXFRxuMCk= golang.org/x/net v0.0.0-20210503060351-7fd8e65b6420/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20210505024714-0287a6fb4125/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= @@ -1490,6 +1544,7 @@ golang.org/x/net v0.0.0-20210525063256-abc453219eb5/go.mod h1:9nx3DQGgdP8bBQD5qx golang.org/x/net v0.0.0-20210614182718-04defd469f4e/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20210726213435-c6fcb2dbf985/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= +golang.org/x/net v0.0.0-20210813160813-60bc85c4be6d/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20211020060615-d418f374d309/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20211111083644-e5c967477495 h1:cjxxlQm6d4kYbhpZ2ghvmI8xnq0AG+jXmzrhzfkyu5A= golang.org/x/net v0.0.0-20211111083644-e5c967477495/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= @@ -1509,6 +1564,7 @@ golang.org/x/oauth2 v0.0.0-20210514164344-f6687ab2804c/go.mod h1:KelEdhl1UZF7XfJ golang.org/x/oauth2 v0.0.0-20210628180205-a41e5a781914/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= golang.org/x/oauth2 v0.0.0-20210805134026-6f1e6394065a/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= golang.org/x/oauth2 v0.0.0-20210819190943-2bc19b11175f/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= +golang.org/x/oauth2 v0.0.0-20211005180243-6b3c2da341f1/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8 h1:RerP+noqYHUQ8CMRcPlC2nvTa4dcBIjegkuWdcUDuqg= golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -1572,6 +1628,7 @@ golang.org/x/sys v0.0.0-20200113162924-86b910548bc1/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200120151820-655fe14d7479/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200122134326-e047566fdf82/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200124204421-9fbb57f87de9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200212091648-12a6c2dcc1e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -1621,10 +1678,16 @@ golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210806184541-e5e7981a1069/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210816183151-1e6c022a8912/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210823070655-63515b42dcdf/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210908233432-aa78b53d3365/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210921065528-437939a70204/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20211110154304-99a53858aa08 h1:WecRHqgE09JBkh/584XIE6PMz5KKE/vER4izNUi30AQ= +golang.org/x/sys v0.0.0-20211007075335-d3039528d8ac/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20211110154304-99a53858aa08/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20211124211545-fe61309f8881/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20211205182925-97ca703d548d h1:FjkYO/PPp4Wi0EAUOVLxePm7qVW4r4ctbWpURyuOD0E= +golang.org/x/sys v0.0.0-20211205182925-97ca703d548d/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201113234701-d7a72108b828/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw= golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= @@ -1763,6 +1826,12 @@ google.golang.org/api v0.48.0/go.mod h1:71Pr1vy+TAZRPkPs/xlCf5SsU8WjuAWv1Pfjbtuk google.golang.org/api v0.50.0/go.mod h1:4bNT5pAuq5ji4SRZm+5QIkjny9JAyVD/3gaSihNefaw= google.golang.org/api v0.51.0/go.mod h1:t4HdrdoNgyN5cbEfm7Lum0lcLDLiise1F8qDKX00sOU= google.golang.org/api v0.54.0/go.mod h1:7C4bFFOvVDGXjfDTAsgGwDgAxRDeQ4X8NvUedIt6z3k= +google.golang.org/api v0.55.0/go.mod h1:38yMfeP1kfjsl8isn0tliTjIb1rJXcQi4UXlbqivdVE= +google.golang.org/api v0.56.0/go.mod h1:38yMfeP1kfjsl8isn0tliTjIb1rJXcQi4UXlbqivdVE= +google.golang.org/api v0.57.0/go.mod h1:dVPlbZyBo2/OjBpmvNdpn2GRm6rPy75jyU7bmhdrMgI= +google.golang.org/api v0.59.0/go.mod h1:sT2boj7M9YJxZzgeZqXogmhfmRWDtPzT31xkieUbuZU= +google.golang.org/api v0.61.0/go.mod h1:xQRti5UdCmoCEqFxcz93fTl338AVqDgyaDRuOZ3hg9I= +google.golang.org/api v0.62.0/go.mod h1:dKmwPCydfsad4qCH08MSdgWjfHOyfpd4VtDGgRFdavw= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= @@ -1827,9 +1896,21 @@ google.golang.org/genproto v0.0.0-20210716133855-ce7ef5c701ea/go.mod h1:AxrInvYm google.golang.org/genproto v0.0.0-20210728212813-7823e685a01f/go.mod h1:ob2IJxKrgPT52GcgX759i1sleT07tiKowYBGbczaW48= google.golang.org/genproto v0.0.0-20210805201207-89edb61ffb67/go.mod h1:ob2IJxKrgPT52GcgX759i1sleT07tiKowYBGbczaW48= google.golang.org/genproto v0.0.0-20210813162853-db860fec028c/go.mod h1:cFeNkxwySK631ADgubI+/XFU/xp8FD5KIVV4rj8UC5w= +google.golang.org/genproto v0.0.0-20210821163610-241b8fcbd6c8/go.mod h1:eFjDcFEctNawg4eG61bRv87N7iHBWyVhJu7u1kqDUXY= +google.golang.org/genproto v0.0.0-20210828152312-66f60bf46e71/go.mod h1:eFjDcFEctNawg4eG61bRv87N7iHBWyVhJu7u1kqDUXY= +google.golang.org/genproto v0.0.0-20210831024726-fe130286e0e2/go.mod h1:eFjDcFEctNawg4eG61bRv87N7iHBWyVhJu7u1kqDUXY= +google.golang.org/genproto v0.0.0-20210903162649-d08c68adba83/go.mod h1:eFjDcFEctNawg4eG61bRv87N7iHBWyVhJu7u1kqDUXY= +google.golang.org/genproto v0.0.0-20210909211513-a8c4777a87af/go.mod h1:eFjDcFEctNawg4eG61bRv87N7iHBWyVhJu7u1kqDUXY= +google.golang.org/genproto v0.0.0-20210924002016-3dee208752a0/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= +google.golang.org/genproto v0.0.0-20211008145708-270636b82663/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= google.golang.org/genproto v0.0.0-20211020151524-b7c3a969101a/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= -google.golang.org/genproto v0.0.0-20211129164237-f09f9a12af12 h1:DN5b3HU13J4sMd/QjDx34U6afpaexKTDdop+26pdjdk= +google.golang.org/genproto v0.0.0-20211028162531-8db9c33dc351/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= +google.golang.org/genproto v0.0.0-20211118181313-81c1377c94b1/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= google.golang.org/genproto v0.0.0-20211129164237-f09f9a12af12/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= +google.golang.org/genproto v0.0.0-20211203200212-54befc351ae9/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= +google.golang.org/genproto v0.0.0-20211206160659-862468c7d6e0/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= +google.golang.org/genproto v0.0.0-20211208223120-3a66f561d7aa h1:I0YcKz0I7OAhddo7ya8kMnvprhcWM045PmkBdMO9zN0= +google.golang.org/genproto v0.0.0-20211208223120-3a66f561d7aa/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= google.golang.org/grpc v1.8.0/go.mod h1:yo6s7OP7yaDglbqo1J04qKzAhqBH6lvTonzMVmEdcZw= google.golang.org/grpc v1.17.0/go.mod h1:6QZJwpn2B+Zp71q/5VxRsJ6NXXVCE5NRUHRo+f3cWCs= google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= @@ -1859,6 +1940,7 @@ google.golang.org/grpc v1.38.0/go.mod h1:NREThFqKR1f3iQ6oBuvc5LadQuXVGo9rkm5ZGrQ google.golang.org/grpc v1.39.0/go.mod h1:PImNr+rS9TWYb2O4/emRugxiyHZ5JyHW5F+RPnDzfrE= google.golang.org/grpc v1.39.1/go.mod h1:PImNr+rS9TWYb2O4/emRugxiyHZ5JyHW5F+RPnDzfrE= google.golang.org/grpc v1.40.0/go.mod h1:ogyxbiOoUXAkP+4+xa6PZSE9DZgIHtSpzjDTB9KAK34= +google.golang.org/grpc v1.40.1/go.mod h1:ogyxbiOoUXAkP+4+xa6PZSE9DZgIHtSpzjDTB9KAK34= google.golang.org/grpc v1.41.0/go.mod h1:U3l9uK9J0sini8mHphKoXyaqDA/8VyGnDee1zzIUK6k= google.golang.org/grpc v1.42.0/go.mod h1:k+4IHHFw41K8+bbowsex27ge2rCb65oeWqe4jJ590SU= google.golang.org/grpc v1.43.0 h1:Eeu7bZtDZ2DpRCsLhUlcrLnvYaMK1Gz86a+hMVvELmM= @@ -1903,8 +1985,8 @@ gopkg.in/ini.v1 v1.51.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= gopkg.in/ini.v1 v1.51.1/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= gopkg.in/ini.v1 v1.57.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= gopkg.in/ini.v1 v1.62.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= -gopkg.in/ini.v1 v1.63.2 h1:tGK/CyBg7SMzb60vP1M03vNZ3VDu3wGQJwn7Sxi9r3c= -gopkg.in/ini.v1 v1.63.2/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= +gopkg.in/ini.v1 v1.66.2 h1:XfR1dOYubytKy4Shzc2LHrrGhU0lDCfDGG1yLPmpgsI= +gopkg.in/ini.v1 v1.66.2/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= gopkg.in/ns1/ns1-go.v2 v2.4.4/go.mod h1:GMnKY+ZuoJ+lVLL+78uSTjwTz2jMazq6AfGKQOYhsPk= gopkg.in/resty.v1 v1.12.0/go.mod h1:mDo4pnntr5jdWRML875a/NmxYqAlA73dVijT2AXvQQo= gopkg.in/square/go-jose.v2 v2.4.0/go.mod h1:M9dMgbHiYLoDGQrXy7OpJDJWiKiU//h+vD76mk0e1AI= From b6e518339f21b41b78b39b8a00277f4bc9c5eed1 Mon Sep 17 00:00:00 2001 From: Swikriti Tripathi Date: Wed, 15 Dec 2021 10:58:53 +0545 Subject: [PATCH 51/56] remove a test scenario from expected to fail file --- .drone.env | 4 ++-- tests/acceptance/expected-failures-API-on-OCIS-storage.md | 3 --- 2 files changed, 2 insertions(+), 5 deletions(-) diff --git a/.drone.env b/.drone.env index fd230a574b..60b17ac9e8 100644 --- a/.drone.env +++ b/.drone.env @@ -1,6 +1,6 @@ # The test runner source for API tests -CORE_COMMITID=2987544bfd6f6823df68f6ea392ba41c1ea4f07c -CORE_BRANCH=acceptance-test-changes-waiting-2021-11 +CORE_COMMITID=8f0575dda6a23643f51e97321fa8de34a093e5dc +CORE_BRANCH=bug-demo-test-webdav-put # The test runner source for UI tests WEB_COMMITID=ed4a8b32240b59666a6c2b162d95b50ff7191eb8 diff --git a/tests/acceptance/expected-failures-API-on-OCIS-storage.md b/tests/acceptance/expected-failures-API-on-OCIS-storage.md index 1f3c7b46e5..a53dd46f52 100644 --- a/tests/acceptance/expected-failures-API-on-OCIS-storage.md +++ b/tests/acceptance/expected-failures-API-on-OCIS-storage.md @@ -1482,6 +1482,3 @@ Not everything needs to be implemented for ocis. While the oc10 testsuite covers ### [Content-type is not multipart/byteranges when downloading file with Range Header](https://github.com/owncloud/ocis/issues/2677) - [apiWebdavOperations/downloadFile.feature:169](https://github.com/owncloud/core/blob/master/tests/acceptance/features/apiWebdavOperations/downloadFile.feature#L169) - [apiWebdavOperations/downloadFile.feature:170](https://github.com/owncloud/core/blob/master/tests/acceptance/features/apiWebdavOperations/downloadFile.feature#L170) - -### [send PUT requests to another user's webDav endpoints as normal user](https://github.com/owncloud/ocis/issues/2759) -- [apiAuthWebDav/webDavPUTAuth.feature:40](https://github.com/owncloud/core/blob/master/tests/acceptance/features/apiAuthWebDav/webDavPUTAuth.feature#L40) From 9b17b88b0f04bbd5d65f0938b13465aa9528096e Mon Sep 17 00:00:00 2001 From: Swikriti Tripathi Date: Wed, 15 Dec 2021 16:09:16 +0545 Subject: [PATCH 52/56] Bump the commit id --- .drone.env | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.drone.env b/.drone.env index 60b17ac9e8..484f572405 100644 --- a/.drone.env +++ b/.drone.env @@ -1,6 +1,6 @@ # The test runner source for API tests -CORE_COMMITID=8f0575dda6a23643f51e97321fa8de34a093e5dc -CORE_BRANCH=bug-demo-test-webdav-put +CORE_COMMITID=38ee68e6358443e980ba5e7036cf1fb554443814 +CORE_BRANCH=acceptance-test-changes-waiting-2021-11 # The test runner source for UI tests WEB_COMMITID=ed4a8b32240b59666a6c2b162d95b50ff7191eb8 From 8607525f65e5a83049b06313f36ebf45b7d83da1 Mon Sep 17 00:00:00 2001 From: Willy Kloucek Date: Tue, 14 Dec 2021 14:04:16 +0100 Subject: [PATCH 53/56] update reva --- changelog/unreleased/update-reva.md | 8 ++++++++ go.mod | 6 +++--- go.sum | 16 ++++++++-------- 3 files changed, 19 insertions(+), 11 deletions(-) create mode 100644 changelog/unreleased/update-reva.md diff --git a/changelog/unreleased/update-reva.md b/changelog/unreleased/update-reva.md new file mode 100644 index 0000000000..d5924214b4 --- /dev/null +++ b/changelog/unreleased/update-reva.md @@ -0,0 +1,8 @@ +Enhancement: Update REVA to xxx + +Updated REVA to xxx +This update includes: + + * TODO: update changelog before oCIS release + +https://github.com/owncloud/ocis/pull/2878 \ No newline at end of file diff --git a/go.mod b/go.mod index 2c969652e0..cc6da7c02a 100644 --- a/go.mod +++ b/go.mod @@ -19,7 +19,7 @@ require ( github.com/blevesearch/bleve/v2 v2.2.2 github.com/coreos/go-oidc/v3 v3.1.0 github.com/cs3org/go-cs3apis v0.0.0-20211104090126-8e972dca8304 - github.com/cs3org/reva v1.17.1-0.20211212151213-778de37266ff + github.com/cs3org/reva v1.17.1-0.20211215132908-5cde0187454b github.com/disintegration/imaging v1.6.2 github.com/glauth/glauth/v2 v2.0.0-20211021011345-ef3151c28733 github.com/go-chi/chi/v5 v5.0.7 @@ -82,7 +82,7 @@ require ( github.com/Masterminds/sprig v2.22.0+incompatible // indirect github.com/Microsoft/go-winio v0.5.1 // indirect github.com/ProtonMail/go-crypto v0.0.0-20210920160938-87db9fbc61c7 // indirect - github.com/ReneKroon/ttlcache/v2 v2.9.0 // indirect + github.com/ReneKroon/ttlcache/v2 v2.10.0 // indirect github.com/RoaringBitmap/roaring v0.9.4 // indirect github.com/acomagu/bufpipe v1.0.3 // indirect github.com/asaskevich/govalidator v0.0.0-20210307081110-f21760c49a8d // indirect @@ -171,7 +171,7 @@ require ( github.com/miekg/dns v1.1.43 // indirect github.com/mileusna/useragent v1.0.2 // indirect github.com/minio/md5-simd v1.1.2 // indirect - github.com/minio/minio-go/v7 v7.0.16 // indirect + github.com/minio/minio-go/v7 v7.0.18 // indirect github.com/minio/sha256-simd v1.0.0 // indirect github.com/mitchellh/copystructure v1.2.0 // indirect github.com/mitchellh/go-homedir v1.1.0 // indirect diff --git a/go.sum b/go.sum index 83b3bd6054..ddb0a5c9c7 100644 --- a/go.sum +++ b/go.sum @@ -103,8 +103,8 @@ github.com/ProtonMail/go-crypto v0.0.0-20210920160938-87db9fbc61c7/go.mod h1:z4/ github.com/PuerkitoBio/purell v1.1.0/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE= -github.com/ReneKroon/ttlcache/v2 v2.9.0 h1:NzwfErbifoNA3djEGwQJXKp/386imbyrc6Qmns5IX7c= -github.com/ReneKroon/ttlcache/v2 v2.9.0/go.mod h1:mBxvsNY+BT8qLLd6CuAJubbKo6r0jh3nb5et22bbfGY= +github.com/ReneKroon/ttlcache/v2 v2.10.0 h1:y4g2gs2fSqAugSseq7vUB1jClqzT/e0CjSrMJywoWa8= +github.com/ReneKroon/ttlcache/v2 v2.10.0/go.mod h1:mBxvsNY+BT8qLLd6CuAJubbKo6r0jh3nb5et22bbfGY= github.com/RoaringBitmap/roaring v0.9.4 h1:ckvZSX5gwCRaJYBNe7syNawCU5oruY9gQmjXlp4riwo= github.com/RoaringBitmap/roaring v0.9.4/go.mod h1:icnadbWcNyfEHlYdr+tDlOTih1Bf/h+rzPpv4sbomAA= github.com/Shopify/sarama v1.19.0/go.mod h1:FVkBWblsNy7DGZRfXLU0O9RCGt5g3g3yEuWXgklEdEo= @@ -312,8 +312,10 @@ github.com/crewjam/saml v0.4.5/go.mod h1:qCJQpUtZte9R1ZjUBcW8qtCNlinbO363ooNl02S github.com/cs3org/cato v0.0.0-20200828125504-e418fc54dd5e/go.mod h1:XJEZ3/EQuI3BXTp/6DUzFr850vlxq11I6satRtz0YQ4= github.com/cs3org/go-cs3apis v0.0.0-20211104090126-8e972dca8304 h1:e/nIPR518vyvrulo9goAZTtYD6gFfu/2/9MDe6mTGcw= github.com/cs3org/go-cs3apis v0.0.0-20211104090126-8e972dca8304/go.mod h1:UXha4TguuB52H14EMoSsCqDj7k8a/t7g4gVP+bgY5LY= -github.com/cs3org/reva v1.17.1-0.20211212151213-778de37266ff h1:IUj5GfxDa8WgNzeH+5MnJREzjoneejygmnMgU2hAWCg= -github.com/cs3org/reva v1.17.1-0.20211212151213-778de37266ff/go.mod h1:gtsVzMfDrUiUjH6qlHx+QqiRKsSYjVO6wEcCzANiqUg= +github.com/cs3org/reva v1.17.1-0.20211215074511-8236ffa3f824 h1:eVNZAbZehrPQLtYCkqoGCU64V7xA4po4U7/Pa0twH9Q= +github.com/cs3org/reva v1.17.1-0.20211215074511-8236ffa3f824/go.mod h1:iXr05MZR63XQXX657oEtylokVz0SmmJU88uTju7+6uo= +github.com/cs3org/reva v1.17.1-0.20211215132908-5cde0187454b h1:pXH63+1AWhp61MzLQZgndp0GcpR8boci7D1VnR2hINA= +github.com/cs3org/reva v1.17.1-0.20211215132908-5cde0187454b/go.mod h1:iXr05MZR63XQXX657oEtylokVz0SmmJU88uTju7+6uo= github.com/cubewise-code/go-mime v0.0.0-20200519001935-8c5762b177d8 h1:Z9lwXumT5ACSmJ7WGnFl+OMLLjpz5uR2fyz7dC255FI= github.com/cubewise-code/go-mime v0.0.0-20200519001935-8c5762b177d8/go.mod h1:4abs/jPXcmJzYoYGF91JF9Uq9s/KL5n1jvFDix8KcqY= github.com/cyberdelia/templates v0.0.0-20141128023046-ca7fffd4298c/go.mod h1:GyV+0YP4qX0UQ7r2MoYZ+AvYDp12OF5yg4q8rGnyNh4= @@ -936,8 +938,8 @@ github.com/mileusna/useragent v1.0.2/go.mod h1:3d8TOmwL/5I8pJjyVDteHtgDGcefrFUX4 github.com/minio/md5-simd v1.1.0/go.mod h1:XpBqgZULrMYD3R+M28PcmP0CkI7PEMzB3U77ZrKZ0Gw= github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34= github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM= -github.com/minio/minio-go/v7 v7.0.16 h1:GspaSBS8lOuEUCAqMe0W3UxSoyOA4b4F8PTspRVI+k4= -github.com/minio/minio-go/v7 v7.0.16/go.mod h1:pUV0Pc+hPd1nccgmzQF/EXh48l/Z/yps6QPF1aaie4g= +github.com/minio/minio-go/v7 v7.0.18 h1:fncn6iacnK+i2uYfNc5aVPG7bEqQH0nU4yAGMSunY0w= +github.com/minio/minio-go/v7 v7.0.18/go.mod h1:SyQ1IFeJuaa+eV5yEDxW7hYE1s5VVq5sgImDe27R+zg= github.com/minio/sha256-simd v0.1.1/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM= github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g= github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM= @@ -1370,10 +1372,8 @@ go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.2 go.opentelemetry.io/otel v1.2.0/go.mod h1:aT17Fk0Z1Nor9e0uisf98LrntPGMnk4frBO9+dkf69I= go.opentelemetry.io/otel v1.3.0 h1:APxLf0eiBwLl+SOXiJJCVYzA1OOJNyAoV8C5RNRyy7Y= go.opentelemetry.io/otel v1.3.0/go.mod h1:PWIKzi6JCp7sM0k9yZ43VX+T345uNbAkDKwHVjb2PTs= -go.opentelemetry.io/otel/exporters/jaeger v1.2.0/go.mod h1:KJLFbEMKTNPIfOxcg/WikIozEoKcPgJRz3Ce1vLlM8E= go.opentelemetry.io/otel/exporters/jaeger v1.3.0 h1:HfydzioALdtcB26H5WHc4K47iTETJCdloL7VN579/L0= go.opentelemetry.io/otel/exporters/jaeger v1.3.0/go.mod h1:KoYHi1BtkUPncGSRtCe/eh1ijsnePhSkxwzz07vU0Fc= -go.opentelemetry.io/otel/sdk v1.2.0/go.mod h1:jNN8QtpvbsKhgaC6V5lHiejMoKD+V8uadoSafgHPx1U= go.opentelemetry.io/otel/sdk v1.3.0 h1:3278edCoH89MEJ0Ky8WQXVmDQv3FX4ZJ3Pp+9fJreAI= go.opentelemetry.io/otel/sdk v1.3.0/go.mod h1:rIo4suHNhQwBIPg9axF8V9CA72Wz2mKF1teNrup8yzs= go.opentelemetry.io/otel/trace v1.2.0/go.mod h1:N5FLswTubnxKxOJHM7XZC074qpeEdLy3CgAVsdMucK0= From fad9cad0153c292f09bf8cca477db8d858b6cf87 Mon Sep 17 00:00:00 2001 From: Willy Kloucek Date: Wed, 15 Dec 2021 15:04:36 +0100 Subject: [PATCH 54/56] update expected failures --- ...xpected-failures-webUI-on-OCIS-storage-ocisSmokeTest.md | 3 --- .../acceptance/expected-failures-webUI-on-OCIS-storage.md | 7 ------- 2 files changed, 10 deletions(-) diff --git a/tests/acceptance/expected-failures-webUI-on-OCIS-storage-ocisSmokeTest.md b/tests/acceptance/expected-failures-webUI-on-OCIS-storage-ocisSmokeTest.md index da270db559..8bf9c61e4e 100644 --- a/tests/acceptance/expected-failures-webUI-on-OCIS-storage-ocisSmokeTest.md +++ b/tests/acceptance/expected-failures-webUI-on-OCIS-storage-ocisSmokeTest.md @@ -10,9 +10,6 @@ Other free text and Markdown formatting can be used elsewhere in the document if Only the web scenarios tagged ocisSmokeTest are run by default in OCIS CI. This file lists the expected-failures of those ocisSmokeTest scenarios. -### [unexpected behavior when renaming files](https://github.com/owncloud/web/issues/4893) -- [webUIFilesCopy/copy.feature:36](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIFilesCopy/copy.feature#L36) - ### [enable re-sharing is not possible](https://github.com/owncloud/ocis/issues/1743) - [webUISharingFilePermissionMultipleUsers/shareFileWithMultipleUsers.feature:67](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingFilePermissionMultipleUsers/shareFileWithMultipleUsers.feature#L67) - [webUISharingFilePermissionMultipleUsers/shareFileWithMultipleUsers.feature:66](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUISharingFilePermissionMultipleUsers/shareFileWithMultipleUsers.feature#L66) diff --git a/tests/acceptance/expected-failures-webUI-on-OCIS-storage.md b/tests/acceptance/expected-failures-webUI-on-OCIS-storage.md index e187d962b3..5fe9b7f132 100644 --- a/tests/acceptance/expected-failures-webUI-on-OCIS-storage.md +++ b/tests/acceptance/expected-failures-webUI-on-OCIS-storage.md @@ -8,13 +8,6 @@ Level-3 headings should be used for the references to the relevant issues. Inclu Other free text and markdown formatting can be used elsewhere in the document if needed. But if you want to explain something about the issue, then please post that in the issue itself. -### [unexpected behavior when renaming files](https://github.com/owncloud/web/issues/4893) -- [webUIFilesCopy/copy.feature:36](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIFilesCopy/copy.feature#L36) -- [webUIFilesCopy/copy.feature:68](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIFilesCopy/copy.feature#L68) -- [webUIFilesCopy/copy.feature:69](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIFilesCopy/copy.feature#L69) -- [webUIFilesCopy/copy.feature:70](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIFilesCopy/copy.feature#L70) -- [webUIFilesCopy/copy.feature:71](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIFilesCopy/copy.feature#L71) - ### [Media Viewer does not support mp3 files](https://github.com/owncloud/ocis/issues/1106) - [webUIPreview/imageMediaViewer.feature:84](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIPreview/imageMediaViewer.feature#L84) - [webUIPreview/imageMediaViewer.feature:91](https://github.com/owncloud/web/blob/master/tests/acceptance/features/webUIPreview/imageMediaViewer.feature#L91) From 33d418f533d6352e1374ac99f2af07d4255c0cbc Mon Sep 17 00:00:00 2001 From: Willy Kloucek <34452982+wkloucek@users.noreply.github.com> Date: Wed, 15 Dec 2021 14:13:16 +0000 Subject: [PATCH 55/56] Automated changelog update [skip ci] --- CHANGELOG.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index a3afca8451..ddc3ca84e6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ The following sections list the changes for unreleased. * Change - Update libre-graph-api to v0.3.0: [#2858](https://github.com/owncloud/ocis/pull/2858) * Change - Return not found when updating non existent space: [#2869](https://github.com/cs3org/reva/pull/2869) +* Enhancement - Add new file url of the app provider to the ocs capabilities: [#2884](https://github.com/owncloud/ocis/pull/2884) * Enhancement - Support signature auth in the public share auth middleware: [#2831](https://github.com/owncloud/ocis/pull/2831) ## Details @@ -24,6 +25,15 @@ The following sections list the changes for unreleased. https://github.com/cs3org/reva/pull/2869 +* Enhancement - Add new file url of the app provider to the ocs capabilities: [#2884](https://github.com/owncloud/ocis/pull/2884) + + We've added the new file capability of the app provider to the ocs capabilities, so that clients + can discover this url analogous to the app list and file open urls. + + https://github.com/owncloud/ocis/pull/2884 + https://github.com/cs3org/reva/pull/2379 + https://github.com/owncloud/web/pull/5890#issuecomment-993905242 + * Enhancement - Support signature auth in the public share auth middleware: [#2831](https://github.com/owncloud/ocis/pull/2831) Enabled public share requests to be authenticated using the public share signature. From 8715969556fed6126108bee3249997c67e86bcba Mon Sep 17 00:00:00 2001 From: Willy Kloucek <34452982+wkloucek@users.noreply.github.com> Date: Wed, 15 Dec 2021 16:19:21 +0000 Subject: [PATCH 56/56] Automated changelog update [skip ci] --- CHANGELOG.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index ddc3ca84e6..10c72f6340 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ The following sections list the changes for unreleased. * Change - Return not found when updating non existent space: [#2869](https://github.com/cs3org/reva/pull/2869) * Enhancement - Add new file url of the app provider to the ocs capabilities: [#2884](https://github.com/owncloud/ocis/pull/2884) * Enhancement - Support signature auth in the public share auth middleware: [#2831](https://github.com/owncloud/ocis/pull/2831) +* Enhancement - Update REVA to xxx: [#2878](https://github.com/owncloud/ocis/pull/2878) ## Details @@ -39,6 +40,14 @@ The following sections list the changes for unreleased. Enabled public share requests to be authenticated using the public share signature. https://github.com/owncloud/ocis/pull/2831 + +* Enhancement - Update REVA to xxx: [#2878](https://github.com/owncloud/ocis/pull/2878) + + Updated REVA to xxx This update includes: + + * TODO: update changelog before oCIS release + + https://github.com/owncloud/ocis/pull/2878 # Changelog for [1.16.0] (2021-12-10) The following sections list the changes for 1.16.0.