mirror/opencloud
1
0
Fork 0
mirror of https://github.com/opencloud-eu/opencloud.git synced 2026-02-26 15:39:18 -06:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #1817 from owncloud/migration-adr-docs

Browse Source
This commit is contained in:
Alex Unger
2021-03-23 12:19:23 +01:00
committed by GitHub
parent 3ba9005992 63abba7e6a
commit 39937cec1b
2 changed files with 370 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

81
docs/ocis/adr/0004-support-hot-migration.md Normal file
View File

@@ -0,0 +1,81 @@
---
title: "4. Support Hot Migration"
date: 2020-12-09T20:21:00+01:00
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0004-support-hot-migration.md
---
# 4. Support Hot Migration
* Status: proposed
* Deciders: @butonic, @micbar, @dragotin, @hodyroff, @pmaier1
* Date: 2021-03-16
Technical Story: \[description | ticket/issue URL\]
## Context and Problem Statement
Migration is one of the most important topics of the oCIS story. We need to provide a concept how to migrate from oC10 to oCIS.
## Decision Drivers
- Do not lose file blob or meta data.
- To prevent a sync surge from clients the etag for files should be migrated.
- To prevent internal links from breaking or pointing to wrong files the file id of existing files needs to be migrated.
- To prevent data loss trash and version blobs should be migrated.
- Existing shares like public links and federated shares must remain functional after the migration.
- To prevent internal shares the share type, permissions and expiry needs to be migrated.
- To prevent public links from breaking the url token, permissions, expiry and password needs to be migrated.
- *What about federated shares?*
- *What about additional share permissions, eg. comment on office files?*
- Legacy clients need to keep working
- To keep existing clients working the `remote.php/webdav` and `dav/files/<username>` webdav endpoints as well as the ocs API need to be available.
- *What about [app passwords/tokens](https://doc.owncloud.com/server/user_manual/personal_settings/security.html#app-passwords-tokens)?*
## Considered Options
1. Cold Migration: migrate data while systems are not online, so no user interaction happens in between.
2. Hot Migration: one or both systems are online during migration.
## Decision Outcome
Chosen option: "\[option 1\]", because \[justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | … | comes out best (see below)\].
### Positive Consequences
- \[e.g., improvement of quality attribute satisfaction, follow-up decisions required, …\]
- …
### Negative Consequences
- \[e.g., compromising quality attribute, follow-up decisions required, …\]
- …
## Pros and Cons of the Options
### Cold Migration
The migration happens while the service is offline. File metadata, blobs and share data is exported from ownCloud 10 and imported in oCIS. This can happen user by user, where every user export would contain the file blobs, their metadata, trash, versions, shares and all metadata that belongs to the users storage. To prevent group shares from breaking, users in the same groups must be migrated in batch. Depending on the actual group shares in an instance this may effectively require a complete migration in a single batch.
- Good, because oCIS can be tested in a staging system without writing to the production system.
- Good, because file layout on disk can be changed to support new storage driver capabilities.
- Bad, because the export and import might require significant amounts of storage.
- Bad, because a rollback to the state before the migration might cause data loss of the changes that happend in between.
- Bad, because the cold migration can mean significant downtime.
### Hot Migration
The migration happens in subsequent stages while the service is online.
- Good, because the admin can migrate users from old to new backend in a controlled way.
- Good, because users and admins can learn to trust the new system.
- Good, because there can be preparations even long before the migrations happens in parallel on the oC10 codebase, ie. addition of metadata that is needed while the system operates.
- Good, because the downtime of the system can be fairly small.
- Bad, because it is more complex and might drag on for a long time.
## Links
- [Clarify responsibilities of share providers and storage providers · Issue #1377 · cs3org/reva (github.com)](https://github.com/cs3org/reva/issues/1377) because the share manager for oCIS should store share information on the storage system. And [storageprovider should persist share creator · Issue #93 · cs3org/cs3apis (github.com)](https://github.com/cs3org/cs3apis/issues/93) finally: [eos: store share id in inherited xattr · Issue #543 · cs3org/reva (github.com)](https://github.com/cs3org/reva/issues/543)

289
docs/ocis/migration.md Normal file
View File

@@ -0,0 +1,289 @@
---
title: "Migration"
date: 2021-03-16T16:17:00+01:00
weight: 41
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis
geekdocFilePath: migration.md
---
## Migration
The migration happens in subsequent stages while the service is online. First all users need to migrate to the new architecture, then the data on disk can be migrated user by user by switching the storage driver.
### User Stories
As an admin I need to avoid downtime.
As an admin I want to migrate certain groups of users before others.
As a user, I need a seamless migration and not lose data by any chance.
### Stage-0
Is the pre-migration stage having a functional ownCloud 10 instance.
### Stage-1
Introduce OpenID Connect to server and Clients
### Stage-2
Install and introduce ownCloud Web and let users test it voluntarily.
### Stage-3
Start oCIS backend and make read only tests on existing data using the `owncloud` storage driver which will read (and write)
- blobs from the same datadirectory layout as in ownCloud 10 and
- metadata from the ownCloud 10 databas
The oCIS share manager will read share infomation from the owncloud database as well.
- [ ] *we need a share manager that can read from the oc 10 db as well as from whatever new backend will be used for a pure oCIS setup. Currently, that would be the json file. Or that is migrated after all users have switched to oCIS. -- jfd*
### Stage-4
Test writing data with oCIS into the existing ownCloud 10 datafolder using the `owncloud` storage driver.
### Stage-5
Introduce reverse proxy and switch over early adoptors, let admins gain trust in the new backend by comparing metrics of the two systems and having it running in parallel.
### Stage-6
Voluntary transition period and subsequent hard deadline for all users
### Stage-7
Disable oc10 in the proxy, all requests are now handled by oCIS, shut down oc10 web servers and redis (or keep for calendar & contacts only? rip out files from oCIS?)
### Stage-8
User by user storage migration from owncloud driver to `ocis`/`s3ng`/`cephfs`...
### Stage-9
Migrate share data to &lt;yet to determine&gt; share manager backend and shut down owncloud database
### Stage-10
Profit! (db for file metadata no longer necessary, less maintenance effort)
## Architectural differences
The fundamental difference between ownCloud 10 and oCIS is that the file metadata is moved from the database in the `oc_filecache` table (which is misnamed, as it actually is an index) to the storage provider who can place metadata as close to the underlying storage system as possible. In effect, the file metadata is sharded over multiple specialized services.
## Data that will be migrated
Currently, oCIS focuses on file sync and share use cases.
### Blob data
In ownCloud 10 the files are laid out on disk in the *data directory* using the following layout:
```
data
├── einstein
│ ├── cache
│ ├── files
│ │ ├── Photos
│ │ │ └── Portugal.jpg
│ │ ├── Projects
│ │ │ └── Notes.md
│ │ └── ownCloud Manual.pdf
│ ├── files_external
│ ├── files_trashbin
│ │ ├── files
│ │ │ ├── Documents.d1564687985
│ │ │ ├── TODO.txt.d1565721976
│ │ │ └── welcome.txt.d1564775872
│ │ └── versions
│ │ │ ├── TODO.txt.v1564605543.d1565721976
│ │ │ └── TODO.txt.v1564775936.d1565721976
│ ├── files_versions
│ │ ├── Projects
│ │ │ ├── Notes.md.v1496912691
│ │ │ └── Notes.md.v1540305560
│ │ └── ownCloud Manual.pdf.v1396628249
│ ├── thumbnails
│ │ └── 123
│ │ │ ├── 2048-1536-max.png
│ │ │ └── 32-32.png // the file id, eg. of /Photos/Portugal.jpg
│ └── uploads
├── einstein
│ ├── files
│ ├── files_trash
│ └── files_versions
│ …
├── marie
…
```
The *data directory* may also contain subfolders for owncloud 10 applications like `avatars`, `gallery`, `files_external` and `cache`.
When an objectstorage is used as the primary storage all file blobs are stored by their file id and a prefix, eg.: `urn:oid:<fileid>`.
The three types of blobs we need to migrate are stored in
- `files` for file blobs, the current file content,
- `files_trashbin` for trashed files (and their versions) and
- `files_versions` for file blobs of older versions.
### Filecache table
In both cases the file metadata, including a full replication of the file tree, is stored in the `oc_filecache` table of an ownCloud 10 database. The primary key of a row is the file id. It is used to attach additional metadata like shares, tags, favorites or arbitrary file properties.
The `filecache` table itself has more metadata:
| Field | Type | Null | Key | Default | Extra | Comment | Migration |
|--------------------|---------------|------|-----|---------|----------------|----------------|----------------|
| `fileid` | bigint(20) | NO | PRI | NULL | auto_increment | | MUST become the oCIS `opaqueid` of a file reference. `ocis` driver stores it in extendet attributes and can use numbers as node ids on disk. for eos see note below table |
| `storage` | int(11) | NO | MUL | 0 | | *the filecache holds metadata for multiple storages* | corresponds to an oCIS *storage space* |
| `path` | varchar(4000) | YES | | NULL | | *the path relative to the storages root* | MUST become the `path` relative to the storage root. `files` prefix needs to be trimmed. |
| `path_hash` | varchar(32) | NO | | | | *mysql once had problems indexing long paths, so we stored a hash for lookup by path. | - |
| `parent` | bigint(20) | NO | MUL | 0 | | *used to implement the hierarchy and listing children of a folder by id. redundant with `path`* | - |
| `name` | varchar(250) | YES | | NULL | | *basename of `path`* | - |
| `mimetype` | int(11) | NO | | 0 | | *joined with the `oc_mimetypes` table. only relevant for objectstorage deployments* | can be determined from blob / file extension |
| `mimepart` | int(11) | NO | | 0 | | *"* | can be determined from blob / file extension |
| `size` | bigint(20) | NO | | 0 | | *same as blob size unless encryption is used* | MAY become size, can be determined from blob |
| `mtime` | bigint(20) | NO | | 0 | | *same as blob mtime* | for files MAY become mtime (can be determined from blob as well), for directories MUST become tmtime |
| `encrypted` | int(11) | NO | | 0 | | *encrypted flag* | oCIS currently does not support encryption |
| `etag` | varchar(40) | YES | | NULL | | *used to propagate changes in a tree* | MUST be migrated (or calculated in the same way) to prevent clients from syncing unnecessarily |
| `unencrypted_size` | bigint(20) | NO | | 0 | | *same as blob size* | oCIS currently does not support encryption |
| `storage_mtime` | bigint(20) | NO | | 0 | | *used to detect external storage changes* | oCIS delegates that to the storage providers and drivers |
| `permissions` | int(11) | YES | | 0 | | *used as the basis for permissions. synced from disk when running a file scan. * | oCIS delegates that to the storage providers and drivers |
| `checksum` | varchar(255) | YES | | NULL | | *same as blob checksum* | SHOULD become the checksum in the storage provider. eos calculates it itself, `ocis` driver stores it in extendetd attributes |
> Note: for EOS a hot migration only works seamlessly if file ids in oc10 are already read from eos. otherwise either a mapping from the oc10 filecache file id to the new eos file id has to be created under the assumption that these id sets do not intersect or files and corresponding shares need to be exported and imported offline to generate a new set of ids. While this will preserve public links, user, group and even federated shares, old internal links may still point to different files because they contain the oc10 fileid
### share table
used to store
- Public links
- Private shares with users and groups
- Federated shares *partly*
- Guest shares
| Field | Type | Null | Key | Default | Extra | Comment | [CS3 API](https://cs3org.github.io/cs3apis/) |
|---------------|--------------|------|-----|---------|----------------|---------|-|
| `id` | int(11) | NO | PRI | NULL | auto_increment | | `ShareId.opaqueid` string |
| `share_type` | smallint(6) | NO | | 0 | | *in CS3 every type is handled by a dedicated API. See below the table* | does NOT map to [`Share.ShareType`](https://cs3org.github.io/cs3apis/#cs3.sharing.ocm.v1beta1.Share.ShareType) *TODO clarify* |
| `share_with` | varchar(255) | YES | MUL | NULL | | | `Share.grantee` [`Grantee`](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.Grantee) |
| `uid_owner` | varchar(64) | NO | | | | | `ShareId.owner` [`UserID`](https://cs3org.github.io/cs3apis/#cs3.identity.user.v1beta1.UserId) |
| `parent` | int(11) | YES | | NULL | | | - |
| `item_type` | varchar(64) | NO | MUL | | | | `Share.resource_id` [`ResourceId`](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId) |
| `item_source` | varchar(255) | YES | MUL | NULL | | | `Share.resource_id` [`ResourceId`](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId) |
| `item_target` | varchar(255) | YES | | NULL | | | `Share.resource_id` [`ResourceId`](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId) |
| `file_source` | bigint(20) | YES | MUL | NULL | | *cannot store uuid style file ids from ocis. when all users have migrated to ocis the share manager needs to be updated / migrated to a version that does.* | `Share.resource_id` [`ResourceId`](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId) |
| `file_target` | varchar(512) | YES | | NULL | | | `Share.resource_id` [`ResourceId`](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId) |
| `permissions` | smallint(6) | NO | | 0 | | | `Share.Permissions` [`SharePermissions`](https://cs3org.github.io/cs3apis/#cs3.sharing.ocm.v1beta1.SharePermissions) |
| `stime` | bigint(20) | NO | | 0 | | | `Share.ctime`, `Share.mtime` |
| `accepted` | smallint(6) | NO | | 0 | | | `ReceivedShare.ShareState` [`ShareState`](https://cs3org.github.io/cs3apis/#cs3.sharing.collaboration.v1beta1.ShareState) |
| `expiration` | datetime | YES | | NULL | | *only used for the Link API and storage provider api, currently cannot be added using the Collaboration or OCM API* | [`Grant`](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.Grant) |
| `token` | varchar(32) | YES | MUL | NULL | | | [`PublicShare.token`](https://cs3org.github.io/cs3apis/#cs3.sharing.link.v1beta1.PublicShare) |
| `mail_send` | smallint(6) | NO | | 0 | | | - |
| `uid_initiator` | varchar(64) | YES | | NULL | | | `ShareId.creator` [`UserID`](https://cs3org.github.io/cs3apis/#cs3.identity.user.v1beta1.UserId) |
| `share_name` | varchar(64) | YES | | NULL | | *only exists for public shares* | [`PublicShare.display_name`](https://cs3org.github.io/cs3apis/#cs3.sharing.link.v1beta1.PublicShare) |
| `attributes` | longtext | YES | | NULL | | *additional share attributes* | *could be implemented using opaque data, but should be added to the CS3 api* |
In the CS3 API
1. public links are handled by the PublicShareProvider using the [Link API](https://cs3org.github.io/cs3apis/#cs3.sharing.link.v1beta1.LinkAPI)
2. internal shares are handled by the UserShareProvider using the [Collaboration API](https://cs3org.github.io/cs3apis/#cs3.sharing.collaboration.v1beta1.CollaborationAPI). This covers user and group shares.
3. federated shares are handled by the OcmShareProvider using the [OCM Share Provider AP](https://cs3org.github.io/cs3apis/#cs3.sharing.ocm.v1beta1.OcmAPI) aka. Open Cloud Mesh.
### share_external
Used to store additional metadata for federated shares.
| Field | Type | Null | Key | Default | Extra | Comment |
|-----------------|---------------|------|-----|---------|----------------|---------|
| `id` | bigint(20) | NO | PRI | NULL | auto_increment | |
| `remote` | varchar(512) | NO | | NULL | | Url of the remote owncloud instance |
| `share_token` | varchar(64) | NO | | NULL | | Public share token |
| `password` | varchar(64) | YES | | NULL | | Optional password for the public share |
| `name` | varchar(64) | NO | | NULL | | Original name on the remote server |
| `owner` | varchar(64) | NO | | NULL | | User that owns the public share on the remote server |
| `user` | varchar(64) | NO | MUL | NULL | | Local user which added the external share |
| `mountpoint` | varchar(4000) | NO | | NULL | | Full path where the share is mounted |
| `mountpoint_hash` | varchar(32) | NO | | NULL | | md5 hash of the mountpoint |
| `remote_id` | varchar(255) | NO | | -1 | | |
| `accepted` | int(11) | NO | | 0 | | |
*TODO document how the reva OCM service currently persists the data*
### trusted_servers
used to determine if federated shares can automatically be accepted
| Field | Type | Null | Key | Default | Extra | Comment |
|---------------|--------------|------|-----|---------|----------------|---------|
| `id` | int(11) | NO | PRI | NULL | auto_increment | |
| `url` | varchar(512) | NO | | NULL | | Url of trusted server |
| `url_hash` | varchar(255) | NO | UNI | | | sha1 hash of the url without the protocol |
| `token` | varchar(128) | YES | | NULL | | token used to exchange the shared secret |
| `shared_secret` | varchar(256) | YES | | NULL | | shared secret used to authenticate |
| `status` | int(11) | NO | | 2 | | current status of the connection |
| `sync_token` | varchar(512) | YES | | NULL | | cardDav sync token |
*TODO clarify how OCM handles this andt where we store / configure this. It seems related to trusted IdPs*
### user data
Users are migrated in two steps:
1. They should all be authenticated using openid connect, which already moves them to a common identity management system.
2. To search share recipients, both, ownCloud 10 and oCIS need acces to the same user directory using eg. LDAP.
*TODO: add state to CS3 API, so we can 'disable' users*
*TODO: how do we map (sub) admins? -> map to roles & permissions*
accounts:
| Field | Type | Null | Key | Default | Extra | Comment |
|---------------|---------------------|------|-----|---------|----------------|---------|
| `id` | bigint(20) unsigned | NO | PRI | NULL | auto_increment | |
| `email` | varchar(255) | YES | MUL | NULL | | |
| `user_id` | varchar(255) | NO | UNI | NULL | | |
| `lower_user_id` | varchar(255) | NO | UNI | NULL | | |
| `display_name` | varchar(255) | YES | MUL | NULL | | |
| `quota` | varchar(32) | YES | | NULL | | |
| `last_login` | int(11) | NO | | 0 | | |
| `backend` | varchar(64) | NO | | NULL | | |
| `home` | varchar(1024) | NO | | NULL | | |
| `state` | smallint(6) | NO | | 0 | | |
users:
| Field | Type | Null | Key | Default | Extra | Comment |
|-------------|--------------|------|-----|---------|-------|---------|
| `uid` | varchar(64) | NO | PRI | | |
| `password` | varchar(255) | NO | | | |
| `displayname` | varchar(64) | YES | | NULL | |
groups:
The groups table really only contains the group name.
| Field | Type | Null | Key | Default | Extra |
|-------|-------------|------|-----|---------|-------|
| `gid` | varchar(64) | NO | PRI | | |
### LDAP
TODO clarify if metadata from ldap & user_shibboleth needs to be migrated
- the `dn` -> *owncloud internal username* mapping that currently lives in the `oc_ldap_user_mapping` table needs to move into a dedicated ownclouduuid attribute in the LDAP server. The idp should send it as a claim so the proxy does not have to look up the user using LDAP again. The username cannot be changed in ownCloud 10 and the oCIS provisioning API will not allow changing it as well. When we introduce the graph api we may allow changing usernames when all clients have moved to that api.
The problem is that the username in owncloud 10 and in oCIS also need to be the same, which might not be the case when the ldap mapping used a different column. In that case we should add another owncloudusername attribute to the ldap server ...
### activities
*dedicated service, not yet implemented, requires decisions about an event system -- jfd*
| Field | Type | Null | Key | Default | Extra | Comment |
|---------------|---------------|------|-----|---------|----------------|---------|
| `activity_id` | bigint(20) | NO | PRI | NULL | auto_increment |
| `timestamp` | int(11) | NO | MUL | 0 | |
| `priority` | int(11) | NO | | 0 | |
| `type` | varchar(255) | YES | | NULL | |
| `user` | varchar(64) | YES | | NULL | |
| `affecteduser` | varchar(64) | NO | MUL | NULL | |
| `app` | varchar(255) | NO | | NULL | |
| `subject` | varchar(255) | NO | | NULL | |
| `subjectparams` | longtext | NO | | NULL | |
| `message` | varchar(255) | YES | | NULL | |
| `messageparams` | longtext | YES | | NULL | |
| `file` | varchar(4000) | YES | | NULL | |
| `link` | varchar(4000) | YES | | NULL | |
| `object_type` | varchar(255) | YES | MUL | NULL | |
| `object_id` | bigint(20) | NO | | 0 | |
## Links
The [data_exporter](https://github.com/owncloud/data_exporter) has logic that allows exporting and importing users, including shares. The [model classes](https://github.com/owncloud/data_exporter/tree/master/lib/Model) contain the exact mapping.