From ef54417b6279fdb0bfcc839c3e65b75acc449369 Mon Sep 17 00:00:00 2001 From: mmattel Date: Tue, 21 Feb 2023 12:08:08 +0100 Subject: [PATCH 1/4] Add a basic storage-users readme only containing cli commands --- services/storage-users/README.md | 80 +++++++++++++++++++++ services/storage-users/pkg/config/config.go | 4 +- 2 files changed, 82 insertions(+), 2 deletions(-) create mode 100644 services/storage-users/README.md diff --git a/services/storage-users/README.md b/services/storage-users/README.md new file mode 100644 index 000000000..a234eba04 --- /dev/null +++ b/services/storage-users/README.md @@ -0,0 +1,80 @@ +# Storage-Users Service + +Purpose and description to be added + +## CLI Commands + +### Manage Unfinished Uploads + + + +When using Infinite Scale as user storage, a directory named `storage/users/uploads` can be found in the Infinite Scale data folder. This is an intermediate directory based on [TUS](https://tus.io) which is an open protocol for resumable uploads. Each upload consists of a _blob_ and a _blob.info_ file. Note that the term _blob_ is just a placeholder. + +* **If an upload succeeds**, the _blob_ file will be moved to the target and the _blob.info_ file will be deleted. + +* **In case of incomplete uploads**, the _blob_ and _blob.info_ files will continue to recieve data until either the upload succeeds in time or the upload expires based on the `STORAGE_USERS_UPLOAD_EXPIRATION` variable, see the table below for details. + +* **In case of expired uploads**, the _blob_ and _blob.info_ files will _not_ be removed automatically. Thus a lot of data can pile up over time wasting storage space. + +Example cases for expired uploads + +* When a user uploads a big file but the file exceeds the user-quota, the upload can't be moved to the target after it has finished. The file stays at the upload location until it is manually cleared. +* If the bandwith is limited and the file to transfer can't be transferred completely before the upload expiration time is reached, the file expires and can't be processed. + +There are two commands available to manage unfinished uploads + +```bash +ocis storage-users uploads +``` + +```plaintext +COMMANDS: + list Print a list of all incomplete uploads + clean Clean up leftovers from expired uploads +``` + +#### Command Examples + +Command to identify incomplete uploads + +```bash +ocis storage-users uploads list +``` + +```plaintext +Incomplete uploads: + - 455bd640-cd08-46e8-a5a0-9304908bd40a (file_example_PPT_1MB.ppt, Size: 1028608, Expires: 2022-08-17T12:35:34+02:00) +``` + +Command to clear expired uploads +```bash +ocis storage-users uploads clean +``` + +```plaintext +Cleaned uploads: +- 455bd640-cd08-46e8-a5a0-9304908bd40a (Filename: file_example_PPT_1MB.ppt, Size: 1028608, Expires: 2022-08-17T12:35:34+02:00) +``` + +### Purge Expired Space Trash-Bins Items + + + +This command is about to purge old trash-bin items of `project` spaces (spaces that have been created manually) and `personal` spaces . + +```bash +ocis storage-users trash-bin +``` + +```plaintext +COMMANDS: + purge-expired Purge all expired items from the trashbin +``` + +The configuration for the `purge-expired` command is done by using the following environment variables. + +* `STORAGE_USERS_PURGE_TRASH_BIN_USER_ID` is used to obtain space trash-bin information and takes the system admin user as the default which is the `OCIS_ADMIN_USER_ID` but can be set individually. It should be noted, that the `OCIS_ADMIN_USER_ID` is only assigned automatically when using the single binary deployment and must be manually assigned in all other deployments. The command only considers spaces to which the assigned user has access and delete permission. + +* `STORAGE_USERS_PURGE_TRASH_BIN_PERSONAL_DELETE_BEFORE` has a default value of `30 days`, which means the command will delete all files older than `30 days`. The value is human-readable, valid values are `24h`, `60m`, `60s` etc. `0` is equivalent to disable and prevents the deletion of `personal space` trash-bin files. + +* `STORAGE_USERS_PURGE_TRASH_BIN_PROJECT_DELETE_BEFORE` has a default value of `30 days`, which means the command will delete all files older than `30 days`. The value is human-readable, valid values are `24h`, `60m`, `60s` etc. `0` is equivalent to disable and prevents the deletion of `project space` trash-bin files. diff --git a/services/storage-users/pkg/config/config.go b/services/storage-users/pkg/config/config.go index 30e60057a..4c28aa741 100644 --- a/services/storage-users/pkg/config/config.go +++ b/services/storage-users/pkg/config/config.go @@ -229,6 +229,6 @@ type Tasks struct { // PurgeTrashBin contains all necessary configurations to clean up the respective trash cans type PurgeTrashBin struct { UserID string `yaml:"user_id" env:"OCIS_ADMIN_USER_ID;STORAGE_USERS_PURGE_TRASH_BIN_USER_ID" desc:"ID of the user who collects all necessary information for deletion."` - PersonalDeleteBefore time.Duration `yaml:"personal_delete_before" env:"STORAGE_USERS_PURGE_TRASH_BIN_PERSONAL_DELETE_BEFORE" desc:"Specifies the period of time in which items that have been in the personal trash-bin for longer than this value should be deleted. A value of 0 means no automatic deletion"` - ProjectDeleteBefore time.Duration `yaml:"project_delete_before" env:"STORAGE_USERS_PURGE_TRASH_BIN_PROJECT_DELETE_BEFORE" desc:"Specifies the period of time in which items that have been in the project trash-bin for longer than this value should be deleted. A value of 0 means no automatic deletion"` + PersonalDeleteBefore time.Duration `yaml:"personal_delete_before" env:"STORAGE_USERS_PURGE_TRASH_BIN_PERSONAL_DELETE_BEFORE" desc:"Specifies the period of time in which items that have been in the personal trash-bin for longer than this value should be deleted. A value of 0 means no automatic deletion. The value is human-readable, valid values are `24h`, `60m`, `60s` etc."` + ProjectDeleteBefore time.Duration `yaml:"project_delete_before" env:"STORAGE_USERS_PURGE_TRASH_BIN_PROJECT_DELETE_BEFORE" desc:"Specifies the period of time in which items that have been in the project trash-bin for longer than this value should be deleted. A value of 0 means no automatic deletion. The value is human-readable, valid values are `24h`, `60m`, `60s` etc."` } From 24e77820ab5c7e2bb635d849cd9475e9c73da7b9 Mon Sep 17 00:00:00 2001 From: mmattel Date: Tue, 21 Feb 2023 16:55:43 +0100 Subject: [PATCH 2/4] fix quotes --- services/storage-users/pkg/config/config.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/services/storage-users/pkg/config/config.go b/services/storage-users/pkg/config/config.go index 4c28aa741..27038b346 100644 --- a/services/storage-users/pkg/config/config.go +++ b/services/storage-users/pkg/config/config.go @@ -229,6 +229,6 @@ type Tasks struct { // PurgeTrashBin contains all necessary configurations to clean up the respective trash cans type PurgeTrashBin struct { UserID string `yaml:"user_id" env:"OCIS_ADMIN_USER_ID;STORAGE_USERS_PURGE_TRASH_BIN_USER_ID" desc:"ID of the user who collects all necessary information for deletion."` - PersonalDeleteBefore time.Duration `yaml:"personal_delete_before" env:"STORAGE_USERS_PURGE_TRASH_BIN_PERSONAL_DELETE_BEFORE" desc:"Specifies the period of time in which items that have been in the personal trash-bin for longer than this value should be deleted. A value of 0 means no automatic deletion. The value is human-readable, valid values are `24h`, `60m`, `60s` etc."` - ProjectDeleteBefore time.Duration `yaml:"project_delete_before" env:"STORAGE_USERS_PURGE_TRASH_BIN_PROJECT_DELETE_BEFORE" desc:"Specifies the period of time in which items that have been in the project trash-bin for longer than this value should be deleted. A value of 0 means no automatic deletion. The value is human-readable, valid values are `24h`, `60m`, `60s` etc."` + PersonalDeleteBefore time.Duration `yaml:"personal_delete_before" env:"STORAGE_USERS_PURGE_TRASH_BIN_PERSONAL_DELETE_BEFORE" desc:"Specifies the period of time in which items that have been in the personal trash-bin for longer than this value should be deleted. A value of 0 means no automatic deletion. The value is human-readable, valid values are '24h', '60m', '60s' etc."` + ProjectDeleteBefore time.Duration `yaml:"project_delete_before" env:"STORAGE_USERS_PURGE_TRASH_BIN_PROJECT_DELETE_BEFORE" desc:"Specifies the period of time in which items that have been in the project trash-bin for longer than this value should be deleted. A value of 0 means no automatic deletion. The value is human-readable, valid values are '24h', '60m', '60s' etc."` } From 68835fa697c9f82ac2d00001e89581885f2edf57 Mon Sep 17 00:00:00 2001 From: kobergj Date: Wed, 22 Feb 2023 10:22:04 +0100 Subject: [PATCH 3/4] Update services/storage-users/README.md Co-authored-by: Martin --- services/storage-users/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/services/storage-users/README.md b/services/storage-users/README.md index a234eba04..d6b4e41bc 100644 --- a/services/storage-users/README.md +++ b/services/storage-users/README.md @@ -16,6 +16,8 @@ When using Infinite Scale as user storage, a directory named `storage/users/uplo * **In case of expired uploads**, the _blob_ and _blob.info_ files will _not_ be removed automatically. Thus a lot of data can pile up over time wasting storage space. +* **In the rare case of a failure**, after the upload succeeded but the file was not moved to its target location, which can happen when postprocessing fails, the situation is the same as with expired uploads. + Example cases for expired uploads * When a user uploads a big file but the file exceeds the user-quota, the upload can't be moved to the target after it has finished. The file stays at the upload location until it is manually cleared. From 21a0c0a90bff5e6c99139e32f7ba6228c86f0698 Mon Sep 17 00:00:00 2001 From: Martin Date: Wed, 22 Feb 2023 10:26:00 +0100 Subject: [PATCH 4/4] Apply suggestions from code review Co-authored-by: Jan --- services/storage-users/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/services/storage-users/README.md b/services/storage-users/README.md index d6b4e41bc..e020087a4 100644 --- a/services/storage-users/README.md +++ b/services/storage-users/README.md @@ -12,7 +12,7 @@ When using Infinite Scale as user storage, a directory named `storage/users/uplo * **If an upload succeeds**, the _blob_ file will be moved to the target and the _blob.info_ file will be deleted. -* **In case of incomplete uploads**, the _blob_ and _blob.info_ files will continue to recieve data until either the upload succeeds in time or the upload expires based on the `STORAGE_USERS_UPLOAD_EXPIRATION` variable, see the table below for details. +* **In case of incomplete uploads**, the _blob_ and _blob.info_ files will continue to receive data until either the upload succeeds in time or the upload expires based on the `STORAGE_USERS_UPLOAD_EXPIRATION` variable, see the table below for details. * **In case of expired uploads**, the _blob_ and _blob.info_ files will _not_ be removed automatically. Thus a lot of data can pile up over time wasting storage space. @@ -21,7 +21,7 @@ When using Infinite Scale as user storage, a directory named `storage/users/uplo Example cases for expired uploads * When a user uploads a big file but the file exceeds the user-quota, the upload can't be moved to the target after it has finished. The file stays at the upload location until it is manually cleared. -* If the bandwith is limited and the file to transfer can't be transferred completely before the upload expiration time is reached, the file expires and can't be processed. +* If the bandwidth is limited and the file to transfer can't be transferred completely before the upload expiration time is reached, the file expires and can't be processed. There are two commands available to manage unfinished uploads @@ -62,7 +62,7 @@ Cleaned uploads: -This command is about to purge old trash-bin items of `project` spaces (spaces that have been created manually) and `personal` spaces . +This command is about purging old trash-bin items of `project` spaces (spaces that have been created manually) and `personal` spaces. ```bash ocis storage-users trash-bin