work on docs

This commit is contained in:
Willy Kloucek
2020-11-24 08:46:04 +01:00
parent d856d5470e
commit 34845f9414
65 changed files with 326 additions and 2298 deletions
+1 -3
View File
@@ -1,10 +1,8 @@
---
title: "Development"
date: 2020-10-01T20:35:00+01:00
weight: -12
weight: 10
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/development
geekdocFilePath: _index.md
---
{{< toc >}}
+20
View File
@@ -0,0 +1,20 @@
---
title: "Documentation"
date: 2020-07-27T08:39:38+00:00
weight: 99
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/development
geekdocFilePath: build-docs.md
---
{{< toc >}}
## Build the documentation
Just run `make -C docs docs-serve` from within the root level of the oCIS git repository.
## Add changes to the documentation
Please keep this documentation in sync with the oCIS source code.
Changes on the documentation are automatically applied to [this site](https://owncloud.github.io) when merged to the master branch.
+48
View File
@@ -0,0 +1,48 @@
---
title: "Build"
date: 2020-02-27T20:35:00+01:00
weight: 30
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/development
geekdocFilePath: build.md
---
{{< toc >}}
## Build requirements
see [Development - Getting Started]({{< relref "getting-started.md/#requirements">}})
## Get the sources
{{< highlight txt >}}
git clone https://github.com/owncloud/ocis.git
cd ocis
{{< / highlight >}}
## Build the oCIS binary
The oCIS binary source is in the `ocis` folder inside the oCIS repository. In this folder you can build the oCIS binary:
{{< highlight txt >}}
make generate
make build
{{< / highlight >}}
After building you have the binary within the `bin/` folder. Try to run it: `./bin/ocis -h`
## Build a local oCIS docker image
If you are developing and want to run your local changes in a docker or docker-compose setup, you have to build an image locally.
Therefore run following commands in the root of the oCIS repository:
{{< highlight txt >}}
docker build -t owncloud/ocis:dev .
{{< / highlight >}}
Then you can test as usual via
{{< highlight txt >}}
docker run --rm -ti owncloud/ocis:dev
{{< / highlight >}}
-40
View File
@@ -1,40 +0,0 @@
---
title: "Build the documentation"
date: 2020-07-27T08:39:38+00:00
weight: 99
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/development
geekdocFilePath: building-docs.md
---
## Buildling the documentation
Following steps can be applied for every oCIS extension repository.
### Setting up
- Install [hugo](https://gohugo.io/getting-started/installing/)
- Run `make docs`
### Viewing the documentation
To view the rendered docs in the browser run:
```bash
cd hugo
hugo -D server
```
Then open "http://localhost:1313/"
When making changes to the docs, run
```bash
rsync -ax --delete ../docs/ content/
```
in the hugo folder and the server will pick up the changes and reload the page automatically.
### Deploying the documentation
The documentation is automatically deployed from the master branch to https://owncloud.github.io
-46
View File
@@ -1,46 +0,0 @@
---
title: "Build ocis"
date: 2020-02-27T20:35:00+01:00
weight: 30
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/development
geekdocFilePath: building.md
---
## Build requirements
All required tools besides `go` and `make` are bundled or getting automatically installed within the `GOPATH`. All commands to build this project are part of our `Makefile`.
The installation of Go is out of the scope of this document, please follow the official documentation for [Go](https://golang.org/doc/install), to build this project you have to install Go >= v1.13.
## Get the sources
{{< highlight txt >}}
git clone https://github.com/owncloud/ocis.git
cd ocis
{{< / highlight >}}
## Build the oCIS binary
The oCIS binary source is in the ocis/ocis folder. In this folder you can build the ocis binary:
{{< highlight txt >}}
make generate
make build
{{< / highlight >}}
Finally, you should have the binary within the `bin/` folder now, give it a try with `./bin/ocis -h` to see all available options.
## Build a local ocis docker image
If you are developing on a local branch based on docker / docker-compose setup, here is how to build a new ocis image. In the root folder:
{{< highlight txt >}}
docker build -t owncloud/ocis:dev .
{{< / highlight >}}
Then you can test as usual via
{{< highlight txt >}}
docker run --rm -ti owncloud/ocis:dev
{{< / highlight >}}
@@ -0,0 +1,56 @@
---
title: "Continuous Integration"
date: 2020-10-01T20:35:00+01:00
weight: 100
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/development
geekdocFilePath: continuous-integration.md
---
{{< toc >}}
oCIS uses [DRONE](https://www.drone.io/) as CI system. You can find the pipeline logs [here](https://drone.owncloud.com/owncloud/ocis) or in your PR.
## Concepts
The pipeline is defined in [Starlark](https://github.com/bazelbuild/starlark) and transformed to YAML upon pipeline run. This enables us to do a highly dynamic and non repeating pipeline configuration.
Upon running the pipeline, your branch gets merged to the master branch. This ensures that we always test your changeset if as it was applied to the master of oCIS. Please note that this does not apply to the pipeline definition (`.drone.star`).
## Things done in CI
- static code analysis
- linting
- running UI tests
- running ownCloud 10 test suite against oCIS
- build and release docker images
- build and release binaries
- build and release documentation
## Flags in commit message and PR title
You may add flags to your commit message or PR title in order to speed up pipeline runs and take load from the CI runners.
- `[CI SKIP]`: no ci is run on the commit or PR
- `[docs-only]`: please add this flag, if you only changed documentation. This will only trigger documentation related CI steps.
### Knowledge base
- My pipeline fails because some CI related files or commands are missing.
Please make sure to rebase your branch onto the lastest master of oCIS. It could be that the pipeline definition (`.drone.star`) was changed on the master branch. This is is the only file, that will not be auto merged to master upon pipeline run. So things could be out of sync.
- How can I see the YAML drone pipeline definition?
In order to see the Yaml pipeline definition you can use the drone-cli to convert the Starlark file.
```
drone starlark
```
{{< hint info >}}
If you experience a `"build" struct has no .title attribute` error you need a patched drone-cli binary.
You need to build it yourself from this [source code](https://github.com/micbar/drone-cli/tree/add-more-flags).
(There is also an open [PR](https://github.com/drone/drone-cli/pull/175) for that on drone-cli)
{{< /hint >}}
+15 -12
View File
@@ -7,6 +7,8 @@ geekdocEditPath: edit/master/docs/ocis/development
geekdocFilePath: debugging.md
---
{{< toc >}}
## Debugging
As a single binary for easy deployment running `ocis server` just forks itself to start all the services, which makes debugging those processes a little harder.
@@ -52,22 +54,23 @@ Then you can set a breakpoint in the service you need and attach to the process
```json
{
"version": "0.2.0",
"configurations": [
{
"name": "ocis attach",
"type": "go",
"request": "attach",
"mode": "local",
"processId": 12897,
}
]
"version": "0.2.0",
"configurations": [
{
"name": "ocis attach",
"type": "go",
"request": "attach",
"mode": "local",
"processId": 12897
}
]
}
```
#### Start all services independently to replace one of them with a debug process
1. You can use this `./ocis.sh` script to start all services independently, so they don't get restrarted by the runtime when you kill them:
```bash
#/bin/sh
LOG_LEVEL="debug"
@@ -128,7 +131,6 @@ bin/ocis --log-level=$LOG_LEVEL proxy &
3. Kill the service you want to start in debug mode:
```console
# kill 17628
```
@@ -203,15 +205,16 @@ Debug wherever the call trace leads you to ... good luck!
You can either run and manage the services independently, or you can update the `go.mod` file and replace dependencies with your local version.
To debug the reva frontend we need to add two replacements:
```
// use the local ocis-reva repo
replace github.com/owncloud/ocis-reva => ../ocis-reva
// also use the local reva repo
replace github.com/cs3org/reva => ../reva
```
{{< hint info >}}
The username and password only work when basic auth is available. Otherwise you have to obtain a bearer token, eg. by grabbing it from the browser.
{{< /hint >}}
Rebuild ocis to make sure the dependency is used. It should be sufficient to just restart the service you want to debug.
+233
View File
@@ -0,0 +1,233 @@
---
title: "Extensions"
date: 2020-02-27T20:35:00+01:00
weight: 40
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/development
geekdocFilePath: extensions.md
---
{{< toc >}}
## How to build and run ocis-simple
ocis uses build tags to build different flavors of the binary. In order to work on a new extension we are going to reduce the scope a little and use the `simple` tag. Let us begin by creating a dedicated folder:
```console
mkdir ocis-extension-workshop && ocis-extension-workshop
```
Following https://github.com/owncloud/ocis
```console
git clone https://github.com/owncloud/ocis.git
cd ocis
TAGS=simple make generate build
```
*Q: Can you specify which version of phoenix to use?*
*A: No, the phoenix that is used is compiled into the [assets of ocis-phoenix](https://github.com/owncloud/ocis-phoenix/blob/master/pkg/assets/embed.go) which is currently not automatically updated. We'll see how to use a custom phoenix later.*
`bin/ocis server`
Open the browser at http://localhost:9100
1. You land on the login screen. click login
2. You are redirected to an idp at http://localhost:9140/oauth2/auth with a login mask. Use `einstein:relativity`to login (one of the three demo users)
3. You are redirected to http://localhost:9100/#/hello the ocis-hello app
4. Replace `World` with something else and submit. You should see `Hello %something else%`
*Q: One of the required ports is already in use. Ocis seems to be trying to restart the service over and over. What gives?*
*A: Using the ocis binary to start the server will case ocis to keep track of the different services and restart them in case they crash.*
## Hacking ocis-hello
go back to the ocis-extension-workshop folder
```console
cd ..
```
Following https://github.com/owncloud/ocis-hello
```
git clone https://github.com/owncloud/ocis-hello.git
cd ocis-hello
yarn install
# this actually creates the assets
yarn build
# this will compile the assets into the binary
make generate build
```
Two options:
1. run only the necessery services from ocis and ocis-hello independently
2. compile ocis with the updated ocis-hello
### Option 1:
get a list of ocis services:
```console
ps ax | grep ocis
```
Try to kill `ocis hello`
Remember: for now, killing a service will cause ocis to restart it. This is subject to change.
In order to be able to manage the processes ourselves we need to start them independently:
`bin/ocis server` starts the same services as:
```
bin/ocis micro &
bin/ocis phoenix &
bin/ocis hello &
bin/ocis reva &
```
Now we can kill the `ocis hello` and use our custom built ocis-hello binary:
```console
cd ../ocis-hello
bin/ocis-hello server
```
## Hacking phoenix (and ocis-phoenix)
Following https://github.com/owncloud/phoenix we are going to build the current phoenix
```
git clone https://github.com/owncloud/phoenix.git
cd phoenix
yarn install
yarn dist
```
We can tell ocis to use the compiled assets:
Kill `ocis phoenix`, then use the compiled assets when starting phoenix.
```console
cd ../ocis
PHOENIX_ASSET_PATH="`pwd`/../phoenix/dist" bin/ocis phoenix
```
## The ownCloud design system
The [ownCloud design system](https://owncloud.design/) contains a set of ownCloud vue components for phoenix or your own ocis extensions. Please use it for a consistent look and feel.
## External phoenix apps
This is what hello is: copy and extend!
1. Phoenix is configured using the config.json which is served by the phoenix service (either `bin/ocis phoenix` or `bin/ocis-phoenix server`)
2. point ocis phoenix to the web config which you extended with an external app:
`PHOENIX_WEB_CONFIG="`pwd`/../phoenix/config.json" PHOENIX_ASSET_PATH="`pwd`/../phoenix/dist" bin/ocis phoenix`
```json
{
"server": "http://localhost:9140",
"theme": "owncloud",
"version": "0.1.0",
"openIdConnect": {
"metadata_url": "http://localhost:9140/.well-known/openid-configuration",
"authority": "http://localhost:9140",
"client_id": "phoenix",
"response_type": "code",
"scope": "openid profile email"
},
"apps": [],
"external_apps": [
{
"id": "hello",
"path": "http://localhost:9105/hello.js",
"config": {
"url": "http://localhost:9105"
}
},
{
"id": "myapp",
"path": "http://localhost:6789/superapp.js",
"config": {
"backend": "http://someserver:1234",
"myconfig": "is awesome"
}
}
]
}
```
## Phoenix extension points
{{< hint info >}}
For an up to date list check out [the phoenix documentation](https://github.com/owncloud/phoenix/issues/2423).
{{< /hint >}}
Several ones available:
### Phoenix core
- App switcher (defined in config.json)
- App container (loads UI of your extension)
### Files app
- File action
- Create new file action
- Sidebar
- Quick access for sidebar inside of file actions (in the file row)
Example of a file action in the `app.js`:
```js
const appInfo = {
name: 'MarkdownEditor',
id: 'markdown-editor',
icon: 'text',
isFileEditor: true,
extensions: [{
extension: 'txt',
newFileMenu: {
menuTitle ($gettext) {
return $gettext('Create new plain text file…')
}
}
},
{
extension: 'md',
newFileMenu: {
menuTitle ($gettext) {
return $gettext('Create new mark-down file…')
}
}
}]
}
```
For the side bar have a look at the files app, `defaults.js` & `fileSideBars`
## API driven development
Until now we only had a look at the ui and how the extensions are managed on the cli. But how do apps actually talk to the server?
Short answer: any way you like
Long answer: micro and ocis-hello follow a protocol driven development:
- specify the API using protobuf
- generate client and server code
- evolve based on the protocol
- CS3 api uses protobuf as well and uses GRPC
- ocis uses go-micro, which provides http and grpc gateways
- the gateways and protocols are optional
- owncloud and kopano are looking into a [MS graph](https://developer.microsoft.com/de-de/graph) like api to handle phoenix requests.
- they might be about user, contacrs, calendars ... which is covered by the graph api
- we want to integrate with eg. kopano and provide a commen api (file sync and share is covered as well)
- as an example for protobuf take a look at [ocis-hello](https://github.com/owncloud/ocis-hello/tree/master/pkg/proto/v0)
+33 -56
View File
@@ -1,5 +1,5 @@
---
title: "Getting Started with Development"
title: "Getting Started"
date: 2020-07-07T20:35:00+01:00
weight: 15
geekdocRepo: https://github.com/owncloud/ocis
@@ -9,72 +9,49 @@ geekdocFilePath: getting-started.md
{{< toc >}}
## Docker dev environment
## Requirements
### Option 1: Plain docker
We want contribution to oCIS and the creation of extensions to be as easy as possible.
So we are trying to reflect this the used tooling. It should be kept simple and quick to be set up.
To build and run your local ocis code with default storage driver
Besides of standard development tools like git and a text editor, you need following software for development:
```
docker run --rm -ti --name ocis -v $PWD:/ocis -p 9200:9200 owncloud/eos-ocis-dev
```
- Go >= v1.13 ([install instructions](https://golang.org/doc/install))
- Yarn ([install instructions](https://classic.yarnpkg.com/en/docs/install))
- docker ([install instructions](https://docs.docker.com/get-docker/))
- docker-compose ([install instructions](https://docs.docker.com/compose/install/))
The eos-ocis-dev container will build and run oCIS using the ownCloud storage driver and store files in the container at `/var/tmp/reva/data/<username>/files`
If you find tools needed besides the mentioned above, please feel free to open an issue or open a PR.
To check the uploaded files start digging with: `docker exec -it ocis ls -l /var/tmp/reva/`
## Repository structure
{{< hint info >}}
On MacOS do not mount a local folder to the `/var/tmp/reva/` path. The fuse driver used by docker [does not support extended attributes](https://docs.docker.com/v18.09/docker-for-mac/osxfs/). See [#182](https://github.com/owncloud/ocis/issues/182) for more details.
{{< /hint >}}
This repository follows the [golang-standard project-layout](https://github.com/golang-standards/project-layout).
oCIS consists of multiple micro services, also called extensions. We started by having standalone repositories for each of them but quickly noticed, that this adds a time consuming overhead for developers. So we ended up with a monorepo housing all the extensions in one repository.
### Option 2: Docker compose
Each of the extensions live in a subfolder (eg. `accounts` or `settings`) in this repository, technically creating independant Go modules.
With the `docker-compose.yml` file in ocis repo you can also start ocis via compose:
The `ocis` folder does also contain a Go module but is no extension at all. Instead this module is used to import all extensions and furthermore implement commands to start the extensions. With the resulting oCIS binary you can start single extensions or even all extensions at the same time.
```
docker-compose up -d ocis
```
The `docs` folder contains the source for the [oCIS documentation](https://owncloud.github.io/ocis/).
{{< hint info >}}
We are only starting the `ocis` container here.
{{< /hint >}}
The `deployments` folder contains documented deployment configurations and templates.
## Verification
The `scripts` folder contains scripts to perform various build, install, analysis, etc operations.
Check the services are running
## Starting points
```
$ docker-compose exec ocis ./bin/ocis list
+--------------------------+-----+
| EXTENSION | PID |
+--------------------------+-----+
| accounts | 172 |
| api | 204 |
| glauth | 187 |
| graph | 41 |
| graph-explorer | 55 |
| konnectd | 196 |
| ocs | 59 |
| phoenix | 29 |
| proxy | 22 |
| registry | 226 |
| reva-auth-basic | 96 |
| reva-auth-bearer | 104 |
| reva-frontend | 485 |
| reva-gateway | 78 |
| reva-sharing | 286 |
| reva-storage-eos | 129 |
| reva-storage-eos-data | 134 |
| reva-storage-home | 442 |
| reva-storage-home-data | 464 |
| reva-storage-oc | 149 |
| reva-storage-oc-data | 155 |
| reva-storage-public-link | 168 |
| reva-users | 420 |
| settings | 23 |
| thumbnails | 201 |
| web | 218 |
| webdav | 63 |
+--------------------------+-----+
```
Depending on what you want do develop there are different starting points. These will be described below.
### Developing oCIS
If you want to contribute to oCIS:
- see [contribution guidelines](https://github.com/owncloud/ocis#contributing)
- make sure the tooling is set up by [building oCIS]({{< relref "build.md" >}}) and [building the docs]({{< relref "build-docs.md" >}})
- create or pick an [open issue](https://github.com/owncloud/ocis/issues) to develop on
- open a PR and get things done
### Developing extensions
If you want to develop an extension, start here: [Extensions]({{< relref "extensions.md">}})
+6 -1
View File
@@ -7,6 +7,7 @@ geekdocEditPath: edit/master/docs/ocis/development
geekdocFilePath: testing.md
---
{{< toc >}}
## Acceptance tests
@@ -15,6 +16,7 @@ We are using the ownCloud 10 acceptance test suite against oCIS. To set this up
### Getting the tests
All you need to do to get the acceptance tests is check out the core repo:
```
git clone https://github.com/owncloud/core.git
```
@@ -28,6 +30,7 @@ File versions need a redis server. Start one with docker by using:
### Run ocis
To start ocis:
```
PROXY_ENABLE_BASIC_AUTH=true bin/ocis server
```
@@ -35,6 +38,7 @@ PROXY_ENABLE_BASIC_AUTH=true bin/ocis server
`PROXY_ENABLE_BASIC_AUTH` will allow the acceptance tests to make requests against the provisioning api (and other endpoints) using basic auth.
### Run the acceptance tests
First we will need to clone the testing app in owncloud which contains the skeleton files required for running the tests.
In the ownCloud 10 core clone the testing app with the following command:
@@ -42,7 +46,7 @@ In the ownCloud 10 core clone the testing app with the following command:
git clone https://github.com/owncloud/testing apps/testing
```
Then run the api acceptance tests with the following command from the root of the OCIS repo:
Then run the api acceptance tests with the following command from the root of the oCIS repository:
```
make test-acceptance-api \
TEST_SERVER_URL=https://localhost:9200 \
@@ -87,6 +91,7 @@ If you want to work on a specific issue
2. locally run each of the tests marked with that issue in the expected failures file.
E.g.:
```
make test-acceptance-api \
TEST_SERVER_URL=https://localhost:9200 \
+32 -30
View File
@@ -7,41 +7,43 @@ geekdocEditPath: edit/master/docs/ocis/development
geekdocFilePath: tracing.md
---
{{< toc >}}
By default, we use [Jaeger](https://www.jaegertracing.io) for request tracing within oCIS. You can follow these steps
to get started:
1. Start Jaeger by using the all-in-one docker image:
```console
docker run -d --name jaeger \
-e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
-p 5775:5775/udp \
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 5778:5778 \
-p 16686:16686 \
-p 14268:14268 \
-p 14250:14250 \
-p 9411:9411 \
jaegertracing/all-in-one:1.17
```
```console
docker run -d --name jaeger \
-e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
-p 5775:5775/udp \
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 5778:5778 \
-p 16686:16686 \
-p 14268:14268 \
-p 14250:14250 \
-p 9411:9411 \
jaegertracing/all-in-one:1.17
```
2. Every single oCIS service has its own environment variables for enabling and configuring tracing.
1. You can enable and configure tracing on each service individually. For example, enable tracing
in Reva when starting the oCIS single binary like this:
```console
REVA_TRACING_ENABLED=true \
REVA_TRACING_ENDPOINT=localhost:6831 \
REVA_TRACING_COLLECTOR=http://localhost:14268/api/traces \
./bin/ocis server
```
2. Enabling and configuring tracing on oCIS itself will forward the configuration to all services:
```console
OCIS_TRACING_ENABLED=true \
OCIS_TRACING_ENDPOINT=localhost:6831 \
OCIS_TRACING_COLLECTOR=http://localhost:14268/api/traces \
./bin/ocis server
```
If you want to set individual tracing configuration for each service, make sure to set
`OCIS_TRACING_ENABLED=false`.
1. You can enable and configure tracing on each service individually. For example, enable tracing
in Reva when starting the oCIS single binary like this:
```console
REVA_TRACING_ENABLED=true \
REVA_TRACING_ENDPOINT=localhost:6831 \
REVA_TRACING_COLLECTOR=http://localhost:14268/api/traces \
./bin/ocis server
```
2. Enabling and configuring tracing on oCIS itself will forward the configuration to all services:
```console
OCIS_TRACING_ENABLED=true \
OCIS_TRACING_ENDPOINT=localhost:6831 \
OCIS_TRACING_COLLECTOR=http://localhost:14268/api/traces \
./bin/ocis server
```
If you want to set individual tracing configuration for each service, make sure to set
`OCIS_TRACING_ENABLED=false`.
3. Make the actual request that you want to trace.
4. Open up the [Jaeger UI](http://localhost:16686) to analyze request traces.