Merge branch 'master' into ocis-1715-lighter-runtime

docs/ocis/_index.md

@@ -13,13 +13,34 @@ geekdocFilePath: _index.md
 
 Welcome to oCIS, the modern file-sync and share platform, which is based on our knowledge and experience with the PHP based [ownCloud server](https://owncloud.com/#server).
 
-### oCIS server
+### oCIS microservice runtime
 
-The oCIS server implementation follows [Go](https://golang.org/) best practices and is based on the [go-micro](https://go-micro.dev/) framework and [REVA](https://reva.link/). We love and stick to [12 Factor](https://12factor.net/).
-oCIS is a micro-service based server, which allows scale-out of individual services to meet your specific performance requirements.
-We run a huge [test suite](https://github.com/owncloud/core/tree/master/tests), which originated in ownCloud 10 and continues to grow.
+The oCIS runtime allows us to dynamically manage services running in a single process. We use [suture](https://github.com/thejerf/suture) to create a supervisor tree that starts each service in a dedicated goroutine. Be default oCIS will start all built in oCIS extensions in a single process. Individual services can be moved to other nodes to scale-out and meet specific performance requirements. A [go-micro](https://github.com/asim/go-micro/blob/master/registry/registry.go) based registry allows services in multiple nodes to form a distributed microservice architecture.
+
+### oCIS extensions
+
+Every oCIS extension uses [ocis-pkg](https://github.com/owncloud/ocis/ocis-pkg), which implements the [go-micro](https://go-micro.dev/) interfaces for [servers](https://github.com/asim/go-micro/blob/v3.5.0/server/server.go#L17-L37) to register and [clients](https://github.com/asim/go-micro/blob/v3.5.0/client/client.go#L11-L23) to lookup nodes with a service [registry](https://github.com/asim/go-micro/blob/v3.5.0/registry/registry.go).
+We are following the [12 Factor](https://12factor.net/)methodology with oCIS. The uniformity of services also allows us to use the same command, logging and configuration mechanism. Configurations are forwarded from the 
+oCIS runtime to the individual extensions.
+
+
+### go-micro
+
+While the [go-micro](https://go-micro.dev/) framework provides abstractions as well as implementations for the different components in a microservice architecture it uses a more developer focused runtime philosophy: it is used to download a services from a repo, compile them on the fly and start them as individual processes. For oCIS we decided to use a more admin friendly runtime: you can download a single binary and start the contained oCIS extensions with a single `bin/ocis server`. This also makes packaging easier.
+
+We use[ocis-pkg](https://github.com/owncloud/ocis/ocis-pkg) to configure the default implementations for the go-micro [grpc server](https://github.com/asim/go-micro/tree/v3.5.0/plugins/server/grpc), [client](https://github.com/asim/go-micro/tree/v3.5.0/plugins/client/grpc) and [mdns registry](https://github.com/asim/go-micro/blob/v3.5.0/registry/mdns_registry.go), swapping them out as needed, eg. to use the [kubernetes registry plugin](https://github.com/asim/go-micro/tree/v3.5.0/plugins/registry/kubernetes).
+
+### REVA
+A lot of embedded services in oCIS are built upon the [REVA](https://reva.link/) runtime. We decided to bundle some of the [CS3 services](https://github.com/cs3org/cs3apis) to logically group them. A [home storage provider](https://github.com/owncloud/ocis/blob/v1.2.0/storage/pkg/command/storagehome.go#L93-L108), which is dealing with [metadata](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ProviderAPI), and the corresponding [data provider](https://github.com/owncloud/ocis/blob/v1.2.0/storage/pkg/command/storagehome.go#L109-L123), which is dealing with [up and download](https://cs3org.github.io/cs3apis/#cs3.gateway.v1beta1.FileUploadProtocol), are one example. The [frontend](https://github.com/owncloud/ocis/blob/v1.2.0/storage/pkg/command/frontend.go) with the [oc flavoured webdav](https://github.com/owncloud/ocis/blob/v1.2.0/storage/pkg/command/frontend.go#L132-L138), [ocs handlers](https://github.com/owncloud/ocis/blob/v1.2.0/storage/pkg/command/frontend.go#L139-L148) and a [datagateway](https://github.com/owncloud/ocis/blob/v1.2.0/storage/pkg/command/frontend.go#L126-L131) are another.
+
+### Protocol driven development
+Interacting with oCIS involves a multitude af APIs. The server and all clients rely on [OpenID Connect](https://openid.net/connect/) for authentication. The [embedded konnectd](https://github.com/owncloud/ocis/tree/master/idp) can be replaced with any other OpenID Connect Identity Provider. Clients use the [WebDAV](http://webdav.org/) based [oc sync protocol](https://github.com/cernbox/smashbox/blob/master/protocol/protocol.md) to manage files and folders, [ocs to manage shares](https://doc.owncloud.com/server/developer_manual/core/apis/ocs-share-api.html) and [TUS](https://tus.io/protocols/resumable-upload.html) to upload files in a resumable way. On the server side [REVA](https://reva.link/) is the reference implementation of the [CS3 apis](https://github.com/cs3org/cs3apis) which is defined using [protobuf](https://developers.google.com/protocol-buffers/). By embedding [glauth](https://github.com/glauth/glauth/), oCIS provides a read only [LDAP](https://tools.ietf.org/html/rfc2849) interface to make accounts, including guests available to firewalls and other systems. In the future, we are looking into [the Microsoft Graph API](https://docs.microsoft.com/en-us/graph/api/overview?view=graph-rest-1.0), which is based on [odata](http://docs.oasis-open.org/odata/odata/v4.0/odata-v4.0-part1-protocol.html) as a well defined REST/JSON dialect for the existing endpoints.
+
+### Acceptance test suite
+We run a huge [test suite](https://github.com/owncloud/core/tree/master/tests), which originated in ownCloud 10 and continues to grow. A detailed description can be found in the developer docs for [testing]({{< relref "development/testing.md" >}}).
 
 ### Architecture Overview
 
+Running `bin/ocis server` will start the below services, all of which can be scaled and deployed on a single node or in a cloud native environment, as needed.
 
 {{< svg src="ocis/static/architecture-overview.drawio.svg" >}}

docs/ocis/development/getting-started.md

@@ -16,7 +16,7 @@ So we are trying to reflect this in the tooling. It should be kept simple and qu
 
 Besides standard development tools like git and a text editor, you need the following software for development:
 
-- Go >= v1.13 ([install instructions](https://golang.org/doc/install))
+- Go >= v1.15 ([install instructions](https://golang.org/doc/install))
 - Yarn ([install instructions](https://classic.yarnpkg.com/en/docs/install))
 - docker ([install instructions](https://docs.docker.com/get-docker/))
 - docker-compose ([install instructions](https://docs.docker.com/compose/install/))
@@ -25,17 +25,15 @@ If you find tools needed besides the mentioned above, please feel free to open a
 
 ## Repository structure
 
-This repository follows the [golang-standard project-layout](https://github.com/golang-standards/project-layout).
-
 oCIS consists of multiple micro services, also called extensions. We started by having standalone repositories for each of them but quickly noticed, that this adds a time consuming overhead for developers. So we ended up with a monorepo housing all the extensions in one repository.
 
-Each of the extensions live in a subfolder (eg. `accounts` or `settings`) in this repository, technically creating independent Go modules.
+Each extension lives in a subfolder (eg. `accounts` or `settings`) in this respository as an independent Go module, following the [golang-standard project-layout](https://github.com/golang-standards/project-layout). They have common Makefile targets and can be used to change, build and run individual extensions.
 
-The `ocis` folder does also contain a Go module but is no extension at all. Instead this module is used to import all extensions and furthermore implement commands to start the extensions. With the resulting oCIS binary you can start single extensions or even all extensions at the same time.
+The `ocis` folder contains our [go-micro](https://github.com/asim/go-micro/) and [suture](https://github.com/thejerf/suture) based runtime. It is used to import all extensions and implements commands to manage them, similar to a small orchestrator. With the resulting oCIS binary you can start single extensions or even all extensions at the same time.
 
 The `docs` folder contains the source for the [oCIS documentation](https://owncloud.github.io/ocis/).
 
-The `deployments` folder contains documented deployment configurations and templates.
+The `deployments` folder contains documented deployment configurations and templates. On a single node, running a single ocis runtime is a resource efficient way to deploy ocis. For multiple nodes docker compose or helm charts for kubernetes examples can be found here.
 
 The `scripts` folder contains scripts to perform various build, install, analysis, etc operations.