From 37db60f4d47f3853ba90cf07092ac42c67fe3016 Mon Sep 17 00:00:00 2001 From: Kegsay Date: Tue, 14 Jul 2020 12:58:00 +0100 Subject: [PATCH] Update README/docs (#1199) * Update README * Update docs --- README.md | 68 ++++++++++++++++---- docs/CODE_STYLE.md | 42 ++----------- docs/INSTALL.md | 151 +++++++++++++++++++-------------------------- 3 files changed, 122 insertions(+), 139 deletions(-) diff --git a/README.md b/README.md index aa9060f1..168a864a 100644 --- a/README.md +++ b/README.md @@ -1,22 +1,68 @@ # Dendrite [![Build Status](https://badge.buildkite.com/4be40938ab19f2bbc4a6c6724517353ee3ec1422e279faf374.svg?branch=master)](https://buildkite.com/matrix-dot-org/dendrite) [![Dendrite Dev on Matrix](https://img.shields.io/matrix/dendrite-dev:matrix.org.svg?label=%23dendrite-dev%3Amatrix.org&logo=matrix&server_fqdn=matrix.org)](https://matrix.to/#/#dendrite-dev:matrix.org) [![Dendrite on Matrix](https://img.shields.io/matrix/dendrite:matrix.org.svg?label=%23dendrite%3Amatrix.org&logo=matrix&server_fqdn=matrix.org)](https://matrix.to/#/#dendrite:matrix.org) -Dendrite will be a second-generation Matrix homeserver written in Go. +Dendrite is a second-generation Matrix homeserver written in Go. It is not recommended to use Dendrite as +a production homeserver at this time as there is no stable release. An overview of the design can be found +in [DESIGN.md](docs/DESIGN.md). -It's still very much a work in progress, but installation instructions can be -found in [INSTALL.md](docs/INSTALL.md). It is not recommended to use Dendrite as a -production homeserver at this time. +# Quick start + +Requires Go 1.13+ and SQLite3 (Postgres is also supported): + +```bash +$ git clone https://github.com/matrix-org/dendrite +$ cd dendrite + +# generate self-signed certificate and an event signing key for federation +$ go build ./cmd/generate-keys +$ ./generate-keys --private-key matrix_key.pem --tls-cert server.crt --tls-key server.key + +# Copy and modify the config file: +# you'll need to set a server name and paths to the keys at the very least, along with setting +# up the database filenames +$ cp dendrite-config.yaml dendrite.yaml + +# build and run the server +$ go build ./cmd/dendrite-monolith-server +$ ./dendrite-monolith-server --tls-cert server.crt --tls-key server.key --config dendrite.yaml +``` + +Then point your favourite Matrix client at `http://localhost:8008`. For full installation information, see +[INSTALL.md](docs/INSTALL.md). For running in Docker, see [build/docker](build/docker). + +# Progress + +We use a script called Are We Synapse Yet which checks Sytest compliance rates. Sytest is a black-box homeserver +test rig with around 900 tests. The script works out how many of these tests are passing on Dendrite and it +updates with CI. As of July 2020 we're at around 48% CS API coverage and 50% Federation coverage, though check +CI for the latest numbers. In practice, this means you can communicate locally and via federation with Synapse +servers such as matrix.org reasonably well. There's a long list of features that are not implemented, notably: + - Receipts + - Push + - Search and Context + - User Directory + - Presence + - Guests + - E2E keys and device lists + +We are prioritising features that will benefit single-user homeservers first (e.g Receipts, E2E) rather +than features that massive deployments may be interested in (User Directory, OpenID, Guests, Admin APIs, AS API). +This means Dendrite supports amongst others: + - Core room functionality (creating rooms, invites, auth rules) + - Federation in rooms v1-v6 + - Backfilling locally and via federation + - Accounts, Profiles and Devices + - Published room lists + - Typing + - Media APIs + - Redaction + - Tagging -An overview of the design can be found in [DESIGN.md](docs/DESIGN.md). # Contributing Everyone is welcome to help out and contribute! See [CONTRIBUTING.md](docs/CONTRIBUTING.md) to get started! -Please note that, as of February 2020, Dendrite now only targets Go 1.13 or -later. Please ensure that you are using at least Go 1.13 when developing for -Dendrite. - # Discussion For questions about Dendrite we have a dedicated room on Matrix @@ -24,8 +70,4 @@ For questions about Dendrite we have a dedicated room on Matrix discussion should happen in [#dendrite-dev:matrix.org](https://matrix.to/#/#dendrite-dev:matrix.org). -# Progress - -There's plenty still to do to make Dendrite usable! We're tracking progress in a -[project board](https://github.com/matrix-org/dendrite/projects/2). diff --git a/docs/CODE_STYLE.md b/docs/CODE_STYLE.md index 1f40c76f..248d7610 100644 --- a/docs/CODE_STYLE.md +++ b/docs/CODE_STYLE.md @@ -1,12 +1,7 @@ # Code Style -We follow the standard Go style using goimports, but with a few extra -considerations. - -## Linters - -We use `golangci-lint` to run a number of linters, the exact list can be found -under linters in [.golangci.yml](.golangci.yml). +In addition to standard Go code style (`gofmt`, `goimports`), we use `golangci-lint` +to run a number of linters, the exact list can be found under linters in [.golangci.yml](.golangci.yml). [Installation](https://github.com/golangci/golangci-lint#install) and [Editor Integration](https://github.com/golangci/golangci-lint#editor-integration) for it can be found in the readme of golangci-lint. @@ -22,37 +17,10 @@ The linters can be run using [scripts/find-lint.sh](scripts/find-lint.sh) [scripts/build-test-lint.sh](scripts/build-test-lint.sh). -## HTTP Error Handling - -Unfortunately, converting errors into HTTP responses with the correct status -code and message can be done in a number of ways in golang: - -1. Having functions return `JSONResponse` directly, which can then either set - it to an error response or a `200 OK`. -2. Have the HTTP handler try and cast error values to types that are handled - differently. -3. Have the HTTP handler call functions whose errors can only be interpreted - one way, for example if a `validate(...)` call returns an error then handler - knows to respond with a `400 Bad Request`. - -We attempt to always use option #3, as it more naturally fits with the way that -golang generally does error handling. In particular, option #1 effectively -requires reinventing a new error handling scheme just for HTTP handlers. - - -## Line length - -We strive for a line length of roughly 80 characters, though less than 100 is -acceptable if necessary. Longer lines are fine if there is nothing of interest -after the first 80-100 characters (e.g. long string literals). - - -## TODOs and FIXMEs - -The majority of TODOs and FIXMEs should have an associated tracking issue on -github. These can be added just before merging of the PR to master, and the -issue number should be added to the comment, e.g. `// TODO(#324): ...` +## Labels +In addition to `TODO` and `FIXME` we also use `NOTSPEC` to identify deviations +from the Matrix specification. ## Logging diff --git a/docs/INSTALL.md b/docs/INSTALL.md index c9735180..cf8adee8 100644 --- a/docs/INSTALL.md +++ b/docs/INSTALL.md @@ -3,20 +3,21 @@ Dendrite can be run in one of two configurations: * **Polylith mode**: A cluster of individual components, dealing with different - aspects of the Matrix protocol (see [WIRING.md](WIRING.md)). Components communicate with each other using internal HTTP APIs and [Apache Kafka](https://kafka.apache.org). This will almost certainly be the preferred model - for large-scale deployments. + aspects of the Matrix protocol (see [WIRING.md](WIRING-Current.md)). Components communicate + with each other using internal HTTP APIs and [Apache Kafka](https://kafka.apache.org). + This will almost certainly be the preferred model for large-scale deployments. * **Monolith mode**: All components run in the same process. In this mode, Kafka is completely optional and can instead be replaced with an in-process - lightweight implementation called [Naffka](https://github.com/matrix-org/naffka). This will usually be the preferred model for low-volume, low-user - or experimental deployments. + lightweight implementation called [Naffka](https://github.com/matrix-org/naffka). This + will usually be the preferred model for low-volume, low-user or experimental deployments. -Regardless of whether you are running in polylith or monolith mode, each Dendrite component that requires storage has its own database. Both Postgres -and SQLite are supported and can be mixed-and-matched across components as -needed in the configuration file. +Regardless of whether you are running in polylith or monolith mode, each Dendrite component that +requires storage has its own database. Both Postgres and SQLite are supported and can be +mixed-and-matched across components as needed in the configuration file. -Be advised that Dendrite is still developmental and it's not recommended for -use in production environments yet! +Be advised that Dendrite is still in development and it's not recommended for +use in production environments just yet! ## Requirements @@ -119,16 +120,10 @@ Assuming that Postgres 9.5 (or later) is installed: Each Dendrite server requires unique server keys. -Generate the self-signed SSL certificate for federation: +Generate the self-signed SSL certificate for federation and the server signing key: ```bash -test -f server.key || openssl req -x509 -newkey rsa:4096 -keyout server.key -out server.crt -days 3650 -nodes -subj /CN=localhost -``` - -Generate the server signing key: - -``` -test -f matrix_key.pem || ./bin/generate-keys -private-key matrix_key.pem +./bin/generate-keys --private-key matrix_key.pem --tls-cert server.crt --tls-key server.key ``` ### Configuration file @@ -152,7 +147,8 @@ public keys for dead homeservers from somewhere else. ## Starting a monolith server It is possible to use Naffka as an in-process replacement to Kafka when using -the monolith server. To do this, set `use_naffka: true` in your `dendrite.yaml` configuration and uncomment the relevant Naffka line in the `database` section. +the monolith server. To do this, set `use_naffka: true` in your `dendrite.yaml` +configuration and uncomment the relevant Naffka line in the `database` section. Be sure to update the database username and password if needed. The monolith server can be started as shown below. By default it listens for @@ -166,60 +162,7 @@ as shown below, it will also listen for HTTPS connections on port 8448. ## Starting a polylith deployment -The following contains scripts which will run all the required processes in order to point a Matrix client at Dendrite. Conceptually, you are wiring together to form the following diagram: - -``` - - /media +---------------------------+ - +----------->+------------->| dendrite-media-api-server | - ^ ^ +---------------------------+ - | | :7774 - | | - | | - | | - | | - | | - | | - | | - | | - | | /sync +--------------------------+ - | | +--------->| dendrite-sync-api-server |<================++ - | | | +--------------------------+ || - | | | :7773 | ^^ || -Matrix +------------------+ | | | || client_data || -Clients --->| client-api-proxy |-------+ +<-----------+ ++=============++ || - +------------------+ | | | || || - :8008 | | CS API +----------------------------+ || || - | +--------->| dendrite-client-api-server |==++ || - | | +----------------------------+ || - | | :7771 | || - | | | || - | +<-----------+ || - | | || - | | || - | | +----------------------+ room_event || - | +---------->| dendrite-room-server |===============++ - | | +----------------------+ || - | | :7770 || - | | ++==========================++ - | +<------------+ || - | | | VV - | | +-----------------------------------+ Matrix - | | | dendrite-federation-sender-server |------------> Servers - | | +-----------------------------------+ - | | :7776 - | | - +---------->+ +<-----------+ - | | -Matrix +----------------------+ SS API +--------------------------------+ -Servers --->| federation-api-proxy |--------->| dendrite-federation-api-server | - +----------------------+ +--------------------------------+ - :8448 :7772 - - - A --> B = HTTP requests (A = client, B = server) - A ==> B = Kafka (A = producer, B = consumer) -``` +The following contains scripts which will run all the required processes in order to point a Matrix client at Dendrite. ### Client proxy @@ -248,21 +191,13 @@ to support federation. ### Client API server -This is what implements message sending. Clients talk to this via the proxy in -order to send messages. +This is what implements CS API endpoints. Clients talk to this via the proxy in +order to send messages, create and join rooms, etc. ```bash ./bin/dendrite-client-api-server --config=dendrite.yaml ``` -### Room server - -This is what implements the room DAG. Clients do not talk to this. - -```bash -./bin/dendrite-room-server --config=dendrite.yaml -``` - ### Sync server This is what implements `/sync` requests. Clients talk to this via the proxy @@ -283,7 +218,7 @@ order to upload and retrieve media. ### Federation API server -This implements federation requests. Servers talk to this via the proxy in +This implements the federation API. Servers talk to this via the proxy in order to send transactions. This is only required if you want to support federation. @@ -291,7 +226,30 @@ federation. ./bin/dendrite-federation-api-server --config dendrite.yaml ``` -### Federation sender +### Internal components + +This refers to components that are not directly spoken to by clients. They are only +contacted by other components. This includes the following components. + +#### Room server + +This is what implements the room DAG. Clients do not talk to this. + +```bash +./bin/dendrite-room-server --config=dendrite.yaml +``` + +#### Current state server + +This tracks the current state of rooms which various components need to know. For example, +`/publicRooms` implemented by client API asks this server for the room names, joined member +counts, etc. + +```bash +./bin/dendrite-current-state-server --config=dendrite.yaml +``` + +#### Federation sender This sends events from our users to other servers. This is only required if you want to support federation. @@ -300,7 +258,7 @@ you want to support federation. ./bin/dendrite-federation-sender-server --config dendrite.yaml ``` -### Appservice server +#### Appservice server This sends events from the network to [application services](https://matrix.org/docs/spec/application_service/unstable.html) @@ -311,16 +269,31 @@ application services on your homeserver. ./bin/dendrite-appservice-server --config dendrite.yaml ``` -### Key server +#### Key server -This manages end-to-end encryption keys (or rather, it will do when it's -finished). +This manages end-to-end encryption keys for users. ```bash ./bin/dendrite-key-server --config dendrite.yaml ``` -### User server +#### Server Key server + +This manages signing keys for servers. + +```bash +./bin/dendrite-server-key-api-server --config dendrite.yaml +``` + +#### EDU server + +This manages processing EDUs such as typing, send-to-device events and presence. Clients do not talk to + +```bash +./bin/dendrite-edu-server --config dendrite.yaml +``` + +#### User server This manages user accounts, device access tokens and user account data, amongst other things.