Update README/docs (#1199)

* Update README

* Update docs

README.md

@@ -1,22 +1,68 @@
 # Dendrite [![Build Status](https://badge.buildkite.com/4be40938ab19f2bbc4a6c6724517353ee3ec1422e279faf374.svg?branch=master)](https://buildkite.com/matrix-dot-org/dendrite) [![Dendrite Dev on Matrix](https://img.shields.io/matrix/dendrite-dev:matrix.org.svg?label=%23dendrite-dev%3Amatrix.org&logo=matrix&server_fqdn=matrix.org)](https://matrix.to/#/#dendrite-dev:matrix.org) [![Dendrite on Matrix](https://img.shields.io/matrix/dendrite:matrix.org.svg?label=%23dendrite%3Amatrix.org&logo=matrix&server_fqdn=matrix.org)](https://matrix.to/#/#dendrite:matrix.org)
 
-Dendrite will be a second-generation Matrix homeserver written in Go.
+Dendrite is a second-generation Matrix homeserver written in Go. It is not recommended to use Dendrite as
+a production homeserver at this time as there is no stable release. An overview of the design can be found
+in [DESIGN.md](docs/DESIGN.md).
 
-It's still very much a work in progress, but installation instructions can be
+# Quick start
-found in [INSTALL.md](docs/INSTALL.md). It is not recommended to use Dendrite as a
+
-production homeserver at this time.
+Requires Go 1.13+ and SQLite3 (Postgres is also supported):
+
+```bash
+$ git clone https://github.com/matrix-org/dendrite
+$ cd dendrite
+
+# generate self-signed certificate and an event signing key for federation
+$ go build ./cmd/generate-keys
+$ ./generate-keys --private-key matrix_key.pem --tls-cert server.crt --tls-key server.key
+
+# Copy and modify the config file:
+# you'll need to set a server name and paths to the keys at the very least, along with setting
+# up the database filenames
+$ cp dendrite-config.yaml dendrite.yaml
+
+# build and run the server
+$ go build ./cmd/dendrite-monolith-server
+$ ./dendrite-monolith-server --tls-cert server.crt --tls-key server.key --config dendrite.yaml
+```
+
+Then point your favourite Matrix client at `http://localhost:8008`. For full installation information, see
+[INSTALL.md](docs/INSTALL.md). For running in Docker, see [build/docker](build/docker).
+
+# Progress
+
+We use a script called Are We Synapse Yet which checks Sytest compliance rates. Sytest is a black-box homeserver
+test rig with around 900 tests. The script works out how many of these tests are passing on Dendrite and it
+updates with CI. As of July 2020 we're at around 48% CS API coverage and 50% Federation coverage, though check
+CI for the latest numbers. In practice, this means you can communicate locally and via federation with Synapse
+servers such as matrix.org reasonably well. There's a long list of features that are not implemented, notably:
+ - Receipts
+ - Push
+ - Search and Context
+ - User Directory
+ - Presence
+ - Guests
+ - E2E keys and device lists
+
+We are prioritising features that will benefit single-user homeservers first (e.g Receipts, E2E) rather
+than features that massive deployments may be interested in (User Directory, OpenID, Guests, Admin APIs, AS API).
+This means Dendrite supports amongst others:
+ - Core room functionality (creating rooms, invites, auth rules)
+ - Federation in rooms v1-v6
+ - Backfilling locally and via federation
+ - Accounts, Profiles and Devices
+ - Published room lists
+ - Typing
+ - Media APIs
+ - Redaction
+ - Tagging
 
-An overview of the design can be found in [DESIGN.md](docs/DESIGN.md).
 
 # Contributing
 
 Everyone is welcome to help out and contribute! See
 [CONTRIBUTING.md](docs/CONTRIBUTING.md) to get started!
 
-Please note that, as of February 2020, Dendrite now only targets Go 1.13 or
-later. Please ensure that you are using at least Go 1.13 when developing for
-Dendrite.
-
 # Discussion
 
 For questions about Dendrite we have a dedicated room on Matrix
@@ -24,8 +70,4 @@ For questions about Dendrite we have a dedicated room on Matrix
 discussion should happen in
 [#dendrite-dev:matrix.org](https://matrix.to/#/#dendrite-dev:matrix.org).
 
-# Progress
-
-There's plenty still to do to make Dendrite usable! We're tracking progress in a
-[project board](https://github.com/matrix-org/dendrite/projects/2).
 

docs/CODE_STYLE.md

@@ -1,12 +1,7 @@
 # Code Style
 
-We follow the standard Go style using goimports, but with a few extra
+In addition to standard Go code style (`gofmt`, `goimports`), we use `golangci-lint`
-considerations.
+to run a number of linters, the exact list can be found under linters in [.golangci.yml](.golangci.yml).
-
-## Linters
-
-We use `golangci-lint` to run a number of linters, the exact list can be found
-under linters in [.golangci.yml](.golangci.yml).
 [Installation](https://github.com/golangci/golangci-lint#install) and [Editor
 Integration](https://github.com/golangci/golangci-lint#editor-integration) for
 it can be found in the readme of golangci-lint.
@@ -22,37 +17,10 @@ The linters can be run using [scripts/find-lint.sh](scripts/find-lint.sh)
 [scripts/build-test-lint.sh](scripts/build-test-lint.sh).
 
 
-## HTTP Error Handling
+## Labels
-
-Unfortunately, converting errors into HTTP responses with the correct status
-code and message can be done in a number of ways in golang:
-
-1. Having functions return `JSONResponse` directly, which can then either set
-   it to an error response or a `200 OK`.
-2. Have the HTTP handler try and cast error values to types that are handled
-   differently.
-3. Have the HTTP handler call functions whose errors can only be interpreted
-   one way, for example if a `validate(...)` call returns an error then handler
-   knows to respond with a `400 Bad Request`.
-
-We attempt to always use option #3, as it more naturally fits with the way that
-golang generally does error handling. In particular, option #1 effectively
-requires reinventing a new error handling scheme just for HTTP handlers.
-
-
-## Line length
-
-We strive for a line length of roughly 80 characters, though less than 100 is
-acceptable if necessary. Longer lines are fine if there is nothing of interest
-after the first 80-100 characters (e.g. long string literals).
-
-
-## TODOs and FIXMEs
-
-The majority of TODOs and FIXMEs should have an associated tracking issue on
-github. These can be added just before merging of the PR to master, and the
-issue number should be added to the comment, e.g. `// TODO(#324): ...`
 
+In addition to `TODO` and `FIXME` we also use `NOTSPEC` to identify deviations
+from the Matrix specification.
 
 ## Logging
 

docs/INSTALL.md

@@ -3,20 +3,21 @@
 Dendrite can be run in one of two configurations:
 
 * **Polylith mode**: A cluster of individual components, dealing with different
-  aspects of the Matrix protocol (see [WIRING.md](WIRING.md)). Components communicate with each other using internal HTTP APIs and [Apache Kafka](https://kafka.apache.org). This will almost certainly be the preferred model
+  aspects of the Matrix protocol (see [WIRING.md](WIRING-Current.md)). Components communicate
-  for large-scale deployments.
+  with each other using internal HTTP APIs and [Apache Kafka](https://kafka.apache.org).
+  This will almost certainly be the preferred model for large-scale deployments.
 
 * **Monolith mode**: All components run in the same process. In this mode,
    Kafka is completely optional and can instead be replaced with an in-process
-   lightweight implementation called [Naffka](https://github.com/matrix-org/naffka). This will usually be the preferred model for low-volume, low-user
+   lightweight implementation called [Naffka](https://github.com/matrix-org/naffka). This
-   or experimental deployments.
+   will usually be the preferred model for low-volume, low-user or experimental deployments.
 
-Regardless of whether you are running in polylith or monolith mode, each Dendrite component that requires storage has its own database. Both Postgres
+Regardless of whether you are running in polylith or monolith mode, each Dendrite component that
-and SQLite are supported and can be mixed-and-matched across components as
+requires storage has its own database. Both Postgres and SQLite are supported and can be
-needed in the configuration file.
+mixed-and-matched across components as needed in the configuration file.
 
-Be advised that Dendrite is still developmental and it's not recommended for
+Be advised that Dendrite is still in development and it's not recommended for
-use in production environments yet!
+use in production environments just yet!
 
 ## Requirements
 
@@ -119,16 +120,10 @@ Assuming that Postgres 9.5 (or later) is installed:
 
 Each Dendrite server requires unique server keys.
 
-Generate the self-signed SSL certificate for federation:
+Generate the self-signed SSL certificate for federation and the server signing key:
 
 ```bash
-test -f server.key || openssl req -x509 -newkey rsa:4096 -keyout server.key -out server.crt -days 3650 -nodes -subj /CN=localhost
+./bin/generate-keys --private-key matrix_key.pem --tls-cert server.crt --tls-key server.key
-```
-
-Generate the server signing key:
-
-```
-test -f matrix_key.pem || ./bin/generate-keys -private-key matrix_key.pem
 ```
 
 ### Configuration file
@@ -152,7 +147,8 @@ public keys for dead homeservers from somewhere else.
 ## Starting a monolith server
 
 It is possible to use Naffka as an in-process replacement to Kafka when using
-the monolith server. To do this, set `use_naffka: true` in your `dendrite.yaml` configuration and uncomment the relevant Naffka line in the `database` section.
+the monolith server. To do this, set `use_naffka: true` in your `dendrite.yaml`
+configuration and uncomment the relevant Naffka line in the `database` section.
 Be sure to update the database username and password if needed.
 
 The monolith server can be started as shown below. By default it listens for
@@ -166,60 +162,7 @@ as shown below, it will also listen for HTTPS connections on port 8448.
 
 ## Starting a polylith deployment
 
-The following contains scripts which will run all the required processes in order to point a Matrix client at Dendrite. Conceptually, you are wiring together to form the following diagram:
+The following contains scripts which will run all the required processes in order to point a Matrix client at Dendrite.
-
-```
-
-                                         /media   +---------------------------+
-                      +----------->+------------->| dendrite-media-api-server |
-                      ^            ^              +---------------------------+
-                      |            |            :7774
-                      |            |
-                      |            |
-                      |            |   
-                      |            |   
-                      |            |   
-                      |            |  
-                      |            |  
-                      |            |  
-                      |            |     /sync    +--------------------------+                 
-                      |            |   +--------->| dendrite-sync-api-server |<================++
-                      |            |   |          +--------------------------+                 ||
-                      |            |   |        :7773    |         ^^                          ||
-Matrix      +------------------+   |   |                 |         ||    client_data           ||
-Clients --->| client-api-proxy |-------+    +<-----------+         ++=============++           ||
-            +------------------+   |   |    |                                     ||           ||
-          :8008                    |   | CS API   +----------------------------+  ||           ||
-                                   |   +--------->| dendrite-client-api-server |==++           ||
-                                   |        |     +----------------------------+               ||
-                                   |        |   :7771    |                                     ||
-                                   |        |            |                                     ||
-                                   |        +<-----------+                                     ||
-                                   |        |                                                  ||
-                                   |        |                                                  ||
-                                   |        |           +----------------------+    room_event ||
-                                   |        +---------->| dendrite-room-server |===============++
-                                   |        |           +----------------------+               ||
-                                   |        |         :7770                                    ||
-                                   |        |                      ++==========================++
-                                   |        +<------------+        ||
-                                   |        |             |        VV
-                                   |        |     +-----------------------------------+              Matrix
-                                   |        |     | dendrite-federation-sender-server |------------> Servers
-                                   |        |     +-----------------------------------+
-                                   |        |   :7776
-                                   |        |
-                       +---------->+        +<-----------+
-                       |                                 |
-Matrix      +----------------------+  SS API  +--------------------------------+
-Servers --->| federation-api-proxy |--------->| dendrite-federation-api-server |
-            +----------------------+          +--------------------------------+
-          :8448                             :7772
-
-
-   A --> B  = HTTP requests (A = client, B = server)
-   A ==> B  = Kafka (A = producer, B = consumer)
-```
 
 ### Client proxy
 
@@ -248,21 +191,13 @@ to support federation.
 
 ### Client API server
 
-This is what implements message sending. Clients talk to this via the proxy in
+This is what implements CS API endpoints. Clients talk to this via the proxy in
-order to send messages.
+order to send messages, create and join rooms, etc.
 
 ```bash
 ./bin/dendrite-client-api-server --config=dendrite.yaml
 ```
 
-### Room server
-
-This is what implements the room DAG. Clients do not talk to this.
-
-```bash
-./bin/dendrite-room-server --config=dendrite.yaml
-```
-
 ### Sync server
 
 This is what implements `/sync` requests. Clients talk to this via the proxy
@@ -283,7 +218,7 @@ order to upload and retrieve media.
 
 ### Federation API server
 
-This implements federation requests. Servers talk to this via the proxy in
+This implements the federation API. Servers talk to this via the proxy in
 order to send transactions.  This is only required if you want to support
 federation.
 
@@ -291,7 +226,30 @@ federation.
 ./bin/dendrite-federation-api-server --config dendrite.yaml
 ```
 
-### Federation sender
+### Internal components
+
+This refers to components that are not directly spoken to by clients. They are only
+contacted by other components. This includes the following components.
+
+#### Room server
+
+This is what implements the room DAG. Clients do not talk to this.
+
+```bash
+./bin/dendrite-room-server --config=dendrite.yaml
+```
+
+#### Current state server
+
+This tracks the current state of rooms which various components need to know. For example,
+`/publicRooms` implemented by client API asks this server for the room names, joined member
+counts, etc.
+
+```bash
+./bin/dendrite-current-state-server --config=dendrite.yaml
+```
+
+#### Federation sender
 
 This sends events from our users to other servers.  This is only required if
 you want to support federation.
@@ -300,7 +258,7 @@ you want to support federation.
 ./bin/dendrite-federation-sender-server --config dendrite.yaml
 ```
 
-### Appservice server
+#### Appservice server
 
 This sends events from the network to [application
 services](https://matrix.org/docs/spec/application_service/unstable.html)
@@ -311,16 +269,31 @@ application services on your homeserver.
 ./bin/dendrite-appservice-server --config dendrite.yaml
 ```
 
-### Key server
+#### Key server
 
-This manages end-to-end encryption keys (or rather, it will do when it's
+This manages end-to-end encryption keys for users.
-finished).
 
 ```bash
 ./bin/dendrite-key-server --config dendrite.yaml
 ```
 
-### User server
+#### Server Key server
+
+This manages signing keys for servers.
+
+```bash
+./bin/dendrite-server-key-api-server --config dendrite.yaml
+```
+
+#### EDU server
+
+This manages processing EDUs such as typing, send-to-device events and presence. Clients do not talk to
+
+```bash
+./bin/dendrite-edu-server --config dendrite.yaml
+```
+
+#### User server
 
 This manages user accounts, device access tokens and user account data,
 amongst other things.