Update INSTALL.md, move docs (#1034)

* Update INSTALL.md

* Move some things

* Clean up

* Move some more things

* Don't build all the things for the monolith

* Update INSTALL.md

* Nuke hooks
This commit is contained in:
Neil Alexander 2020-05-14 16:49:18 +01:00 committed by GitHub
parent 1b34130a5b
commit 3cb04e8004
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
10 changed files with 153 additions and 134 deletions

View File

@ -1,18 +0,0 @@
root = true
[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true
[*.go]
indent_style = tab
indent_size = 4
[*.md]
trim_trailing_whitespace = false
[*.{yml,yaml}]
indent_style = space

View File

@ -3,15 +3,15 @@
Dendrite will be a second-generation Matrix homeserver written in Go. Dendrite will be a second-generation Matrix homeserver written in Go.
It's still very much a work in progress, but installation instructions can be It's still very much a work in progress, but installation instructions can be
found in [INSTALL.md](INSTALL.md). It is not recommended to use Dendrite as a found in [INSTALL.md](docs/INSTALL.md). It is not recommended to use Dendrite as a
production homeserver at this time. production homeserver at this time.
An overview of the design can be found in [DESIGN.md](DESIGN.md). An overview of the design can be found in [DESIGN.md](docs/DESIGN.md).
# Contributing # Contributing
Everyone is welcome to help out and contribute! See Everyone is welcome to help out and contribute! See
[CONTRIBUTING.md](CONTRIBUTING.md) to get started! [CONTRIBUTING.md](docs/CONTRIBUTING.md) to get started!
Please note that, as of February 2020, Dendrite now only targets Go 1.13 or Please note that, as of February 2020, Dendrite now only targets Go 1.13 or
later. Please ensure that you are using at least Go 1.13 when developing for later. Please ensure that you are using at least Go 1.13 when developing for

View File

@ -2,38 +2,66 @@
Dendrite can be run in one of two configurations: Dendrite can be run in one of two configurations:
* A cluster of individual components, dealing with different aspects of the * **Polylith mode**: A cluster of individual components, dealing with different
Matrix protocol (see [WIRING.md](./WIRING.md)). Components communicate with aspects of the Matrix protocol (see [WIRING.md](WIRING.md)). Components communicate with each other using internal HTTP APIs and [Apache Kafka](https://kafka.apache.org). This will almost certainly be the preferred model
one another via [Apache Kafka](https://kafka.apache.org). for large-scale deployments.
* A monolith server, in which all components run in the same process. In this * **Monolith mode**: All components run in the same process. In this mode,
configuration, Kafka can be replaced with an in-process implementation Kafka is completely optional and can instead be replaced with an in-process
called [naffka](https://github.com/matrix-org/naffka). lightweight implementation called [Naffka](https://github.com/matrix-org/naffka). This will usually be the preferred model for low-volume, low-user
or experimental deployments.
Regardless of whether you are running in polylith or monolith mode, each Dendrite component that requires storage has its own database. Both Postgres
and SQLite are supported and can be mixed-and-matched across components as
needed in the configuration file.
Be advised that Dendrite is still developmental and it's not recommended for
use in production environments yet!
## Requirements ## Requirements
- Go 1.13+ Dendrite requires:
- Postgres 9.5+
- For Kafka (optional if using the monolith server):
- Unix-based system (https://kafka.apache.org/documentation/#os)
- JDK 1.8+ / OpenJDK 1.8+
- Apache Kafka 0.10.2+ (see [scripts/install-local-kafka.sh](scripts/install-local-kafka.sh) for up-to-date version numbers)
* Go 1.13 or higher
* Postgres 9.5 or higher (if using Postgres databases, not needed for SQLite)
## Setting up a development environment If you want to run a polylith deployment, you also need:
Assumes Go 1.13+ and JDK 1.8+ are already installed and are on PATH. * Apache Kafka 0.10.2+
## Building up a monolith deploment
Start by cloning the code:
```bash ```bash
# Get the code
git clone https://github.com/matrix-org/dendrite git clone https://github.com/matrix-org/dendrite
cd dendrite cd dendrite
```
# Build it Then build it:
```bash
go build -o bin/dendrite-monolith-server ./cmd/dendrite-monolith-server
go build -o bin/generate-keys ./cmd/generate-keys
```
## Building up a polylith deployment
Start by cloning the code:
```bash
git clone https://github.com/matrix-org/dendrite
cd dendrite
```
Then build it:
```bash
./build.sh ./build.sh
``` ```
If using Kafka, install and start it (c.f. [scripts/install-local-kafka.sh](scripts/install-local-kafka.sh)): Install and start Kafka (c.f. [scripts/install-local-kafka.sh](scripts/install-local-kafka.sh)):
```bash ```bash
KAFKA_URL=http://archive.apache.org/dist/kafka/2.1.0/kafka_2.11-2.1.0.tgz KAFKA_URL=http://archive.apache.org/dist/kafka/2.1.0/kafka_2.11-2.1.0.tgz
@ -51,7 +79,7 @@ kafka/bin/zookeeper-server-start.sh -daemon kafka/config/zookeeper.properties
kafka/bin/kafka-server-start.sh -daemon kafka/config/server.properties kafka/bin/kafka-server-start.sh -daemon kafka/config/server.properties
``` ```
On MacOS, you can use [homebrew](https://brew.sh/) for easier setup of kafka On macOS, you can use [Homebrew](https://brew.sh/) for easier setup of Kafka:
```bash ```bash
brew install kafka brew install kafka
@ -61,15 +89,24 @@ brew services start kafka
## Configuration ## Configuration
### SQLite database setup
Dendrite can use the built-in SQLite database engine for small setups.
The SQLite databases do not need to be preconfigured - Dendrite will
create them automatically at startup.
### Postgres database setup ### Postgres database setup
Dendrite requires a postgres database engine, version 9.5 or later. Assuming that Postgres 9.5 (or later) is installed:
* Create role, choosing a new password when prompted:
* Create role:
```bash ```bash
sudo -u postgres createuser -P dendrite # prompts for password sudo -u postgres createuser -P dendrite
``` ```
* Create databases:
* Create the component databases:
```bash ```bash
for i in account device mediaapi syncapi roomserver serverkey federationsender publicroomsapi appservice naffka; do for i in account device mediaapi syncapi roomserver serverkey federationsender publicroomsapi appservice naffka; do
sudo -u postgres createdb -O dendrite dendrite_$i sudo -u postgres createdb -O dendrite dendrite_$i
@ -78,42 +115,56 @@ Dendrite requires a postgres database engine, version 9.5 or later.
(On macOS, omit `sudo -u postgres` from the above commands.) (On macOS, omit `sudo -u postgres` from the above commands.)
### Crypto key generation ### Server key generation
Generate the keys: Each Dendrite server requires unique server keys.
Generate the self-signed SSL certificate for federation:
```bash ```bash
# Generate a self-signed SSL cert for federation:
test -f server.key || openssl req -x509 -newkey rsa:4096 -keyout server.key -out server.crt -days 3650 -nodes -subj /CN=localhost test -f server.key || openssl req -x509 -newkey rsa:4096 -keyout server.key -out server.crt -days 3650 -nodes -subj /CN=localhost
```
# generate ed25519 signing key Generate the server signing key:
```
test -f matrix_key.pem || ./bin/generate-keys -private-key matrix_key.pem test -f matrix_key.pem || ./bin/generate-keys -private-key matrix_key.pem
``` ```
### Configuration ### Configuration file
Create config file, based on `dendrite-config.yaml`. Call it `dendrite.yaml`. Things that will need editing include *at least*: Create config file, based on `dendrite-config.yaml`. Call it `dendrite.yaml`. Things that will need editing include *at least*:
* `server_name`
* `database/*` (All lines in the database section must have the username and password of the user created with the `createuser` command above. eg:`dendrite:password@localhost`)
* The `server_name` entry to reflect the hostname of your Dendrite server
* The `database` lines with an updated connection string based on your
desired setup, e.g. replacing `component` with the name of the component:
* For Postgres: `postgres://dendrite:password@localhost/component`
* For SQLite on disk: `file:component.db` or `file:///path/to/component.db`
* Postgres and SQLite can be mixed and matched.
* The `use_naffka` option if using Naffka in a monolith deployment
There are other options which may be useful so review them all. In particular,
if you are trying to federate from your Dendrite instance into public rooms
then configuring `key_perspectives` (like `matrix.org` in the sample) can
help to improve reliability considerably by allowing your homeserver to fetch
public keys for dead homeservers from somewhere else.
## Starting a monolith server ## Starting a monolith server
It is possible to use 'naffka' as an in-process replacement to Kafka when using It is possible to use Naffka as an in-process replacement to Kafka when using
the monolith server. To do this, set `use_naffka: true` in `dendrite.yaml` and uncomment the monolith server. To do this, set `use_naffka: true` in your `dendrite.yaml` configuration and uncomment the relevant Naffka line in the `database` section.
the necessary line related to naffka in the `database` section. Be sure to update the Be sure to update the database username and password if needed.
database username and password if needed.
The monolith server can be started as shown below. By default it listens for The monolith server can be started as shown below. By default it listens for
HTTP connections on port 8008, so point your client at HTTP connections on port 8008, so you can configure your Matrix client to use
`http://localhost:8008`. If you set `--tls-cert` and `--tls-key` as shown `http://localhost:8008` as the server. If you set `--tls-cert` and `--tls-key`
below, it will also listen for HTTPS connections on port 8448. as shown below, it will also listen for HTTPS connections on port 8448.
```bash ```bash
./bin/dendrite-monolith-server --tls-cert=server.crt --tls-key=server.key ./bin/dendrite-monolith-server --tls-cert=server.crt --tls-key=server.key
``` ```
## Starting a multiprocess server ## Starting a polylith deployment
The following contains scripts which will run all the required processes in order to point a Matrix client at Dendrite. Conceptually, you are wiring together to form the following diagram: The following contains scripts which will run all the required processes in order to point a Matrix client at Dendrite. Conceptually, you are wiring together to form the following diagram:
@ -170,9 +221,10 @@ Servers --->| federation-api-proxy |--------->| dendrite-federation-api-server |
A ==> B = Kafka (A = producer, B = consumer) A ==> B = Kafka (A = producer, B = consumer)
``` ```
### Run a client api proxy ### Client proxy
This is what Matrix clients will talk to. If you use the script below, point your client at `http://localhost:8008`. This is what Matrix clients will talk to. If you use the script below, point
your client at `http://localhost:8008`.
```bash ```bash
./bin/client-api-proxy \ ./bin/client-api-proxy \
@ -183,51 +235,10 @@ This is what Matrix clients will talk to. If you use the script below, point you
--public-rooms-api-server-url "http://localhost:7775" \ --public-rooms-api-server-url "http://localhost:7775" \
``` ```
### Run a client api ### Federation proxy
This is what implements message sending. Clients talk to this via the proxy in order to send messages. This is what Matrix servers will talk to. This is only required if you want
to support federation.
```bash
./bin/dendrite-client-api-server --config=dendrite.yaml
```
(If this fails with `pq: syntax error at or near "ON"`, check you are using at least postgres 9.5.)
### Run a room server
This is what implements the room DAG. Clients do not talk to this.
```bash
./bin/dendrite-room-server --config=dendrite.yaml
```
### Run a sync server
This is what implements `/sync` requests. Clients talk to this via the proxy in order to receive messages.
```bash
./bin/dendrite-sync-api-server --config dendrite.yaml
```
### Run a media server
This implements `/media` requests. Clients talk to this via the proxy in order to upload and retrieve media.
```bash
./bin/dendrite-media-api-server --config dendrite.yaml
```
### Run public room server
This implements `/directory` requests. Clients talk to this via the proxy in order to retrieve room directory listings.
```bash
./bin/dendrite-public-rooms-api-server --config dendrite.yaml
```
### Run a federation api proxy
This is what Matrix servers will talk to. This is only required if you want to support federation.
```bash ```bash
./bin/federation-api-proxy \ ./bin/federation-api-proxy \
@ -236,7 +247,51 @@ This is what Matrix servers will talk to. This is only required if you want to s
--media-api-server-url "http://localhost:7774" \ --media-api-server-url "http://localhost:7774" \
``` ```
### Run a federation api server ### Client API server
This is what implements message sending. Clients talk to this via the proxy in
order to send messages.
```bash
./bin/dendrite-client-api-server --config=dendrite.yaml
```
### Room server
This is what implements the room DAG. Clients do not talk to this.
```bash
./bin/dendrite-room-server --config=dendrite.yaml
```
### Sync server
This is what implements `/sync` requests. Clients talk to this via the proxy
in order to receive messages.
```bash
./bin/dendrite-sync-api-server --config dendrite.yaml
```
### Media server
This implements `/media` requests. Clients talk to this via the proxy in
order to upload and retrieve media.
```bash
./bin/dendrite-media-api-server --config dendrite.yaml
```
### Public room server
This implements `/directory` requests. Clients talk to this via the proxy
in order to retrieve room directory listings.
```bash
./bin/dendrite-public-rooms-api-server --config dendrite.yaml
```
### Federation API server
This implements federation requests. Servers talk to this via the proxy in This implements federation requests. Servers talk to this via the proxy in
order to send transactions. This is only required if you want to support order to send transactions. This is only required if you want to support
@ -246,7 +301,7 @@ federation.
./bin/dendrite-federation-api-server --config dendrite.yaml ./bin/dendrite-federation-api-server --config dendrite.yaml
``` ```
### Run a federation sender server ### Federation sender
This sends events from our users to other servers. This is only required if This sends events from our users to other servers. This is only required if
you want to support federation. you want to support federation.
@ -255,7 +310,7 @@ you want to support federation.
./bin/dendrite-federation-sender-server --config dendrite.yaml ./bin/dendrite-federation-sender-server --config dendrite.yaml
``` ```
### Run an appservice server ### Appservice server
This sends events from the network to [application This sends events from the network to [application
services](https://matrix.org/docs/spec/application_service/unstable.html) services](https://matrix.org/docs/spec/application_service/unstable.html)
@ -265,3 +320,12 @@ application services on your homeserver.
```bash ```bash
./bin/dendrite-appservice-server --config dendrite.yaml ./bin/dendrite-appservice-server --config dendrite.yaml
``` ```
### Key server
This manages end-to-end encryption keys (or rather, it will do when it's
finished).
```bash
./bin/dendrite-key-server --config dendrite.yaml
```

View File

View File

@ -1,5 +0,0 @@
#! /bin/bash
DOT_GIT="$(dirname $0)/../.git"
ln -s "../../hooks/pre-commit" "$DOT_GIT/hooks/pre-commit"

View File

@ -1,22 +0,0 @@
#! /bin/bash
set -eu
# make the GIT_DIR and GIT_INDEX_FILE absolute, before we change dir
export GIT_DIR=$(readlink -f `git rev-parse --git-dir`)
if [ -n "${GIT_INDEX_FILE:+x}" ]; then
export GIT_INDEX_FILE=$(readlink -f "$GIT_INDEX_FILE")
fi
# create a temp dir. The `trap` incantation will ensure that it is removed
# again when this script completes.
tmpdir=`mktemp -d`
trap 'rm -rf "$tmpdir"' EXIT
cd "$tmpdir"
# get a clean copy of the index (ie, what has been `git add`ed), so that we can
# run the checks against what we are about to commit, rather than what is in
# the working copy.
git checkout-index -a
./scripts/find-lint.sh fast