1f349/dendrite
4
0
Fork 0
mirror of https://github.com/1f349/dendrite.git synced 2024-11-21 19:21:39 +00:00
Code Issues Packages Projects Releases Wiki Activity

Nuke old Docker stuff (#1104)

Browse Source 
* Remove old Docker stuff

* Move hub stuff to upper level docker directory

* Build monolith images

* Update README.md

* Update paths

* Update README.md
This commit is contained in:
Neil Alexander 2020-06-05 19:00:30 +01:00 committed by GitHub
parent e7b19d2c70
commit 67784ecb56
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
26 changed files with 67 additions and 585 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
build/docker/Dockerfile
View File

@ -1,9 +1,10 @@
FROM docker.io/golang:1.13.7-alpine3.11
FROM docker.io/golang:1.13.7-alpine3.11 AS builder
RUN mkdir /build
RUN apk --update --no-cache add bash build-base
WORKDIR /build
RUN apk --update --no-cache add openssl bash git build-base
COPY . /build
CMD ["bash", "docker/build.sh"]
RUN mkdir -p bin
RUN sh ./build.sh

0
build/docker/hub/Dockerfile.component → build/docker/Dockerfile.component
View File

118
build/docker/README.md
View File

@ -1,100 +1,70 @@
Development with Docker
---
# Docker images
With `docker` and `docker-compose` you can easily spin up a development environment
and start working on dendrite.
These are Docker images for Dendrite!
### Requirements
## Dockerfiles
- docker
- docker-compose (version 3+)
The `Dockerfile` builds the base image which contains all of the Dendrite
components. The `Dockerfile.component` file takes the given component, as
specified with `--buildarg component=` from the base image and produce
smaller component-specific images, which are substantially smaller and do
not contain the Go toolchain etc.
### Configuration
## Compose files
Create a directory named `cfg` in the root of the project. Copy the
`dendrite-docker.yaml` file into that directory and rename it to `dendrite.yaml`.
It already contains the defaults used in `docker-compose` for networking so you will
only have to change things like the `server_name` or to toggle `naffka`.
There are three sample `docker-compose` files:
You can run the following `docker-compose` commands either from the top directory
specifying the `docker-compose` file
```
docker-compose -f docker/docker-compose.yml <cmd>
```
or from within the `docker` directory
- `docker-compose.deps.yml` which runs the Postgres and Kafka prerequisites
- `docker-compose.monolith.yml` which runs a monolith Dendrite deployment
- `docker-compose.polylith.yml` which runs a polylith Dendrite deployment
## Configuration
The `docker-compose` files refer to the `/etc/dendrite` volume as where the
runtime config should come from. The mounted folder must contain:
- `dendrite.yaml` configuration file (based on the sample `dendrite-config.yaml`
in the `docker/config` folder in the [Dendrite repository](https://github.com/matrix-org/dendrite)
- `matrix_key.pem` server key, as generated using `cmd/generate-keys`
- `server.crt` certificate file
- `server.key` private key file for the above certificate
To generate keys:
```
docker-compose <cmd>
go run github.com/matrix-org/dendrite/cmd/generate-keys \
--private-key=matrix_key.pem \
--tls-cert=server.crt \
--tls-key=server.key
```
### Starting a monolith server
## Starting Dendrite
For the monolith server you would need a postgres instance
Once in place, start the dependencies:
```
docker-compose up postgres
docker-compose -f docker-compose.deps.yml up
```
and the dendrite component from `bin/dendrite-monolith-server`
Wait a few seconds for Kafka and Postgres to finish starting up, and then start a monolith:
```
docker-compose up monolith
docker-compose -f docker-compose.monolith.yml up
```
The monolith will be listening on `http://localhost:8008`.
You would also have to make the following adjustments to `dendrite.yaml`.
- Set `use_naffka: true`
- Uncomment the `database/naffka` postgres url.
### Starting a multiprocess server
The multiprocess server requires kafka, zookeeper and postgres
... or start the polylith components:
```
docker-compose up kafka zookeeper postgres
docker-compose -f docker-compose.polylith.yml up
```
and the following dendrite components
## Building the images
```
docker-compose up client_api media_api sync_api room_server public_rooms_api edu_server
docker-compose up client_api_proxy
```
The `docker/images-build.sh` script will build the base image, followed by
all of the component images.
The `client-api-proxy` will be listening on `http://localhost:8008`.
The `docker/images-push.sh` script will push them to Docker Hub (subject
to permissions).
You would also have to make the following adjustments to `dendrite.yaml`.
- Set `use_naffka: false`
- Comment out the `database/naffka` postgres url.
### Starting federation
```
docker-compose up federation_api federation_sender
docker-compose up federation_api_proxy
```
You can point other Matrix servers to `http://localhost:8448`.
### Creating a new component
You can create a new dendrite component by adding an entry to the `docker-compose.yml`
file and creating a startup script for the component in `docker/services`.
For more information refer to the official docker-compose [documentation](https://docs.docker.com/compose/).
```yaml
new_component:
container_name: dendrite_room_server
hostname: new_component
# Start up script.
entrypoint: ["bash", "./docker/services/new-component.sh"]
# Use the common Dockerfile for all the dendrite components.
build: ./
volumes:
- ..:/build
depends_on:
- another_component
networks:
- internal
```
If you wish to build and push your own images, rename `matrixdotorg/dendrite` to
the name of another Docker Hub repository in `images-build.sh` and `images-push.sh`.

18
build/docker/build.sh
View File

@ -1,18 +0,0 @@
#!/bin/bash
./build.sh
# Generate the keys if they don't already exist.
if [ ! -f server.key ] || [ ! -f server.crt ] || [ ! -f matrix_key.pem ]; then
echo "Generating keys ..."
rm -f server.key server.crt matrix_key.pem
test -f server.key || openssl req -x509 -newkey rsa:4096 \
-keyout server.key \
-out server.crt \
-days 3650 -nodes \
-subj /CN=localhost
test -f matrix_key.pem || /build/bin/generate-keys -private-key matrix_key.pem
fi

0
build/docker/hub/config/dendrite-config.yaml → build/docker/config/dendrite-config.yaml
View File

129
build/docker/dendrite-docker.yml
View File

@ -1,129 +0,0 @@
# The config file format version
# This is used by dendrite to tell if it understands the config format.
# This will change if the structure of the config file changes or if the meaning
# of an existing config key changes.
version: 0
# The matrix specific config
matrix:
# The name of the server. This is usually the domain name, e.g 'matrix.org', 'localhost'.
server_name: "example.com"
# The path to the PEM formatted matrix private key.
private_key: "matrix_key.pem"
# The x509 certificates used by the federation listeners for this server
federation_certificates: ["server.crt"]
# The list of identity servers trusted to verify third party identifiers by this server.
# Defaults to no trusted servers.
trusted_third_party_id_servers:
- vector.im
- matrix.org
# The media repository config
media:
# The base path to where the media files will be stored. May be relative or absolute.
base_path: /var/dendrite/media
# The maximum file size in bytes that is allowed to be stored on this server.
# Note: if max_file_size_bytes is set to 0, the size is unlimited.
# Note: if max_file_size_bytes is not set, it will default to 10485760 (10MB)
max_file_size_bytes: 10485760
# Whether to dynamically generate thumbnails on-the-fly if the requested resolution is not already generated
# NOTE: This is a possible denial-of-service attack vector - use at your own risk
dynamic_thumbnails: false
# A list of thumbnail sizes to be pre-generated for downloaded remote / uploaded content
# method is one of crop or scale. If omitted, it will default to scale.
# crop scales to fill the requested dimensions and crops the excess.
# scale scales to fit the requested dimensions and one dimension may be smaller than requested.
thumbnail_sizes:
- width: 32
height: 32
method: crop
- width: 96
height: 96
method: crop
- width: 320
height: 240
method: scale
- width: 640
height: 480
method: scale
- width: 800
height: 600
method: scale
# The config for the TURN server
turn:
# Whether or not guests can request TURN credentials
turn_allow_guests: true
# How long the authorization should last
turn_user_lifetime: "1h"
# The list of TURN URIs to pass to clients
turn_uris: []
# Authorization via Shared Secret
# The shared secret from coturn
turn_shared_secret: "<SECRET STRING GOES HERE>"
# Authorization via Static Username & Password
# Hardcoded Username and Password
turn_username: ""
turn_password: ""
# The config for communicating with kafka
kafka:
# Where the kafka servers are running.
addresses: ["kafka:9092"]
# Whether to use naffka instead of kafka.
# Naffka can only be used when running dendrite as a single monolithic server.
# Kafka can be used both with a monolithic server and when running the
# components as separate servers.
# If enabled database.naffka must also be specified.
use_naffka: true
# The names of the kafka topics to use.
topics:
output_room_event: roomserverOutput
output_client_data: clientapiOutput
output_typing_event: eduServerOutput
user_updates: userUpdates
# The postgres connection configs for connecting to the databases e.g a postgres:// URI
database:
account: "postgres://dendrite:itsasecret@postgres/dendrite_account?sslmode=disable"
device: "postgres://dendrite:itsasecret@postgres/dendrite_device?sslmode=disable"
media_api: "postgres://dendrite:itsasecret@postgres/dendrite_mediaapi?sslmode=disable"
sync_api: "postgres://dendrite:itsasecret@postgres/dendrite_syncapi?sslmode=disable"
room_server: "postgres://dendrite:itsasecret@postgres/dendrite_roomserver?sslmode=disable"
server_key: "postgres://dendrite:itsasecret@postgres/dendrite_serverkey?sslmode=disable"
federation_sender: "postgres://dendrite:itsasecret@postgres/dendrite_federationsender?sslmode=disable"
public_rooms_api: "postgres://dendrite:itsasecret@postgres/dendrite_publicroomsapi?sslmode=disable"
appservice: "postgres://dendrite:itsasecret@postgres/dendrite_appservice?sslmode=disable"
# If using naffka you need to specify a naffka database
naffka: "postgres://dendrite:itsasecret@postgres/dendrite_naffka?sslmode=disable"
# The TCP host:port pairs to bind the internal HTTP APIs to.
# These shouldn't be exposed to the public internet.
# These aren't needed when running dendrite as a monolithic server.
listen:
room_server: "room_server:7770"
client_api: "client_api:7771"
federation_api: "federation_api:7772"
sync_api: "sync_api:7773"
media_api: "media_api:7774"
public_rooms_api: "public_rooms_api:7775"
federation_sender: "federation_sender:7776"
edu_server: "typing_server:7777"
# The configuration for tracing the dendrite components.
tracing:
# Config for the jaeger opentracing reporter.
# See https://godoc.org/github.com/uber/jaeger-client-go/config#Configuration
# for documentation.
jaeger:
disabled: true
# A list of application service config files to use
application_services:
config_files: []

2
build/docker/hub/docker-compose.deps.yml → build/docker/docker-compose.deps.yml
View File

@ -5,7 +5,7 @@ services:
image: postgres:9.5
restart: always
volumes:
- ../postgres/create_db.sh:/docker-entrypoint-initdb.d/20-create_db.sh
- ./postgres/create_db.sh:/docker-entrypoint-initdb.d/20-create_db.sh
environment:
POSTGRES_PASSWORD: itsasecret
POSTGRES_USER: dendrite

0
build/docker/hub/docker-compose.monolith.yml → build/docker/docker-compose.monolith.yml
View File

0
build/docker/hub/docker-compose.polylith.yml → build/docker/docker-compose.polylith.yml
View File

202
build/docker/docker-compose.yml
View File

@ -1,202 +0,0 @@
version: "3.4"
services:
riot:
image: vectorim/riot-web
networks:
- internal
ports:
- "8500:80"
monolith:
container_name: dendrite_monolith
hostname: monolith
entrypoint: ["bash", "./docker/services/monolith.sh", "--config", "/etc/dendrite/dendrite.yaml"]
build: ./
volumes:
- ..:/build
- ./build/bin:/build/bin
- ../cfg:/etc/dendrite
networks:
- internal
depends_on:
- postgres
ports:
- "8008:8008"
- "8448:8448"
client_api_proxy:
container_name: dendrite_client_api_proxy
hostname: client_api_proxy
entrypoint: ["bash", "./docker/services/client-api-proxy.sh"]
build: ./
volumes:
- ..:/build
networks:
- internal
depends_on:
- postgres
- sync_api
- client_api
- media_api
- public_rooms_api
ports:
- "8008:8008"
client_api:
container_name: dendrite_client_api
hostname: client_api
entrypoint: ["bash", "./docker/services/client-api.sh"]
build: ./
volumes:
- ..:/build
depends_on:
- postgres
- room_server
networks:
- internal
media_api:
container_name: dendrite_media_api
hostname: media_api
entrypoint: ["bash", "./docker/services/media-api.sh"]
build: ./
volumes:
- ..:/build
depends_on:
- postgres
networks:
- internal
public_rooms_api:
container_name: dendrite_public_rooms_api
hostname: public_rooms_api
entrypoint: ["bash", "./docker/services/public-rooms-api.sh"]
build: ./
volumes:
- ..:/build
depends_on:
- postgres
networks:
- internal
sync_api:
container_name: dendrite_sync_api
hostname: sync_api
entrypoint: ["bash", "./docker/services/sync-api.sh"]
build: ./
volumes:
- ..:/build
depends_on:
- postgres
networks:
- internal
room_server:
container_name: dendrite_room_server
hostname: room_server
entrypoint: ["bash", "./docker/services/room-server.sh"]
build: ./
volumes:
- ..:/build
depends_on:
- postgres
networks:
- internal
edu_server:
container_name: dendrite_edu_server
hostname: edu_server
entrypoint: ["bash", "./docker/services/edu-server.sh"]
build: ./
volumes:
- ..:/build
networks:
- internal
federation_api_proxy:
container_name: dendrite_federation_api_proxy
hostname: federation_api_proxy
entrypoint: ["bash", "./docker/services/federation-api-proxy.sh"]
build: ./
volumes:
- ..:/build
depends_on:
- postgres
- federation_api
- federation_sender
- media_api
networks:
- internal
ports:
- "8448:8448"
federation_api:
container_name: dendrite_federation_api
hostname: federation_api
entrypoint: ["bash", "./docker/services/federation-api.sh"]
build: ./
volumes:
- ..:/build
depends_on:
- postgres
networks:
- internal
federation_sender:
container_name: dendrite_federation_sender
hostname: federation_sender
entrypoint: ["bash", "./docker/services/federation-sender.sh"]
build: ./
volumes:
- ..:/build
depends_on:
- postgres
networks:
- internal
key_server:
container_name: dendrite_key_server
hostname: key_server
entrypoint: ["bash", "./docker/services/key-server.sh"]
build: ./
volumes:
- ..:/build
networks:
- internal
postgres:
container_name: dendrite_postgres
hostname: postgres
image: postgres:9.5
restart: always
volumes:
- ./postgres/create_db.sh:/docker-entrypoint-initdb.d/20-create_db.sh
environment:
POSTGRES_PASSWORD: itsasecret
POSTGRES_USER: dendrite
networks:
- internal
zookeeper:
container_name: dendrite_zk
hostname: zookeeper
image: zookeeper
networks:
- internal
kafka:
container_name: dendrite_kafka
hostname: kafka
image: wurstmeister/kafka
environment:
KAFKA_ADVERTISED_HOST_NAME: "kafka"
KAFKA_DELETE_TOPIC_ENABLE: "true"
KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
depends_on:
- zookeeper
networks:
- internal
networks:
internal:
attachable: true

10
build/docker/hub/Dockerfile
View File

@ -1,10 +0,0 @@
FROM docker.io/golang:1.13.7-alpine3.11 AS builder
RUN apk --update --no-cache add bash build-base
WORKDIR /build
COPY . /build
RUN mkdir -p bin
RUN sh ./build.sh

68
build/docker/hub/README.md
View File

@ -1,68 +0,0 @@
# Docker Hub images
These are Docker Hub-friendly images for Dendrite.
## Dockerfiles
The `Dockerfile` builds the base image which contains all of the Dendrite
components. The `Dockerfile.*` files take components from the base image and
produce smaller component-specific images, which are substantially smaller
and do not contain the Go toolchain etc.
## Compose files
There are three sample `docker-compose` files:
- `docker-compose.deps.yml` which runs the Postgres and Kafka prerequisites
- `docker-compose.monolith.yml` which runs a monolith Dendrite deployment
- `docker-compose.polylith.yml` which runs a polylith Dendrite deployment
## Configuration
The `docker-compose` files refer to the `/etc/dendrite` volume as where the
runtime config should come from. The mounted folder must contain:
- `dendrite.yaml` configuration file (based on the sample `dendrite-config.yaml`
in the `docker/hub/config` folder in the [Dendrite repository](https://github.com/matrix-org/dendrite)
- `matrix_key.pem` server key, as generated using `cmd/generate-keys`
- `server.crt` certificate file
- `server.key` private key file for the above certificate
To generate keys:
```
go run github.com/matrix-org/dendrite/cmd/generate-keys \
--private-key=matrix_key.pem \
--tls-cert=server.crt \
--tls-key=server.key
```
## Starting Dendrite
Once in place, start the dependencies:
```
docker-compose -f docker-compose.deps.yml up
```
Wait a few seconds for Kafka and Postgres to finish starting up, and then start a monolith:
```
docker-compose -f docker-compose.monolith.yml up
```
... or start the polylith components:
```
docker-compose -f docker-compose.polylith.yml up
```
## Building the images
The `docker/hub/images-build.sh` script will build all of the component images.
The `docker/hub/images-push.sh` script will push them to Docker Hub (subject
to permissions).
If you wish to build and push your own images, rename `matrixdotorg/dendrite` to
the name of another Docker Hub repository in `images-build.sh` and `images-push.sh`.

28
build/docker/hub/images-build.sh → build/docker/images-build.sh
View File

@ -2,17 +2,19 @@
cd $(git rev-parse --show-toplevel)
docker build -f build/docker/hub/Dockerfile -t matrixdotorg/dendrite:latest .
docker build -f build/docker/Dockerfile -t matrixdotorg/dendrite:latest .
docker build -t matrixdotorg/dendrite:clientapi --build-arg component=dendrite-client-api-server -f build/docker/hub/Dockerfile.component .
docker build -t matrixdotorg/dendrite:clientproxy --build-arg component=client-api-proxy -f build/docker/hub/Dockerfile.component .
docker build -t matrixdotorg/dendrite:eduserver --build-arg component=dendrite-edu-server -f build/docker/hub/Dockerfile.component .
docker build -t matrixdotorg/dendrite:federationapi --build-arg component=dendrite-federation-api-server -f build/docker/hub/Dockerfile.component .
docker build -t matrixdotorg/dendrite:federationsender --build-arg component=dendrite-federation-sender-server -f build/docker/hub/Dockerfile.component .
docker build -t matrixdotorg/dendrite:federationproxy --build-arg component=federation-api-proxy -f build/docker/hub/Dockerfile.component .
docker build -t matrixdotorg/dendrite:keyserver --build-arg component=dendrite-key-server -f build/docker/hub/Dockerfile.component .
docker build -t matrixdotorg/dendrite:mediaapi --build-arg component=dendrite-media-api-server -f build/docker/hub/Dockerfile.component .
docker build -t matrixdotorg/dendrite:publicroomsapi --build-arg component=dendrite-public-rooms-api-server -f build/docker/hub/Dockerfile.component .
docker build -t matrixdotorg/dendrite:roomserver --build-arg component=dendrite-room-server -f build/docker/hub/Dockerfile.component .
docker build -t matrixdotorg/dendrite:syncapi --build-arg component=dendrite-sync-api-server -f build/docker/hub/Dockerfile.component .
docker build -t matrixdotorg/dendrite:serverkeyapi --build-arg component=dendrite-server-key-api-server -f build/docker/hub/Dockerfile.component .
docker build -t matrixdotorg/dendrite:monolith --build-arg component=dendrite-monolith-server -f build/docker/Dockerfile.component .
docker build -t matrixdotorg/dendrite:clientapi --build-arg component=dendrite-client-api-server -f build/docker/Dockerfile.component .
docker build -t matrixdotorg/dendrite:clientproxy --build-arg component=client-api-proxy -f build/docker/Dockerfile.component .
docker build -t matrixdotorg/dendrite:eduserver --build-arg component=dendrite-edu-server -f build/docker/Dockerfile.component .
docker build -t matrixdotorg/dendrite:federationapi --build-arg component=dendrite-federation-api-server -f build/docker/Dockerfile.component .
docker build -t matrixdotorg/dendrite:federationsender --build-arg component=dendrite-federation-sender-server -f build/docker/Dockerfile.component .
docker build -t matrixdotorg/dendrite:federationproxy --build-arg component=federation-api-proxy -f build/docker/Dockerfile.component .
docker build -t matrixdotorg/dendrite:keyserver --build-arg component=dendrite-key-server -f build/docker/Dockerfile.component .
docker build -t matrixdotorg/dendrite:mediaapi --build-arg component=dendrite-media-api-server -f build/docker/Dockerfile.component .
docker build -t matrixdotorg/dendrite:publicroomsapi --build-arg component=dendrite-public-rooms-api-server -f build/docker/Dockerfile.component .
docker build -t matrixdotorg/dendrite:roomserver --build-arg component=dendrite-room-server -f build/docker/Dockerfile.component .
docker build -t matrixdotorg/dendrite:syncapi --build-arg component=dendrite-sync-api-server -f build/docker/Dockerfile.component .
docker build -t matrixdotorg/dendrite:serverkeyapi --build-arg component=dendrite-server-key-api-server -f build/docker/Dockerfile.component .

2
build/docker/hub/images-push.sh → build/docker/images-push.sh
View File

@ -1,5 +1,7 @@
#!/bin/bash
docker push matrixdotorg/dendrite:monolith
docker push matrixdotorg/dendrite:clientapi
docker push matrixdotorg/dendrite:clientproxy
docker push matrixdotorg/dendrite:eduserver

9
build/docker/services/client-api-proxy.sh
View File

@ -1,9 +0,0 @@
#!/bin/bash
bash ./docker/build.sh
./bin/client-api-proxy --bind-address ":8008" \
--client-api-server-url "http://client_api:7771" \
--sync-api-server-url "http://sync_api:7773" \
--media-api-server-url "http://media_api:7774" \
--public-rooms-api-server-url "http://public_rooms_api:7775" \

5
build/docker/services/client-api.sh
View File

@ -1,5 +0,0 @@
#!/bin/bash
bash ./docker/build.sh
./bin/dendrite-client-api-server --config=dendrite.yaml

5
build/docker/services/edu-server.sh
View File

@ -1,5 +0,0 @@
#!/bin/bash
bash ./docker/build.sh
./bin/dendrite-edu-server --config=dendrite.yaml

7
build/docker/services/federation-api-proxy.sh
View File

@ -1,7 +0,0 @@
#!/bin/bash
bash ./docker/build.sh
./bin/federation-api-proxy --bind-address ":8448" \
--federation-api-url "http://federation_api_server:7772" \
--media-api-server-url "http://media_api:7774" \

5
build/docker/services/federation-api.sh
View File

@ -1,5 +0,0 @@
#!/bin/bash
bash ./docker/build.sh
./bin/dendrite-federation-api-server --config dendrite.yaml

5
build/docker/services/federation-sender.sh
View File

@ -1,5 +0,0 @@
#!/bin/bash
bash ./docker/build.sh
./bin/dendrite-federation-sender-server --config dendrite.yaml

5
build/docker/services/key-server.sh
View File

@ -1,5 +0,0 @@
#!/bin/bash
bash ./docker/build.sh
./bin/dendrite-key-server --config dendrite.yaml

5
build/docker/services/media-api.sh
View File

@ -1,5 +0,0 @@
#!/bin/bash
bash ./docker/build.sh
./bin/dendrite-media-api-server --config dendrite.yaml

5
build/docker/services/monolith.sh
View File

@ -1,5 +0,0 @@
#!/bin/bash
bash ./docker/build.sh
./bin/dendrite-monolith-server --tls-cert=server.crt --tls-key=server.key $@

5
build/docker/services/public-rooms-api.sh
View File

@ -1,5 +0,0 @@
#!/bin/bash
bash ./docker/build.sh
./bin/dendrite-public-rooms-api-server --config dendrite.yaml

5
build/docker/services/room-server.sh
View File

@ -1,5 +0,0 @@
#!/bin/bash
bash ./docker/build.sh
./bin/dendrite-room-server --config=dendrite.yaml

5
build/docker/services/sync-api.sh
View File

@ -1,5 +0,0 @@
#!/bin/bash
bash ./docker/build.sh
./bin/dendrite-sync-api-server --config=dendrite.yaml