Commit Graph

43 Commits

Author SHA1 Message Date
Dan
f05ce478f0
Implement Push Notifications (#1842)
* Add Pushserver component with Pushers API

Co-authored-by: Tommie Gannert <tommie@gannert.se>
Co-authored-by: Dan Peleg <dan@globekeeper.com>

* Wire Pushserver component

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>

* Add PushGatewayClient.

The full event format is required for Sytest.

* Add a pushrules module.

* Change user API account creation to use the new pushrules module's defaults.

Introduces "scope" as required by client API, and some small field
tweaks to make some 61push Sytests pass.

* Add push rules query/put API in Pushserver.

This manipulates account data over User API, and fires sync messages
for changes. Those sync messages should, according to an existing TODO
in clientapi, be moved to userapi.

Forks clientapi/producers/syncapi.go to pushserver/ for later extension.

* Add clientapi routes for push rules to Pushserver.

A cleanup would be to move more of the name-splitting logic into
pushrules.go, to depollute routing.go.

* Output rooms.join.unread_notifications in /sync.

This is the read-side. Pushserver will be the write-side.

* Implement pushserver/storage for notifications.

* Use PushGatewayClient and the pushrules module in Pushserver's room consumer.

* Use one goroutine per user to avoid locking up the entire server for
  one bad push gateway.
* Split pushing by format.
* Send one device per push. Sytest does not support coalescing
  multiple devices into one push. Matches Synapse. Either we change
  Sytest, or remove the group-by-url-and-format logic.
* Write OutputNotificationData from push server. Sync API is already
  the consumer.

* Implement read receipt consumers in Pushserver.

Supports m.read and m.fully_read receipts.

* Add clientapi route for /unstable/notifications.

* Rename to UpsertPusher for clarity and handle pusher update

* Fix linter errors

* Ignore body.Close() error check

* Fix push server internal http wiring

* Add 40 newly passing 61push tests to whitelist

* Add next 12 newly passing 61push tests to whitelist

* Send notification data before notifying users in EDU server consumer

* NATS JetStream

* Goodbye sarama

* Fix `NewStreamTokenFromString`

* Consume on the correct topic for the roomserver

* Don't panic, NAK instead

* Move push notifications into the User API

* Don't set null values since that apparently causes Element upsetti

* Also set omitempty on conditions

* Fix bug so that we don't override the push rules unnecessarily

* Tweak defaults

* Update defaults

* More tweaks

* Move `/notifications` onto `r0`/`v3` mux

* User API will consume events and read/fully read markers from the sync API with stream positions, instead of consuming directly

Co-authored-by: Piotr Kozimor <p1996k@gmail.com>
Co-authored-by: Tommie Gannert <tommie@gannert.se>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-03-03 11:40:53 +00:00
Neil Alexander
bb39149ff8
Fix DendriteJS dockerfile 2022-02-08 16:18:16 +00:00
S7evinK
161f145176
Add NATS JetStream support (#1866)
* Add NATS JetStream support
Update shopify/sarama

* Fix addresses

* Don't change Addresses in Defaults

* Update saramajetstream

* Add missing error check

Keep typing events for at least one minute

* Use all configured NATS addresses

* Update saramajetstream

* Try setting up with NATS

* Make sure NATS uses own persistent directory (TODO: make this configurable)

* Update go.mod/go.sum

* Jetstream package

* Various other refactoring

* Build fixes

* Config tweaks, make random jetstream storage path for CI

* Disable interest policies

* Try to sane default on jetstream base path

* Try to use in-memory for CI

* Restore storage/retention

* Update nats.go dependency

* Adapt changes to config

* Remove unneeded TopicFor

* Dep update

* Revert "Remove unneeded TopicFor"

This reverts commit f5a4e4a339b6f94ec215778dca22204adaa893d1.

* Revert changes made to streams

* Fix build problems

* Update nats-server

* Update go.mod/go.sum

* Roomserver input API queuing using NATS

* Fix topic naming

* Prometheus metrics

* More refactoring to remove saramajetstream

* Add missing topic

* Don't try to populate map that doesn't exist

* Roomserver output topic

* Update go.mod/go.sum

* Message acknowledgements

* Ack tweaks

* Try to resume transaction re-sends

* Try to resume transaction re-sends

* Update to matrix-org/gomatrixserverlib@91dadfb

* Remove internal.PartitionStorer from components that don't consume keychanges

* Try to reduce re-allocations a bit in resolveConflictsV2

* Tweak delivery options on RS input

* Publish send-to-device messages into correct JetStream subject

* Async and sync roomserver input

* Update dendrite-config.yaml

* Remove roomserver tests for now (they need rewriting)

* Remove roomserver test again (was merged back in)

* Update documentation

* Docker updates

* More Docker updates

* Update Docker readme again

* Fix lint issues

* Send final event in `processEvent` synchronously (since this might stop Sytest from being so upset)

* Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that

* Go 1.16 instead of Go 1.13 for upgrade tests and Complement

* Revert "Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that"

This reverts commit 368675283fc44501f227639811bdb16dd5deef8c.

* Don't report any errors on `/send` to see what fun that creates

* Fix panics on closed channel sends

* Enforce state key matches sender

* Do the same for leave

* Various tweaks to make tests happier

Squashed commit of the following:

commit 13f9028e7a63662759ce7c55504a9d2423058668
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 15:47:14 2022 +0000

    Do the same for leave

commit e6be7f05c349fafbdddfe818337a17a60c867be1
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 15:33:42 2022 +0000

    Enforce state key matches sender

commit 85ede6d64bf10ce9b91cdd6d80f87350ee55242f
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 14:07:04 2022 +0000

    Fix panics on closed channel sends

commit 9755494a98bed62450f8001d8128e40481d27e15
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 13:38:22 2022 +0000

    Don't report any errors on `/send` to see what fun that creates

commit 3bb4f87b5dd56882febb4db5621db484c8789b7c
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 13:00:26 2022 +0000

    Revert "Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that"

    This reverts commit 368675283fc44501f227639811bdb16dd5deef8c.

commit fe2673ed7be9559eaca134424e403a4faca100b0
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 12:09:34 2022 +0000

    Go 1.16 instead of Go 1.13 for upgrade tests and Complement

commit 368675283fc44501f227639811bdb16dd5deef8c
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 11:51:45 2022 +0000

    Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that

commit b028dfc08577bcf52e6cb498026e15fa5d46d07c
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 10:29:08 2022 +0000

    Send final event in `processEvent` synchronously (since this might stop Sytest from being so upset)

* Merge in NATS Server v2.6.6 and nats.go v1.13 into the in-process connection fork

* Add `jetstream.WithJetStreamMessage` to make ack/nak-ing less messy, use process context in consumers

* Fix consumer component name in  federation API

* Add comment explaining where streams are defined

* Tweaks to roomserver input with comments

* Finish that sentence that I apparently forgot to finish in INSTALL.md

* Bump version number of config to 2

* Add comments around asynchronous sends to roomserver in processEventWithMissingState

* More useful error message when the config version does not match

* Set version in generate-config

* Fix version in config.Defaults

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-01-05 17:44:49 +00:00
Ben Yanke
83b9cb4d79
Updating example to Postgres v14 (#2062)
See issue #2052
2021-11-29 10:17:16 +00:00
Neil Alexander
ec716793eb
Merge federationapi, federationsender, signingkeyserver components (#2055)
* Initial federation sender -> federation API refactoring

* Move base into own package, avoids import cycle

* Fix build errors

* Fix tests

* Add signing key server tables

* Try to fold signing key server into federation API

* Fix dendritejs builds

* Update embedded interfaces

* Fix panic, fix lint error

* Update configs, docker

* Rename some things

* Reuse same keyring on the implementing side

* Fix federation tests, `NewBaseDendrite` can accept freeform options

* Fix build

* Update create_db, configs

* Name tables back

* Don't rename federationsender consumer for now
2021-11-24 10:45:23 +00:00
Ryan W
1cd4d50181
Added .well-known/matrix/server endpoint (#1988)
* Added .well-known/matrix/server endpoint

Signed-off-by: Ryan Whittington <twentybitdev@gmail.com>

* Replaced tabs with spaces

Signed-off-by: Ryan Whittington <twentybitdev@gmail.com>
2021-09-10 10:05:31 +01:00
Neil Alexander
cf42ae6064
Build Docker images using Go 1.17 2021-08-18 14:13:38 +01:00
Melroy van den Berg
e48a08fef0
Propose config better (#1758)
Better explain where the config file are located and how to deal with the yml file.

Co-authored-by: kegsay <kegan@matrix.org>
2021-07-12 10:13:17 +01:00
Arnaud Venturi
9633ed747d
Remove mention of enabling naffka in install instructions, as it is the default value (#1853)
Signed-off-by: Arnaud Venturi <git@toadjaune.eu>
2021-06-14 14:09:35 +01:00
Ben Yanke
ce574f5e29
Fix Key Generation Docs - resolves #1759 (#1865)
This fixes the issue found in #1759 which broke due to go changes. The new command allows you to generate keys with docker, and drop them in the current working directory.
2021-06-14 14:08:59 +01:00
stoically
41fd15b9b6
Docker fix and tweak (#1781)
* Fix image building with recent docker version

* Mount media path in docker to prevent data loss
2021-03-03 11:20:40 +00:00
Kegan Dougal
bf9c530fdb Unbreak DendriteJS.Dockerfile; Riot is now Element 2021-02-17 15:45:50 +00:00
Marco Kundt
d1496793b9
fix database names to reflect renaming (#1636)
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-02-17 15:20:06 +00:00
David Florness
85aaaf9bb9
Fix a few indentation mistakes (#1752)
Signed-off-by: David Florness <david@florness.com>
2021-02-12 09:25:35 +00:00
Neil Alexander
805a74892e
DNS caching (#1728)
* Allow configuring DNS cache

* Update sample configs

* Fix build errors

* Fix time resolution

* Default 5m

* In seconds

* Use WithDNScache

* Correct field name

* Update go.mod/go.sum to matrix-org/gomatrixserverlib#251
2021-01-22 14:16:59 +00:00
Neil Alexander
cf82e08096
Update GHA Docker Hub builds
Squashed commit of the following:

commit 4a61aa711473deece2adf415cfd65501dbca63b2
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jan 18 12:19:24 2021 +0000

    Set back to matrixdotorg on published releases

commit 6d1ac53f2c0c9b30e1e70c0bb1559e1b8ec874a2
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jan 18 11:55:28 2021 +0000

    Rename

commit 258999f7fb7b655b3a02a06a7ea05e66fb7740fb
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jan 18 11:52:26 2021 +0000

    Refactor multi-stage builds

commit c7ab8e476939899571e7b5668860dec372b9b60f
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jan 18 11:13:19 2021 +0000

    Let's try this again
2021-01-18 12:24:23 +00:00
Jan Pechek
56a7839aed
Fixed links to Docker Hub (#1686) 2021-01-07 13:02:15 +00:00
Will Hunt
e7f2d770df
Update docker-compose.deps.yml to use postgres:11 (#1677) 2021-01-04 10:36:36 +00:00
Neil Alexander
c55361c1b8
Fix max_open_conns in Dendrite sample config 2020-12-11 10:48:24 +00:00
Neil Alexander
d8b526b603
Update docker readme 2020-11-16 11:27:59 +00:00
Neil Alexander
c5888bb64c
Update Docker sample configs 2020-10-27 13:58:58 +00:00
Neil Alexander
39c7a8915c
Multi-personality polylith binary (#1552)
* Initial work oon multipersonality binary

* Remove old binaries

* Monolith and polylith binaries

* Better logging

* dendrite-poly-multi

* Fix path

* Copyright notices etc

* Tweaks

* Update Docker, INSTALL.md

* Take first argument if flags package doesn't find any args

* Postgres 9.6 or later, fix some more Docker stuff

* Don't create unnecessary e2ekey DB

* Run go mod tidy
2020-10-20 16:11:24 +01:00
Neil Alexander
92982a402f
Update Docker (#1542)
* Separate Docker images, rather than tags

* Allow specifying tag to build/push/pull

* Include goose in Docker builds
2020-10-20 11:34:22 +01:00
Neil Alexander
49abe359e6
Start Kafka connections for each component that needs them (#1527)
* Start Kafka connection for each component that needs one

* Fix roomserver unit tests

* Rename to naffkaInstance (@Kegsay review comment)

* Fix import cycle
2020-10-15 13:27:13 +01:00
Neil Alexander
73bc28b11f
Update docker-compose.deps.yml 2020-10-13 09:54:07 +01:00
Neil Alexander
2f578531ba
Update Docker samples 2020-10-13 09:50:11 +01:00
Kegsay
bf7e85848b
Rename serverkeyapi to signingkeyserver (#1492)
* Rename serverkeyapi to signingkeyserver

We use "api" for public facing stuff and "server" for internal stuff.
As the server key API is internal only, we call it 'signing key server',
which also clarifies the type of key (as opposed to TLS keys, E2E keys, etc)

* Convert docker/scripts to use signing-key-server

* Rename missed bits
2020-10-07 16:23:18 +01:00
Neil Alexander
dbae85283f
Bring docker sample config into sync with normal one 2020-10-01 15:37:45 +01:00
Kegsay
c992f4f1f4
Remove current state server (#1405)
* Remove current state server

Closes #1365 #1272 #1357

* Remove current state server from scripts/docs
2020-09-07 14:47:59 +01:00
Neil Alexander
c1f4faf308
Fix Docker builds, polylith components (#1269) 2020-08-13 18:27:19 +01:00
Creed Haymond
f3c482b078
Adding the e2ekey database to the create_db script and docs (#1206)
Signed-off-by: Creed Haymond <creedasaurus@gmail.com>
2020-07-20 09:18:48 +01:00
Kegsay
6c4b8185d7
Implement ExtraPublicRoomsProvider for p2p demos (#1180)
* Change API and rename to ExtraPublicRoomsProvider

* Make dendritejs work again

* Maybe make libp2p demo work again

* Linting
2020-07-03 12:59:00 +01:00
Kegsay
9c1f38621c
Remove publicroomsapi (#1176)
* Remove all of publicroomsapi

* Remove references to publicroomsapi

* Remove doc references to publicroomsapi
2020-07-02 17:11:33 +01:00
Martin Honermeyer
a1352cbd9e
Push serverkeyapi docker image to registry as well (#1154)
Signed-off-by: Martin Honermeyer <maze@strahlungsfrei.de>
2020-06-23 09:51:38 +01:00
Ashley Nelson
61e0482fef
Add appservices component to docker scripts (#1153)
Signed-off-by: Ashley Nelson <amn@fantashley.com>
2020-06-20 21:28:30 +01:00
Neil Alexander
72444e4a4f
User API polylith component (#1151)
* User API polylith component

* Add docker-pull.sh
2020-06-19 09:37:19 +01:00
Neil Alexander
67784ecb56
Nuke old Docker stuff (#1104)
* Remove old Docker stuff

* Move hub stuff to upper level docker directory

* Build monolith images

* Update README.md

* Update paths

* Update README.md
2020-06-05 19:00:30 +01:00
Kegsay
e21d7d4baf
Update DendriteJS.Dockerfile 2020-06-03 16:03:54 +01:00
Kegsay
dc3338d1f2
Remove ENV to avoid confusion 2020-06-02 16:35:07 +01:00
Kegan Dougal
1f43c24f86 Add Dockerfile for building p2p riot
$ docker build -t dendritejs -f DendriteJS.Dockerfile .
$ docker run --rm -p 8090:80 dendritejs
2020-05-29 21:01:58 +01:00
Neil Alexander
7d6461dd3c
Server key component (#1050)
* Server key API (works for monolith but not for polylith yet)

* Re-enable caching on server key API component

* Groundwork for HTTP APIs for server key API

* Hopefully implement HTTP for server key API

* Simplify public key request marshalling from map keys

* Update gomatrixserverlib

* go mod tidy

* Common -> internal

* remove keyring.go

* Update Docker Hub for server key API

* YAML is funny about indentation

* Wire in new server key API into hybrid monolith mode

* Create maps

* Route server key API endpoints on internal API mux

* Fix server key API URLs

* Add fetcher behaviour into server key API implementation

* Return error if we failed to fetch some keys

* Return results anyway

* Move things about a bit

* Remove unused code

* Fix comments, don't use federation sender URL in polylith mode

* Add server_key_api to sample config

* Review comments

* HTTP API to cache keys that have been requested

* Overwrite server_key_api listen in monolith hybrid mode
2020-05-27 10:19:24 +01:00
Neil Alexander
f223da2f35 Fix URL in compose file 2020-05-21 16:22:25 +01:00
Kegsay
24d8df664c
Fix #897 and shuffle directory around (#1054)
* Fix #897 and shuffle directory around

* Update find-lint

* goimports

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-05-21 14:40:13 +01:00