Commit Graph

105 Commits

Author SHA1 Message Date
Neil Alexander
39c7a8915c
Multi-personality polylith binary (#1552)
* Initial work oon multipersonality binary

* Remove old binaries

* Monolith and polylith binaries

* Better logging

* dendrite-poly-multi

* Fix path

* Copyright notices etc

* Tweaks

* Update Docker, INSTALL.md

* Take first argument if flags package doesn't find any args

* Postgres 9.6 or later, fix some more Docker stuff

* Don't create unnecessary e2ekey DB

* Run go mod tidy
2020-10-20 16:11:24 +01:00
Neil Alexander
92982a402f
Update Docker (#1542)
* Separate Docker images, rather than tags

* Allow specifying tag to build/push/pull

* Include goose in Docker builds
2020-10-20 11:34:22 +01:00
Neil Alexander
49abe359e6
Start Kafka connections for each component that needs them (#1527)
* Start Kafka connection for each component that needs one

* Fix roomserver unit tests

* Rename to naffkaInstance (@Kegsay review comment)

* Fix import cycle
2020-10-15 13:27:13 +01:00
Neil Alexander
73bc28b11f
Update docker-compose.deps.yml 2020-10-13 09:54:07 +01:00
Neil Alexander
2f578531ba
Update Docker samples 2020-10-13 09:50:11 +01:00
Kegsay
bf7e85848b
Rename serverkeyapi to signingkeyserver (#1492)
* Rename serverkeyapi to signingkeyserver

We use "api" for public facing stuff and "server" for internal stuff.
As the server key API is internal only, we call it 'signing key server',
which also clarifies the type of key (as opposed to TLS keys, E2E keys, etc)

* Convert docker/scripts to use signing-key-server

* Rename missed bits
2020-10-07 16:23:18 +01:00
Neil Alexander
85c43e1bef
Don't drop federation retries for Yggdrasil demos 2020-10-02 15:14:01 +01:00
Neil Alexander
dbae85283f
Bring docker sample config into sync with normal one 2020-10-01 15:37:45 +01:00
Neil Alexander
05e5386fb0
Update gobind database paths 2020-09-30 15:57:31 +01:00
Kegsay
c992f4f1f4
Remove current state server (#1405)
* Remove current state server

Closes #1365 #1272 #1357

* Remove current state server from scripts/docs
2020-09-07 14:47:59 +01:00
Kegsay
088294ee65
Remove QueryRoomsForUser from current state server (#1398) 2020-09-04 15:58:30 +01:00
Neil Alexander
74743ac8ae
Rate limiting (#1385)
* Initial rate limiting

* Move rate limiting to client API

* Update rate limits to hopefully be self-cleaning

* Use X-Forwarded-For, add comments

* Reduce rate limit threshold

* Tweak interval

* Configurable backoff

* Review comments, set cleanup interval to 30 seconds

* Allow generate-config to produce sane CI config

* Fix Complement dockerfile
2020-09-03 10:12:11 +01:00
Kegsay
d64d0c4be2
Update complement.sh 2020-09-03 10:07:14 +01:00
Neil Alexander
096191ca24
Use federation sender for backfill/getting missing events (#1379)
* Use federation sender for backfill and getting missing events

* Fix internal URL paths

* Update go.mod/go.sum for matrix-org/gomatrixserverlib#218

* Add missing server implementations in HTTP interface
2020-09-02 15:26:30 +01:00
Kegsay
e473320e73
Refactor roomserver/internal - split perform stuff out (#1380)
- New package `perform` which contains all `Perform` functions
- New package `helpers` which contains helper functions used by both
  perform and query/input functions.
- Perform invite/leave have no idea how to `WriteOutputEvents` and this
  is now returned from `PerformInvite` or `PerformLeave` respectively.

Still to do:
 - RSAPI is fed into the inviter/joiner/leaver - this introduces circular
   logic so will need to be removed.
 - Put query operations in a `query` package.
 - Put input operations (and output) in an `input` package.
 - Factor out helper functions as much as possible, possibly rejigging the
   storage layer in the process.
2020-09-02 13:47:31 +01:00
Neil Alexander
737802fc06
Update Complement dockerfile to not wait for apt-get input 2020-08-28 15:19:47 +01:00
Kegsay
9af2f5f1f2
Remove device DB from clientapi (#1352)
* Remove device DB from clientapi

* Remove device DB from startup configuration

It's all an impl detail now in user API
2020-08-27 18:53:40 +01:00
Kegsay
ec95d331a5
Add support for running Complement on current working directories (#1291)
This will be used in the future by Buildkite to run on CI.
2020-08-20 18:35:04 +01:00
Neil Alexander
c1f4faf308
Fix Docker builds, polylith components (#1269) 2020-08-13 18:27:19 +01:00
Neil Alexander
9677a95afc
API setup refactoring (#1266)
* Start HTTP endpoint refactoring

* Update SetupAndServeHTTP

* Fix builds

* Don't set up external listener if no address configured

* TLS HTTP setup

* Break apart client/federation/key/media muxes

* Tweaks

* Fix P2P demos

* Fix media API routing

* Review comments @Kegsay

* Update sample config

* Fix gobind build

* Fix External -> Public in federation API test
2020-08-13 12:16:37 +01:00
Neil Alexander
52eeeb1627
Prefix-defined Kafka topics (#1254)
* Prefix-defined Kafka topics

* Fix current state server test
2020-08-10 15:18:37 +01:00
Neil Alexander
4b09f445c9
Configuration format v1 (#1230)
* Initial pass at refactoring config (not finished)

* Don't forget current state and EDU servers

* More shifting around

* Update server key API tests

* Fix roomserver test

* Fix more tests

* Further tweaks

* Fix current state server test (sort of)

* Maybe fix appservices

* Fix client API test

* Include database connection string in database options

* Fix sync API build

* Update config test

* Fix unit tests

* Fix federation sender build

* Fix gobind build

* Set Listen address for all services in HTTP monolith mode

* Validate config, reinstate appservice derived in directory, tweaks

* Tweak federation API test

* Set MaxOpenConnections/MaxIdleConnections to previous values

* Update generate-config
2020-08-10 14:18:04 +01:00
Neil Alexander
b7491aae03
Yggdrasil demo updates (#1241)
* PerformServersAlive in PerformBroadcastEDU

* Don't double-pointer

* More reliable QUIC session handling

* Direct peer lookup, other tweaks

* Tweaks

* Try to wake up queues on incoming QUIC session

* Set session callbak on gobind build

* Fix incoming session storage

* Stateless reset, other tweaks

* Reset sessions when coordinates change

* Disable HTTP connection reuse, tweak timeouts
2020-08-06 16:00:42 +01:00
Kegsay
0c4e8f6d4f
Send device list updates to servers (outbound only) (#1237)
* Add QueryDeviceMessages to serve up device keys and stream IDs

* Consume key change events in fedsender

Don't yet send them to destinations as we haven't worked them out yet

* Send device list updates to all required servers

* Glue it all together
2020-08-04 11:32:14 +01:00
Kegsay
a7e67e65a8
Notify clients when devices are deleted (#1233)
* Recheck device lists when join/leave events come in

* Add PerformDeviceDeletion

* Notify clients when devices are deleted

* Unbreak things

* Remove debug logging
2020-07-30 18:00:56 +01:00
Neil Alexander
292a9ddd82
Fix Yggdrasil demo builds 2020-07-30 15:47:44 +01:00
Kegsay
98f2f09bb4
keyserver: produce key change events (#1218)
* Produce kafka events when keys are added

* Consume key changes in syncapi with TODO markers for handling them and catching up

* unbreak tests

* Linting
2020-07-23 16:41:36 +01:00
Kegan Dougal
9684efebd7 Fix gobind 2020-07-22 17:06:19 +01:00
Neil Alexander
1e71fd645e
Persistent federation sender blacklist (#1214)
* Initial persistence of blacklists

* Move statistics folder

* Make MaxFederationRetries configurable

* Set lower failure thresholds for Yggdrasil demos

* Still write events into database for blacklisted hosts (they can be tidied up later)

* Review comments
2020-07-22 17:01:29 +01:00
Creed Haymond
f3c482b078
Adding the e2ekey database to the create_db script and docs (#1206)
Signed-off-by: Creed Haymond <creedasaurus@gmail.com>
2020-07-20 09:18:48 +01:00
Neil Alexander
fbf139f20e
Small fix to gobind build 2020-07-16 14:26:04 +01:00
Neil Alexander
e5208c2ec9
Yggdrasil demo updates ("Bare QUIC")
Squashed commit of the following:

commit 86c2388e13ffdbabdd50cea205652dccc40e1860
Merge: b0a3ee6c f5e7e751
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Jul 16 13:47:10 2020 +0100

    Merge branch 'master' into neilalexander/yggbarequic

commit b0a3ee6c5c063962384bb91c59ec753ddc8cfe5f
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Jul 16 13:42:22 2020 +0100

    Add support for broadcasting wake-up EDUs to known hosts

commit 8a5c2020b3a4b705b5d5686a9e71990a49e6d471
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Jul 16 13:42:10 2020 +0100

    Bare QUIC demo working

commit d3939b3d6568cf4262c0391486a5203873b68bfc
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Jul 15 11:42:43 2020 +0100

    Support bare Yggdrasil sessions with encrypted QUIC
2020-07-16 13:52:08 +01:00
Neil Alexander
08e9d996b6
Yggdrasil demo updates
Squashed commit of the following:

commit 6c2c48f862c1b6f8e741c57804282eceffe02487
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Jul 10 16:28:09 2020 +0100

    Add README.md

commit 5eeefdadf8e3881dd7a32559a92be49bd7ddaf47
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Jul 10 10:18:50 2020 +0100

    Fix wedge in federation sender

commit e2ebffbfba25cf82378393940a613ec32bfb909f
Merge: 0883ef88 abf26c12
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Jul 10 09:51:23 2020 +0100

    Merge branch 'master' into neilalexander/yggdrasil

commit 0883ef8870e340f2ae9a0c37ed939dc2ab9911f6
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Jul 10 09:51:06 2020 +0100

    Adjust timeouts

commit ba2d53199910f13b60cc892debe96a962e8c9acb
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Jul 9 16:34:40 2020 +0100

    Try to wake up from peers/sessions properly

commit 73f42eb494741ba5b0e0cef43654708e3c8eb399
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Jul 9 15:43:38 2020 +0100

    Use TransactionWriter to reduce database lock issues on SQLite

commit 08bfe63241a18c58c539c91b9f52edccda63a611
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Jul 9 12:38:02 2020 +0100

    Un-wedge federation

    Squashed commit of the following:

    commit aee933f8785e7a7998105f6090f514d18051a1bd
    Author: Neil Alexander <neilalexander@users.noreply.github.com>
    Date:   Thu Jul 9 12:22:41 2020 +0100

        Un-goroutine the goroutines

    commit 478374e5d18a3056cac6682ef9095d41352d1295
    Author: Neil Alexander <neilalexander@users.noreply.github.com>
    Date:   Thu Jul 9 12:09:31 2020 +0100

        Reduce federation sender wedges

commit 40cc62c54d9e3a863868214c48b7c18e522a4772
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Jul 9 10:02:52 2020 +0100

    Handle switching in/out background more reliably
2020-07-10 16:28:18 +01:00
Neil Alexander
4370a808e3
Fix media path for iOS gobind build 2020-07-08 16:24:31 +01:00
Neil Alexander
af6bc47f16
Squashed commit of the following:
commit b4cb47aa1329d2ada10ae6426fd9d2a69f47536a
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Jul 8 14:13:27 2020 +0100

    Restrict transaction send context time

commit 7c28205cdb5d842071d46b1ec599d09cca708e57
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Jul 8 14:00:06 2020 +0100

    Add to gobind build

commit d9e2c72e0576a2eb0ce6ac48eed6cc9d4761a0ea
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Jul 8 13:43:21 2020 +0100

    Wake up destination queues for new sessions/links

commit 21766c6c52bd00511d28981457e9034358c32a8d
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Jul 8 13:17:18 2020 +0100

    Tweak QUIC parameters
2020-07-08 14:52:48 +01:00
Neil Alexander
b21fad92e5 Tweaks to Dendrite Yggdrasil demo setup 2020-07-07 14:18:58 +01:00
Neil Alexander
00e0df7c87 Yggdrasil demo: Add Stop functions 2020-07-06 17:09:02 +01:00
Neil Alexander
6c4eabbe92 Updated functionality for updating Yggdrasil config in iOS demo 2020-07-06 14:51:59 +01:00
Neil Alexander
ce2a285887 Add public rooms provider into gobind build 2020-07-03 14:29:36 +01:00
Kegsay
6c4b8185d7
Implement ExtraPublicRoomsProvider for p2p demos (#1180)
* Change API and rename to ExtraPublicRoomsProvider

* Make dendritejs work again

* Maybe make libp2p demo work again

* Linting
2020-07-03 12:59:00 +01:00
Neil Alexander
38caf8e5b7
Yggdrasil+QUIC demo, federation sender tweaks (#1177)
* Initial QUIC work

* Update Yggdrasil demo

* Make sure that the federation sender knows how many pending events are in the database when the worker starts

* QUIC tunables

* pprof

* Don't spin

* Set build info for Yggdrasil
2020-07-02 17:43:07 +01:00
Kegsay
9c1f38621c
Remove publicroomsapi (#1176)
* Remove all of publicroomsapi

* Remove references to publicroomsapi

* Remove doc references to publicroomsapi
2020-07-02 17:11:33 +01:00
Neil Alexander
8e7947926c Fix Yggdrasil gobind build, set display name at registration 2020-07-01 13:35:58 +01:00
Martin Honermeyer
a1352cbd9e
Push serverkeyapi docker image to registry as well (#1154)
Signed-off-by: Martin Honermeyer <maze@strahlungsfrei.de>
2020-06-23 09:51:38 +01:00
Ashley Nelson
61e0482fef
Add appservices component to docker scripts (#1153)
Signed-off-by: Ashley Nelson <amn@fantashley.com>
2020-06-20 21:28:30 +01:00
Neil Alexander
7f26b0cd13
Bind build support, further Yggdrasil demo updates (#1152)
* Add gobind builds for Yggdrasil demo

* Massage client API a bit

* Fix build

* Fix gobind build

* Fix gobind client API setup

* Tweaks

* Tweaks

* Update sytest-whitelist, add comment

* Default to sending push rules on initial sync
2020-06-19 13:29:27 +01:00
Neil Alexander
72444e4a4f
User API polylith component (#1151)
* User API polylith component

* Add docker-pull.sh
2020-06-19 09:37:19 +01:00
Neil Alexander
67784ecb56
Nuke old Docker stuff (#1104)
* Remove old Docker stuff

* Move hub stuff to upper level docker directory

* Build monolith images

* Update README.md

* Update paths

* Update README.md
2020-06-05 19:00:30 +01:00
Kegsay
e21d7d4baf
Update DendriteJS.Dockerfile 2020-06-03 16:03:54 +01:00
Kegsay
dc3338d1f2
Remove ENV to avoid confusion 2020-06-02 16:35:07 +01:00
Kegan Dougal
1f43c24f86 Add Dockerfile for building p2p riot
$ docker build -t dendritejs -f DendriteJS.Dockerfile .
$ docker run --rm -p 8090:80 dendritejs
2020-05-29 21:01:58 +01:00
Neil Alexander
02fe38e1f7
Per-user-per-device sync streams (#1068)
* Per-user-per-device sync streams

* Tweaks

* Tweaks

* Pass full device into CompleteSync

* Set user IDs and device IDs properly in tests

* Add new test, fix TestNewEventAndWasPreviouslyJoinedToRoom

* nolint a function that is not used yet

* Add test for waking up single device

* Hopefully unstick test

* Try to ensure that TestCorrectStreamWakeup doesn't block forever

* Update tests
2020-05-28 10:05:04 +01:00
Neil Alexander
7d6461dd3c
Server key component (#1050)
* Server key API (works for monolith but not for polylith yet)

* Re-enable caching on server key API component

* Groundwork for HTTP APIs for server key API

* Hopefully implement HTTP for server key API

* Simplify public key request marshalling from map keys

* Update gomatrixserverlib

* go mod tidy

* Common -> internal

* remove keyring.go

* Update Docker Hub for server key API

* YAML is funny about indentation

* Wire in new server key API into hybrid monolith mode

* Create maps

* Route server key API endpoints on internal API mux

* Fix server key API URLs

* Add fetcher behaviour into server key API implementation

* Return error if we failed to fetch some keys

* Return results anyway

* Move things about a bit

* Remove unused code

* Fix comments, don't use federation sender URL in polylith mode

* Add server_key_api to sample config

* Review comments

* HTTP API to cache keys that have been requested

* Overwrite server_key_api listen in monolith hybrid mode
2020-05-27 10:19:24 +01:00
Neil Alexander
f223da2f35 Fix URL in compose file 2020-05-21 16:22:25 +01:00
Kegsay
24d8df664c
Fix #897 and shuffle directory around (#1054)
* Fix #897 and shuffle directory around

* Update find-lint

* goimports

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-05-21 14:40:13 +01:00