Commit Graph

358 Commits

Author SHA1 Message Date
Till
c2db38d295
Add user profile tests, refactor user API methods (#3030)
This adds tests for `/profile`.
Also, as a first change in this regard, refactors the methods defined on
the `UserInternalAPI` to not use structs as the request/response
parameters.
2023-04-03 20:19:26 +02:00
Till
234ed603e6
Move every db.Prepare to sqlutil.Statementlist, remove trace driver (#3026)
Doesn't buy us much, but makes everything a bit more consistent.

Also removes the SQL trace driver, as it is unused and the output is
hard to read anyway.
2023-03-23 13:52:53 +01:00
Till
5e85a00cb3
Remove BaseDendrite (#3023)
Removes `BaseDendrite` to, hopefully, make testing and composing of
components easier in the future.
2023-03-22 09:21:32 +01:00
Till
5579121c6f
Preparations for removing BaseDendrite (#3016)
Preparations to actually remove/replace `BaseDendrite`.
Quite a few changes:
- SyncAPI accepts an `fulltext.Indexer` interface (fulltext is removed
from `BaseDendrite`)
- Caches are removed from `BaseDendrite`
- Introduces a `Router` struct (likely to change)
  - also fixes #2903
- Introduces a `sqlutil.ConnectionManager`, which should remove
`base.DatabaseConnection` later on
- probably more
2023-03-17 11:09:45 +00:00
Till
6c20f8f742
Refactor StoreEvent, add MaybeRedactEvent, create an EventDatabase (#2989)
This PR changes the following:
- `StoreEvent` now only stores an event (and possibly prev event),
instead of also doing redactions
- Adds a `MaybeRedactEvent` (pulled out from `StoreEvent`), which should
be called after storing events
- a few other things
2023-03-01 17:06:47 +01:00
Till
4594233f89
Merge keyserver & userapi (#2972)
As discussed yesterday, a first draft of merging the keyserver and the
userapi.
2023-02-20 14:58:03 +01:00
Till
11d9b9db0e
Remove polylith/API mode (#2967)
This removes most of the code used for polylith/API mode.

This removes the `/api` internal endpoints entirely. 

Binary size change roughly 5%: 
```
51437560 Feb 13 10:15 dendrite-monolith-server # old
48759008 Feb 13 10:15 dendrite-monolith-server # new
```
2023-02-14 12:47:47 +01:00
devonh
63df85db6d
Relay integration to pinecone demos (#2955)
This extends the dendrite monolith for pinecone to integrate the s&f
features into the mobile apps.
Also makes a few tweaks to federation queueing/statistics to make some
edge cases more robust.
2023-01-28 23:27:53 +00:00
devonh
5b73592f5a
Initial Store & Forward Implementation (#2917)
This adds store & forward relays into dendrite for p2p.
A few things have changed:
- new relay api serves new http endpoints for s&f federation
- updated outbound federation queueing which will attempt to forward
using s&f if appropriate
- database entries to track s&f relays for other nodes
2023-01-23 17:55:12 +00:00
Till
ce2bfc3f2e
Make tests more reliable (#2948)
When using `testrig.CreateBase` and then using that base for other
`NewInternalAPI` calls, we never actually shutdown the components.
`testrig.CreateBase` returns a `close` function, which only removes the
database, so still running components have issues connecting to the
database, since we ripped it out underneath it - which can result in
"Disk I/O" or "pq deadlock detected" issues.
2023-01-20 12:45:56 +01:00
Neil
738686ae68
Add /_dendrite/admin/purgeRoom/{roomID} (#2662)
This adds a new admin endpoint `/_dendrite/admin/purgeRoom/{roomID}`. It
completely erases all database entries for a given room ID.

The roomserver will start by clearing all data for that room and then
will generate an output event to notify downstream components (i.e. the
sync API and federation API) to do the same.

It does not currently clear media and it is currently not implemented
for SQLite since it relies on SQL array operations right now.

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: Till Faelligen <2353100+S7evinK@users.noreply.github.com>
2023-01-19 21:02:32 +01:00
Till
0d0280cf5f
/sync performance optimizations (#2927)
Since #2849 there is no limit for the current state we fetch to
calculate history visibility. In large rooms this can cause us to fetch
thousands of membership events we don't really care about.
This now only gets the state event types and senders in our timeline,
which should significantly reduce the amount of events we fetch from the
database.

Also removes `MaxTopologicalPosition`, as it is an unnecessary DB call,
given we use the result in `topological_position < $1` calls.
2023-01-17 10:08:23 +01:00
Devon Hudson
477a44faa6
Always initialize statistics server map 2023-01-12 09:23:03 -07:00
Till Faelligen
09dff951d6
More flakey tests 2022-12-22 13:04:32 +01:00
Till Faelligen
beea2432e6
Fix flakey test 2022-12-22 11:31:54 +01:00
Till
d3db542fbf
Add federation peeking table tests (#2920)
As the title says, adds tests for inbound/outbound peeking federation
table tests.

Also removes some unused code
2022-12-22 10:56:20 +01:00
Till Faelligen
ded43e0f2d
Fix issue with sending presence events to invalid servers 2022-12-06 13:27:33 +01:00
Till
e245a26f6b
Enable/Disable internal metrics (#2899)
Basically enables us to use `test.WithAllDatabases` when testing
internal HTTP APIs, as this would otherwise result in Prometheus
complaining about already registered metric names.
2022-12-05 13:53:36 +01:00
Neil Alexander
8299da5905
Fix registration for virtual hosting 2022-11-18 13:24:02 +00:00
devonh
a8e7ffc7ab
Add p2p wakeup broadcast handling to pinecone demos (#2841)
Adds wakeup broadcast handling to the pinecone demos.
This will reset their blacklist status and interrupt any ongoing
federation queue backoffs currently in progress for this peer.
The end result is that any queued events will quickly be sent to the
peer if they had disconnected while attempting to send events to them.
2022-11-18 00:29:23 +00:00
Neil Alexander
16325203af
Try that again 2022-11-17 09:32:19 +00:00
Neil Alexander
607819f425
Fix /key/v2/server, add HTTP Host matching 2022-11-17 09:26:56 +00:00
Neil Alexander
163dabc498
Fix bug in a2f72dd9 2022-11-16 15:10:33 +00:00
Neil Alexander
a2f72dd966
Fix slice out of bounds in federation API 2022-11-16 09:39:19 +00:00
Neil Alexander
d558da1c87
Virtual host server name workaround 2022-11-16 09:34:09 +00:00
Neil Alexander
deddf686b9
Tweak /key/v2/server 2022-11-16 09:16:07 +00:00
Neil Alexander
9b8bb55430
Don't get blacklisted hosts when querying joined servers (#2880)
Otherwise we just waste time/CPU.
2022-11-15 17:21:16 +00:00
Neil Alexander
6650712a1c
Federation fixes for virtual hosting 2022-11-15 15:05:23 +00:00
Neil Alexander
529df30b56
Virtual hosting schema and logic changes (#2876)
Note that virtual users cannot federate correctly yet.
2022-11-11 16:41:37 +00:00
Till
0193549201
Send presence to newly added servers (#2869)
This should make `New federated private chats get full presence
information (SYN-115)` happy.
2022-11-11 10:35:17 +01:00
Till
b13cb43785
Send presence to joined hosts only (#2858)
Send presence events only to rooms the user is participating, not all
servers we know about.
Should fix #2752
2022-11-04 13:23:00 +01:00
Neil Alexander
a2706e6498
Refactor claimRemoteKeys 2022-10-27 15:34:26 +01:00
Till
444b4bbdb8
Add AS specific public room list endpoints (#2836)
Adds `PUT
/_matrix/client/v3/directory/list/appservice/{networkId}/{roomId}` and
`DELTE
/_matrix/client/v3/directory/list/appservice/{networkId}/{roomId}`
support, as well as the ability to filter `/publicRooms` on networkID
and including all networks.
2022-10-27 14:40:35 +02:00
devonh
97491a174b
Associate events in db before queueing them to send (#2833)
Fixes a race condition between sending federation events and having them
fully associated in the database.
2022-10-26 17:35:01 +01:00
Neil Alexander
5298dd1133
Update federation API consumers 2022-10-26 14:52:33 +01:00
Neil Alexander
f6dea712d2
Initial support for multiple server names (#2829)
This PR is the first step towards virtual hosting by laying the
groundwork for multiple server names being configured.
2022-10-26 12:59:19 +01:00
Till
9e4c3171da
Optimize inserting pending PDUs/EDUs (#2821)
This optimizes the association of PDUs/EDUs to their destination by
inserting all destinations in one transaction.
2022-10-21 12:50:51 +02:00
devonh
b58c9bb094
Fix flakey queue test (#2818)
Ensure both events are added to the database, even if the destination is
already blacklisted.
2022-10-20 15:37:35 +00:00
Till Faelligen
6a93858125
Fix race condition 2022-10-20 10:45:59 +02:00
devonh
241d5c47df
Refactor Federation Destination Queues (#2807)
This is a refactor of the federation destination queues.
It fixes a few things, namely:
- actually retry outgoing events with backoff behaviour
- obtain enough events from the database to fill messages as much as
possible
- minimize the amount of running goroutines
  - use pure timers for backoff
  - don't restart queue unless necessary
  - close the background task when backing off
- increase max edus in a transaction to match the spec
- cleanup timers more aggresively to reduce memory usage
- add jitter to backoff timers to reduce resource spikes
- add a bunch of tests (with real and fake databases) to ensure
everything is working
2022-10-19 11:03:16 +01:00
Neil Alexander
f3be4b3185
Revert "Federation backoff fixes and tests (#2792)"
This reverts commit dcedd1b6bf.
2022-10-13 16:06:50 +01:00
devonh
dcedd1b6bf
Federation backoff fixes and tests (#2792)
This fixes some edge cases where federation queue backoffs and
blacklisting weren't behaving as expected.
It also adds new tests for the federation queues to ensure their
behaviour continues to work correctly.
2022-10-13 14:38:13 +00:00
Neil Alexander
0a9aebdf01
Private read receipts (#2789)
Implement behaviours for `m.read.private` receipts.
2022-10-11 12:27:21 +01:00
Neil Alexander
9ed8ff6b93
Tweak federation M_NOT_FOUND errors 2022-10-11 10:48:36 +01:00
Neil Alexander
6bf1912525
Fix joined hosts with RewritesState (#2785)
This ensures that the joined hosts in the federation API are correct
after the state is rewritten. This might fix some races around the time
of joining federated rooms.
2022-10-10 16:54:04 +01:00
Till
b000db81ca
Send E2EE related errors to sentry (#2784)
Only sends errors if we're not retrying them in NATS.
Not sure if those should be scoped/tagged with something like "E2EE".
2022-10-10 17:36:26 +02:00
Till
1ca3f3efb5
Fix issue with DMs shown as normal rooms (#2776)
Fixes #2121, test added in
https://github.com/matrix-org/complement/pull/494
2022-10-07 16:00:12 +02:00
Neil Alexander
d605d928bc
Allow specifying old signing keys with the public key and key ID only (#2770)
If the private key file is lost, it's often possible to retrieve the
public key from another server elsewhere, so we should make it possible
to configure it in that way.
2022-10-06 11:56:00 +01:00
Neil Alexander
e53dcb25a9
Tweak logging for federated room joins 2022-10-06 10:07:13 +01:00
Till
8c0c3441d8
Add RoomEventType nats.Header to avoid unneeded unmarshalling (#2765) 2022-10-05 12:12:42 +02:00
Neil Alexander
1b65c97ad1
Remove Suppressing send-to-device log line
The behaviour is correct and I have no idea why we are logging it.
2022-09-29 16:18:42 +01:00
Neil Alexander
f022fc1397
Remove origin field from PDUs (#2737)
This nukes the `origin` field from PDUs as per
matrix-org/matrix-spec#998, matrix-org/gomatrixserverlib#341.
2022-09-26 17:35:35 +01:00
Till
0ddfb0cad4
Tweak InsertMigration to avoid logging (#2720)
We'd still produce logs in Postgres when trying to insert a migration we
already ran. This should stop us from creating those log entries.
2022-09-22 15:53:15 +02:00
Neil Alexander
bd39748b5c
Update dependencies (#2729)
This updates Dendrite dependencies.
2022-09-20 15:01:19 +01:00
Till Faelligen
7bfc3074d1
Fix origin on device list update EDUs 2022-09-16 13:30:20 +02:00
Till
c366ccdfca
Send-to-device consumer/producer tweaks (#2713)
Some tweaks for the send-to-device consumers/producers:
- use `json.RawMessage` without marshalling it first
- try further devices (if available) if we failed to `PublishMsg` in the
producers
- some logging changes (to better debug E2EE issues)
2022-09-13 09:35:45 +02:00
Till
42a82091a8
Fix issue with stale device lists (#2702)
We were only sending the last entry to the worker, so most likely missed
updates.
2022-09-08 12:03:44 +02:00
Till
0d697f6754
Add HTTP status code to FederationClientError (#2699)
Also ensures we wait on more HTTP status codes.
2022-09-07 16:14:09 +02:00
Till
7e8c605f98
Avoid unneeded JSON operations (#2698)
We were `json.Unmarshal`ing the EDU and `json.Marshal`ing right before
sending the EDU to the stream. Those are now removed and the consumer
does `json.Unmarshal` once.
2022-09-07 12:16:04 +02:00
Till
2cfcfddecc
Add a SigningKeyUpdate producer (#2697)
This adds a new stream for signing key updates, this should ensure we
don't lose any updates over federation.
2022-09-07 11:45:12 +02:00
Till
440eb0f3a2
Handle errors differently in the DeviceListUpdater (#2695)
`If a device list update goes missing, the server resyncs on the next
one` was failing because a previous test would receive a `waitTime` of
1h, resulting in the test timing out.
This now tries to handle the returned errors differently, e.g. by using
the default `waitTime` of 2s. Also doesn't try further users in the
list, if one of the errors would cause a longer `waitTime`.
2022-09-07 11:44:27 +02:00
Neil Alexander
51d229b025
Configuration tweaks (#2567)
This makes the following changes:

* The various `Defaults` functions are now responsible for setting sane defaults if `generate` is specified, rather than hiding them in `generate-config`
* Some configuration options have been marked as `omitempty` so that they don't appear in generated configs unnecessarily (monolith-specific vs. polylith-specific options)
* A new option `-polylith` has been added to `generate-config` to create a config that makes sense for polylith deployments (i.e. including the internal/external API listeners and per-component database sections)
* A new option `-normalise` has been added to `generate-config` to take an existing file and add any missing options and/or defaults
2022-09-01 14:15:41 +01:00
Neil Alexander
175f65407a
Allow batching in JetStreamConsumer (#2686)
This allows us to receive more than one message from NATS at a time if we want.
2022-08-31 12:21:56 +01:00
Neil Alexander
16156b0b09
Fix 500s on /state, /state_ids when state not known (#2672)
This was due to bad error bubbling.
2022-08-25 09:51:36 +01:00
Till
78e5d05efc
Only set backOffStarted to false if until is not zero (#2669) 2022-08-23 16:54:42 +02:00
Winter
a379d3e814
De-race TestExpireEDUs (#2654)
In some conditions (fast CPUs), this test would race the clock for EDU expiration when all we want to make sure of is that the expired EDUs are properly deleted. Given this, we set the expiry time to 0 so the specified EDUs are always deleted when DeleteExpiredEDUs is called.
Fixes #2650.

Signed-off-by: Winter <winter@winter.cafe>
2022-08-19 07:28:33 +02:00
Till
7484689ad1
Actually store EDUs once we retrieved from the database (#2651)
We now actually cache the EDUs once we got them from the database and ensures we only evict them if we successfully deleted them.
2022-08-18 11:14:42 +01:00
Neil Alexander
c45d0936b5
Generic-based internal HTTP API (#2626)
* Generic-based internal HTTP API (tested out on a few endpoints in the federation API)

* Add `PerformInvite`

* More tweaks

* Fix metric name

* Fix LookupStateIDs

* Lots of changes to clients

* Some serverside stuff

* Some error handling

* Use paths as metric names

* Revert "Use paths as metric names"

This reverts commit a9323a6a343f5ce6461a2e5bd570fe06465f1b15.

* Namespace metric names

* Remove duplicate entry

* Remove another duplicate entry

* Tweak error handling

* Some more tweaks

* Update error behaviour

* Some more error tweaking

* Fix API path for `PerformDeleteKeys`

* Fix another path

* Tweak federation client proxying

* Fix another path

* Don't return typed nils

* Some more tweaks, not that it makes any difference

* Tweak federation client proxying

* Maybe fix the key backup test
2022-08-11 15:29:33 +01:00
Till
240ae257de
Add housekeeping function to delete old/expired EDUs (#2399)
* Add housekeeping function to delete old/expired EDUs

* Add migrations

* Evict EDUs from cache

* Fix queries

* Fix upgrade

* Use map[string]time.Duration to specify different expiry times

* Fix copy & paste mistake

* Set expires_at to tomorrow

* Don't allow NULL

* Add comment

* Add tests

* Use new testrig package

* Fix migrations

* Never expire m.direct_to_device

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: kegsay <kegan@matrix.org>
2022-08-09 11:15:58 +02:00
Neil Alexander
c8935fb53f
Do not use ioutil as it is deprecated (#2625) 2022-08-05 10:26:59 +01:00
Till
1b7f84250a
Fix linter issues (#2624)
* Try that again

* All hail the mighty linter?

* And once again

* goimport all the things
2022-08-05 11:12:41 +02:00
Brian Meek
de78eab63a
Add race testing to tests, and fix a few small race conditions in the tests (#2587)
* Add race testing to tests, and fix a few small race conditions in the tests

* Enable run-sytest on MacOS

* Remove deadlock detecting mutex, per code review feedback

* Remove autoformatting related changes and a closure that is not needed

* Adjust to importing nats client as 'natsclient'

Signed-off-by: Brian Meek <brian@hntlabs.com>

* Clarify the use of gooseMutex to proect goose internal state

Signed-off-by: Brian Meek <brian@hntlabs.com>

* Remove no longer needed mutex for guarding goose

Signed-off-by: Brian Meek <brian@hntlabs.com>
2022-08-05 09:19:33 +01:00
Till
9a655cb5e7
Only create a new destinationQueue if we don't have one (#2620) 2022-08-05 07:20:34 +02:00
Neil Alexander
eab87ef07d
Stronger checks for /send_join (#2604) 2022-08-02 10:22:17 +01:00
Till
081f5e7226
Update database migrations, remove goose (#2264)
* Add new db migration

* Update migrations
Remove goose

* Add possibility to test direct upgrades

* Try to fix WASM test

* Add checks for specific migrations

* Remove AddMigration
Use WithTransaction
Add Dendrite version to table

* Fix linter issues

* Update tests

* Update comments, outdent if

* Namespace migrations

* Add direct upgrade tests, skipping over one version

* Split migrations

* Update go version in CI

* Fix copy&paste mistake

* Use contexts in migrations

Co-authored-by: kegsay <kegan@matrix.org>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-07-25 10:39:22 +01:00
Neil Alexander
f0c8a03649
Membership updater refactoring (#2541)
* Membership updater refactoring

* Pass in membership state

* Use membership check rather than referring to state directly

* Delete irrelevant membership states

* We don't need the leave event after all

* Tweaks

* Put a log entry in that I might stand a chance of finding

* Be less panicky

* Tweak invite handling

* Don't freak if we can't find the event NID

* Use event NID from `types.Event`

* Clean up

* Better invite handling

* Placate the almighty linter

* Blacklist a Sytest which is otherwise fine under Complement for reasons I don't understand

* Fix the sytest after all (thanks @S7evinK for the spot)
2022-07-22 14:44:04 +01:00
Till
09f0ff14c8
Minor SendToDevice fix (#2565)
* Avoid unnecessary marshalling if sending to the local server

* Fix ordering of ToDevice messages

* Revive SendToDevice test
2022-07-12 08:23:58 +02:00
Neil Alexander
3ea21273bc
Ristretto cache (#2563)
* Try Ristretto cache

* Tweak

* It's beautiful

* Update GMSL

* More strict keyable interface

* Fix that some more

* Make less panicky

* Don't enforce mutability checks for now

* Determine mutability using deep equality

* Tweaks

* Namespace keys

* Make federation caches mutable

* Update cost estimation, add metric

* Update GMSL

* Estimate cost for metrics better

* Reduce counters a bit

* Try caching events

* Some guards

* Try again

* Try this

* Use separate caches for hopefully better hash distribution

* Fix bug with admitting events into cache

* Try to fix bugs

* Check nil

* Try that again

* Preserve order jeezo this is messy

* thanks VS Code for doing exactly the wrong thing

* Try this again

* Be more specific

* aaaaargh

* One more time

* That might be better

* Stronger sorting

* Cache expiries, async publishing of EDUs

* Put it back

* Use a shared cache again

* Cost estimation fixes

* Update ristretto

* Reduce counters a bit

* Clean up a bit

* Update GMSL

* 1GB

* Configurable cache sizees

* Tweaks

* Add `config.DataUnit` for specifying friendly cache sizes

* Various tweaks

* Update GMSL

* Add back some lazy loading caching

* Include key in cost

* Include key in cost

* Tweak max age handling, config key name

* Only register prometheus metrics if requested

* Review comments @S7evinK

* Don't return errors when creating caches (it is better just to crash since otherwise we'll `nil`-pointer exception everywhere)

* Review comments

* Update sample configs

* Update GHA Workflow

* Update Complement images to Go 1.18

* Remove the cache test from the federation API as we no longer guarantee immediate cache admission

* Don't check the caches in the renewal test

* Possibly fix the upgrade tests

* Update to matrix-org/gomatrixserverlib#322

* Update documentation to refer to Go 1.18
2022-07-11 14:31:31 +01:00
Neil Alexander
460dccf93d
Hopefully fix read receipts timestamps (#2557)
This should avoid coercions between signed and unsigned ints which might fix problems like `sql: converting argument $5 type: uint64 values with high bit set are not supported`.
2022-07-05 17:13:26 +01:00
Till
561c159ad7
Silence presence logs (#2547) 2022-06-30 12:34:37 +02:00
Neil Alexander
7120eb6bc9
Add InputDeviceListUpdate to the keyserver, remove old input API (#2536)
* Add `InputDeviceListUpdate` to the keyserver, remove old input API

* Fix copyright

* Log more information when a device list update fails
2022-06-15 14:27:07 +01:00
Till
1b90cc9536
Fix rare panic when returning user devices over federation (#2534) 2022-06-15 12:50:02 +02:00
Neil Alexander
247604979a
Add missing indexes to federation API destination queues (#2502)
* Add missing indexes to federation API destination queues

* Fix duplicate name
2022-05-30 11:01:20 +01:00
Neil Alexander
b541f3043f
Add support for MSC3787 and org.matrix.msc3787 room version (update to matrix-org/gomatrixserverlib#310) 2022-05-26 15:08:17 +01:00
Neil Alexander
ac4d0072cf
Fix using the remote provided event from /send_join (#2490) 2022-05-25 14:31:07 +01:00
Neil Alexander
81843e8836
Restricted join support on /make_join, /send_join (#2478)
* Add `QueryRestrictedJoinAllowed`

* Add `Resident` flag to `QueryRestrictedJoinAllowedResponse`

* Check restricted joins on federation API

* Return `Restricted` to determine if the room was restricted or not

* Populate `AuthorisedVia` properly

* Sign the event on `/send_join`, return it in the `/send_join` response in the `"event"` key

* Kick back joins with invalid authorising user IDs, use event from `"event"` key if returned in `RespSendJoin`

* Use invite helper in `QueryRestrictedJoinAllowed`

* Only use users with the power to invite, change error bubbling a bit

* Placate the almighty linter

One day I will nuke `gocyclo` from orbit and everything in the world will be much better for it.

* Review comments
2022-05-25 10:05:30 +01:00
Neil Alexander
4472267901
Preserve content from /make_join as well as supplied content in the request (#2481) 2022-05-23 17:54:01 +01:00
Neil Alexander
b5a497a0c0
Allow defers to run in TestMain in federation API tests 2022-05-23 14:54:43 +01:00
Kegan Dougal
ac92e04772 Remove debug logging 2022-05-17 13:31:48 +01:00
kegsay
6de29c1cd2
bugfix: E2EE device keys could sometimes not be sent to remote servers (#2466)
* Fix flakey sytest 'Local device key changes get to remote servers'

* Debug logs

* Remove internal/test and use /test only

Remove a lot of ancient code too.

* Use FederationRoomserverAPI in more places

* Use more interfaces in federationapi; begin adding regression test

* Linting

* Add regression test

* Unbreak tests

* ALL THE LOGS

* Fix a race condition which could cause events to not be sent to servers

If a new room event which rewrites state arrives, we remove all joined hosts
then re-calculate them. This wasn't done in a transaction so for a brief period
we would have no joined hosts. During this interim, key change events which arrive
would not be sent to destination servers. This would sporadically fail on sytest.

* Unbreak new tests

* Linting
2022-05-17 13:23:35 +01:00
kegsay
236b16aa6c
Begin adding syncapi component tests (#2442)
* Add very basic syncapi tests

* Add a way to inject jetstream messages

* implement add_state_ids

* bugfixes

* Unbreak tests

* Remove now un-needed API call

* Linting
2022-05-09 17:23:02 +01:00
Neil Alexander
79da75d483
Federation consumer adds_state_event_ids tweak (#2441)
* Don't ask roomserver for events we already have in federation API

* Check number of events returned is as expected

* Preallocate array

* Improve shape a bit
2022-05-09 16:19:35 +01:00
Neil Alexander
09d754cfbf
One NATS instance per BaseDendrite (#2438)
* One NATS instance per `BaseDendrite`

* Fix roomserver
2022-05-09 14:15:24 +01:00
kegsay
85704eff20
Clean up interface definitions (#2427)
* tidy up interfaces

* remove unused GetCreatorIDForAlias

* Add RoomserverUserAPI interface

* Define more interfaces

* Use AppServiceInternalAPI for consistent naming

* clean up federationapi constructor a bit

* Fix monolith in -http mode
2022-05-06 12:39:26 +01:00
kegsay
9957752a9d
Define component interfaces based on consumers (2/2) (#2425)
* convert remaining interfaces

* Tidy up the userapi interfaces
2022-05-05 19:30:38 +01:00
kegsay
506de4bb3d
Define component interfaces based on consumers (1/2) (#2423)
* Specify interfaces used by appservice, do half of clientapi

* convert more deps of clientapi to finer-grained interfaces

* Convert mediaapi and rest of clientapi

* Somehow this got missed
2022-05-05 13:17:38 +01:00
Neil Alexander
dd061a172e
Tidy up AddPublicRoutes (#2412)
* Simplify federation API `AddPublicRoutes`

* Simplify client API `AddPublicRoutes`

* Simplify media API `AddPublicRoutes`

* Simplify sync API `AddPublicRoutes`

* Simplify `AddAllPublicRoutes`
2022-05-03 17:17:02 +01:00
Neil Alexander
4ad5f9c982
Global database connection pool (for monolith mode) (#2411)
* Allow monolith components to share a single database pool

* Don't yell about missing connection strings

* Rename field

* Setup tweaks

* Fix panic

* Improve configuration checks

* Update config

* Fix lint errors

* Update comments
2022-05-03 16:35:06 +01:00
Neil Alexander
31799a3b2a
Device list display name fixes (#2405)
* Get device names from `unsigned` in `/user/devices`

* Fix display name updates

* Fix bug

* Fix another bug
2022-04-29 16:02:55 +01:00
Neil Alexander
6deb10f3f6
Don't answer expensive federation requests for rooms we no longer belong to (#2398)
This includes `/state`, `/state_ids`, `/get_missing_events` and `/backfill`.

This should fix #2396.
2022-04-28 11:45:56 +01:00