Commit Graph

32 Commits

Author SHA1 Message Date
Neil Alexander
9d6b77c58a
Try to retrieve missing auth events from multiple servers (#1516)
* Recursively fetch auth events if needed

* Fix processEvent call

* Ask more servers in lookupEvent

* Don't panic!

* Panic at the Disco

* Find servers more aggressively

* Add getServers

* Fix number of servers to 5, don't bail making RespState if auth events missing

* Fix panic

* Ignore missing state events too

* Report number of servers correctly

* Don't reuse request context for /send_join

* Update federation API tests

* Don't recurse processEvents

* Implement getEvents differently
2020-10-13 11:53:20 +01:00
Kegsay
9096bfcee8
Validate m.room.create events in send_join responses (#1505)
* Validate m.room.create events in send_join responses

For sytest compliance, refs #1315 and #1317

Fixes #1317

* Linting
2020-10-10 00:21:15 +01:00
Neil Alexander
fe5d1400bf
Update federation timeouts (#1504)
* Update to matrix-org/gomatrixserverlib#234

* Update gomatrixserverlib

* Update federation timeouts

* Fix dendritejs

* Increase /send context time in destination queue
2020-10-09 17:08:32 +01:00
Neil Alexander
bf90db5b60
Remove KindRewrite (#1481)
* Don't send rewrite events

* Remove final traces of rewrite events

* Remove test that is no longer needed

* Revert "Remove test that is no longer needed"

This reverts commit 9a45babff690480acd656a52f2c2950a5f7e9ada.

* Update test to use KindOutlier
2020-10-06 11:05:00 +01:00
Neil Alexander
a14b29b526
Initial notary support (#1436)
* Initial work on notary support

* Somewhat working (but not properly filtered) notary support, other tweaks

* Update gomatrixserverlib
2020-09-22 14:40:54 +01:00
Neil Alexander
a7563ede3d
Process federated joins in background context (#1434)
* Return early from federated room join

* Synchronous perform-join as long as possible

* Don't allow multiple federated joins to the same room by the same user
2020-09-22 11:05:45 +01:00
Neil Alexander
965f068d1a
Handle state with input event as new events (#1415)
* SendEventWithState events as new

* Use cumulative state IDs for final event

* Error wrapping in calculateAndSetState

* Handle overwriting same event type and state key

* Hacky way to spot historical events

* Don't exclude from sync

* Don't generate output events when rewriting forward extremities

* Update output event check

* Historical output events

* Define output room event type

* Notify key changes on state

* Don't send our membership event twice

* Deduplicate state entries

* Tweaks

* Remove unnecessary nolint

* Fix current state upsert in sync API

* Send auth events as outliers, state events as rewrite

* Sync API don't consume state events

* Process events actually

* Improve outlier check

* Fix local room check

* Remove extra room check, it seems to break the whole damn world

* Fix federated join check

* Fix nil pointer exception

* Better comments on DeduplicateStateEntries

* Reflow forced federated joins

* Don't force federated join for possibly even local invites

* Comment SendEventWithState better

* Rewrite room state in sync API storage

* Add TODO

* Clean up all room data when receiving create event

* Don't generate output events for rewrites, but instead notify that state is rewritten on the final new event

* Rename to PurgeRoom

* Exclude backfilled messages from /sync

* Split out rewriting state from updating state from state res

Co-authored-by: Kegan Dougal <kegan@matrix.org>
2020-09-15 11:17:46 +01:00
Neil Alexander
668a722ee0
Backoff for 401s (#1410)
* Backoff for 401s

* Human-readable retry_after in logs
2020-09-08 13:41:08 +01:00
Neil Alexander
1602df8752
Ignore state events with invalid signatures when joining rooms (#1407)
* Use state from RespSendJoin post-check

* Don't create input events for invalid events

* Let's try this again

* Update gomatrixserverlib

* Update gomatrixserverlib to matrix-org/gomatrixserverlib@38f437f
2020-09-07 16:54:51 +01:00
Neil Alexander
096191ca24
Use federation sender for backfill/getting missing events (#1379)
* Use federation sender for backfill and getting missing events

* Fix internal URL paths

* Update go.mod/go.sum for matrix-org/gomatrixserverlib#218

* Add missing server implementations in HTTP interface
2020-09-02 15:26:30 +01:00
Kegsay
6d6bb75137
Add FederationClient interface to federationsender (#1284)
* Add FederationClient interface to federationsender

- Use a shim struct in HTTP mode to keep the same API as `FederationClient`.
- Use `federationsender` instead of `FederationClient` in `keyserver`.

* Pointers not values

* Review comments

* Fix unit tests

* Rejig backoff

* Unbreak test

* Remove debug logs

* Review comments and linting
2020-08-20 17:03:07 +01:00
Neil Alexander
6cb1a65809
Synchronous invites (#1273)
* Refactor invites to be synchronous

* Fix synchronous invites

* Fix client API return type for send invite error

* Linter

* Restore PerformError on rsAPI.PerformInvite

* Update sytest-whitelist

* Don't override PerformError with normal errors

* Fix error passing

* Un-whitelist a couple of tests

* Update sytest-whitelist

* Try to handle multiple invite rejections better

* nolint

* Update gomatrixserverlib

* Fix /v1/invite test

* Remove replace from go.mod
2020-08-17 11:40:49 +01:00
Neil Alexander
4b09f445c9
Configuration format v1 (#1230)
* Initial pass at refactoring config (not finished)

* Don't forget current state and EDU servers

* More shifting around

* Update server key API tests

* Fix roomserver test

* Fix more tests

* Further tweaks

* Fix current state server test (sort of)

* Maybe fix appservices

* Fix client API test

* Include database connection string in database options

* Fix sync API build

* Update config test

* Fix unit tests

* Fix federation sender build

* Fix gobind build

* Set Listen address for all services in HTTP monolith mode

* Validate config, reinstate appservice derived in directory, tweaks

* Tweak federation API test

* Set MaxOpenConnections/MaxIdleConnections to previous values

* Update generate-config
2020-08-10 14:18:04 +01:00
Neil Alexander
b7491aae03
Yggdrasil demo updates (#1241)
* PerformServersAlive in PerformBroadcastEDU

* Don't double-pointer

* More reliable QUIC session handling

* Direct peer lookup, other tweaks

* Tweaks

* Try to wake up queues on incoming QUIC session

* Set session callbak on gobind build

* Fix incoming session storage

* Stateless reset, other tweaks

* Reset sessions when coordinates change

* Disable HTTP connection reuse, tweak timeouts
2020-08-06 16:00:42 +01:00
Neil Alexander
1e71fd645e
Persistent federation sender blacklist (#1214)
* Initial persistence of blacklists

* Move statistics folder

* Make MaxFederationRetries configurable

* Set lower failure thresholds for Yggdrasil demos

* Still write events into database for blacklisted hosts (they can be tidied up later)

* Review comments
2020-07-22 17:01:29 +01:00
Neil Alexander
e5208c2ec9
Yggdrasil demo updates ("Bare QUIC")
Squashed commit of the following:

commit 86c2388e13ffdbabdd50cea205652dccc40e1860
Merge: b0a3ee6c f5e7e751
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Jul 16 13:47:10 2020 +0100

    Merge branch 'master' into neilalexander/yggbarequic

commit b0a3ee6c5c063962384bb91c59ec753ddc8cfe5f
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Jul 16 13:42:22 2020 +0100

    Add support for broadcasting wake-up EDUs to known hosts

commit 8a5c2020b3a4b705b5d5686a9e71990a49e6d471
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Jul 16 13:42:10 2020 +0100

    Bare QUIC demo working

commit d3939b3d6568cf4262c0391486a5203873b68bfc
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Jul 15 11:42:43 2020 +0100

    Support bare Yggdrasil sessions with encrypted QUIC
2020-07-16 13:52:08 +01:00
Kegsay
43cddfe00f
Return remote errors from FS.PerformJoin (#1164)
* Return remote errors from FS.PerformJoin

Follows the same pattern as PerformJoin on roomserver (no error return).

Also return the right format for incompatible room version errors.

Makes a bunch of tests pass!

* Handle network errors better when returning remote HTTP errors

* Linting

* Fix tests

* Update whitelist, pass network errors through in API=1 mode
2020-06-25 15:04:48 +01:00
Kegsay
25cd2dd1c9
Remove unused internal APIs (#1117) 2020-06-11 15:07:16 +01:00
Kegsay
399b6ae334
Remove federationsender producer, which in fact was not a producer (#1115)
* Remove federationsender producer, which in fact was not a producer

* Set the signing struct
2020-06-10 16:54:43 +01:00
Neil Alexander
76ff47c052
Use AuthChainProvider to try and speed up federated joins (#1100)
* Use MissingAuthEventHandler on performjoin to try and speed up cases where we have missing events

* Update gomatrixserverlib

* Use supplied room version

* Use AuthChainProvider

* Tweaks

* Update gomatrixserverlib

* Signature checks
2020-06-05 11:48:52 +01:00
Kegsay
f4c676ccdd
Refactor how federationsender gets created (#1095)
* Refactor how federationsender gets created

* s/httpint/inthttp/ for alphabetical niceness with internal package
2020-06-04 14:27:10 +01:00
Kegsay
cfc137652e
Add a way to force federationsender to retry sending transactions (#1077)
* Add a way to force federationsender to retry sending transactions

And use it in P2P mode when we pick up new nodes.

* Linting

* Use atomic bool to stop us blocking on the channel
2020-06-01 18:34:08 +01:00
Kegsay
fe5cf6f880
fedsender: de-duplicate without sorting server names (#1073) 2020-05-29 13:50:06 +01:00
Neil Alexander
6d50212f29
Miscellaneous fixes (#1060)
* Add missing routing for PerformDirectoryLookupRequest

* Tweak output

* Fix some bugs in devices

* Don't default to federated room joins in response to invite

* Update sytest-whitelist

* Update comments

* Return correct room ID from PerformJoin

* Fix appservice and EDU server API setup, update sytest-whitelist

* Update sytest-whitelist
2020-05-26 14:41:16 +01:00
Neil Alexander
fe82e1f725
Separate muxes for public and internal APIs (#1056)
* Separate muxes for public and internal APIs

* Update client-api-proxy and federation-api-proxy so they don't add /api to the path

* Tidy up

* Consistent HTTP setup

* Set up prefixes properly
2020-05-22 11:43:17 +01:00
Kegsay
24d8df664c
Fix #897 and shuffle directory around (#1054)
* Fix #897 and shuffle directory around

* Update find-lint

* goimports

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-05-21 14:40:13 +01:00
Neil Alexander
5f6f8adaa5
Don't prematurely stop trying to join using servers (#1041)
* Don't prematurely stop trying to join using servers

* Factor out performJoinUsingServer
2020-05-15 13:55:14 +01:00
Neil Alexander
17d27331a3 Fix 'input to Unique() must be sorted' panic 2020-05-07 17:14:32 +01:00
Neil Alexander
a16db1c408
Improve federation sender performance, implement backoff and blacklisting, fix up invites a bit (#1007)
* Improve federation sender performance and behaviour, add backoff

* Tweaks

* Tweaks

* Tweaks

* Take copies of events before passing to destination queues

* Don't accidentally drop queued messages

* Don't take copies again

* Tidy up a bit

* Break out statistics (tracked component-wide), report success and failures from Perform actions

* Fix comment, use atomic add

* Improve logic a bit, don't block on wakeup, move idle check

* Don't retry sucessful invites, don't dispatch sendEvent, sendInvite etc

* Dedupe destinations, fix other bug hopefully

* Dispatch sends again

* Federation sender to ignore invites that are destined locally

* Loopback invite events

* Remodel a bit with channels

* Linter

* Only loopback invite event if we know the room

* We should tell other resident servers about the invite if we know about the room

* Correct invite signing

* Fix invite loopback

* Check HTTP response codes, push new invites to front of queue

* Review comments
2020-05-07 12:42:06 +01:00
Neil Alexander
9b1b095b49
Roomserver perform leave (#1004)
* First pass at PerformLeave

* Fix SQLite bulkSelectEventStateKey

* Update gomatrixserverlib

* Fix bugs

* Tidy a bit

* Satisfy King Linter

* Review comments

* Review comments

* Fix constants in SQLite event state keys table
2020-05-04 18:34:09 +01:00
Neil Alexander
5c894efd0e
Roomserver perform join (#1001)
* Add PerformJoin template

* Try roomserver perform join

* Send correct server name to FS API

* Pass through content, try to handle multiple server names

* Fix local server checks

* Don't refer to non-existent error

* Add directory lookups of aliases

* Remove unneeded parameters

* Don't repeat join events into the roomserver

* Unmarshal the content, that would help

* Check if the user is already in the room in the fedeationapi too

* Return incompatible room version error

* Use Membership, don't try more servers than needed

* Review comments, make FS API take list of servernames, dedupe them, break out of loop properly on success

* Tweaks
2020-05-04 13:53:47 +01:00
Neil Alexander
908108c23e
Rename FS queue package to internal (#997) 2020-05-01 13:01:50 +01:00