Commit Graph

29 Commits

Author SHA1 Message Date
Ronnie Ebrin
a677a288bd
federationsender/roomserver: don't panic while federation is disabled (#1615) 2020-12-04 14:08:17 +00:00
Neil Alexander
b5aa7ca3ab
Top-level setup package (#1605)
* Move config, setup, mscs into "setup" top-level folder

* oops, forgot the EDU server

* Add setup

* goimports
2020-12-02 17:41:00 +00:00
Neil Alexander
20a01bceb2
Pass pointers to events — reloaded (#1583)
* Pass events as pointers

* Fix lint errors

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update to matrix-org/gomatrixserverlib#240
2020-11-16 15:44:53 +00:00
S7evinK
bcb89ada5e
Implement read receipts (#1528)
* fix conversion from int to string yields a string of one rune, not a string of digits

* Add receipts table to syncapi

* Use StreamingToken as the since value

* Add required method to testEDUProducer

* Make receipt json creation "easier" to read

* Add receipts api to the eduserver

* Add receipts endpoint

* Add eduserver kafka consumer

* Add missing kafka config

* Add passing tests to whitelist

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Fix copy & paste error

* Fix column count error

* Make outbound federation receipts pass

* Make "Inbound federation rejects receipts from wrong remote" pass

* Don't use errors package

* - Add TODO for batching requests
- Rename variable

* Return a better error message

* - Use OutputReceiptEvent instead of InputReceiptEvent as result
- Don't use the errors package for errors
- Defer CloseAndLogIfError to close rows
- Fix Copyright

* Better creation/usage of JoinResponse

* Query all joined rooms instead of just one

* Update gomatrixserverlib

* Add sqlite3 migration

* Add postgres migration

* Ensure required sequence exists before running migrations

* Clarification on comment

* - Fix a bug when creating client receipts
- Use concrete types instead of interface{}

* Remove dead code
Use key for timestamp

* Fix postgres query...

* Remove single purpose struct

* Use key/value directly

* Only apply receipts on initial sync or if edu positions differ,
otherwise we'll be sending the same receipts over and over again.

* Actually update the id, so it is correctly send in syncs

* Set receipt on request to /read_markers

* Fix issue with receipts getting overwritten

* Use fmt.Errorf instead of pkg/errors

* Revert "Add postgres migration"

This reverts commit 722fe5a04628882b787d096942459961db159b06.

* Revert "Add sqlite3 migration"

This reverts commit d113b03f6495a4b8f8bcf158a3d00b510b4240cc.

* Fix selectRoomReceipts query

* Make golangci-lint happy

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-11-09 18:46:11 +00:00
Neil Alexander
3afc623098
Fix RewritesState bug (#1557)
* Set RewritesState once

* Check if any new state provided

* Obey rewritesState

* Don't nuke everything the sync API knows when purging state

* Fix panic from duplicate insert

* Consistency

* Use HasState

* Remove nolint

* Clean up joined rooms on state rewrite
2020-10-22 10:39:16 +01:00
Kegsay
088294ee65
Remove QueryRoomsForUser from current state server (#1398) 2020-09-04 15:58:30 +01:00
Neil Alexander
89c772fb78
Report which component failed to consume (#1375) 2020-09-01 16:53:38 +01:00
Neil Alexander
6cb1a65809
Synchronous invites (#1273)
* Refactor invites to be synchronous

* Fix synchronous invites

* Fix client API return type for send invite error

* Linter

* Restore PerformError on rsAPI.PerformInvite

* Update sytest-whitelist

* Don't override PerformError with normal errors

* Fix error passing

* Un-whitelist a couple of tests

* Update sytest-whitelist

* Try to handle multiple invite rejections better

* nolint

* Update gomatrixserverlib

* Fix /v1/invite test

* Remove replace from go.mod
2020-08-17 11:40:49 +01:00
Neil Alexander
52eeeb1627
Prefix-defined Kafka topics (#1254)
* Prefix-defined Kafka topics

* Fix current state server test
2020-08-10 15:18:37 +01:00
Neil Alexander
4b09f445c9
Configuration format v1 (#1230)
* Initial pass at refactoring config (not finished)

* Don't forget current state and EDU servers

* More shifting around

* Update server key API tests

* Fix roomserver test

* Fix more tests

* Further tweaks

* Fix current state server test (sort of)

* Maybe fix appservices

* Fix client API test

* Include database connection string in database options

* Fix sync API build

* Update config test

* Fix unit tests

* Fix federation sender build

* Fix gobind build

* Set Listen address for all services in HTTP monolith mode

* Validate config, reinstate appservice derived in directory, tweaks

* Tweak federation API test

* Set MaxOpenConnections/MaxIdleConnections to previous values

* Update generate-config
2020-08-10 14:18:04 +01:00
Kegan Dougal
78ab33f91f Unbreak postgres 2020-08-04 11:41:48 +01:00
Kegsay
0c4e8f6d4f
Send device list updates to servers (outbound only) (#1237)
* Add QueryDeviceMessages to serve up device keys and stream IDs

* Consume key change events in fedsender

Don't yet send them to destinations as we haven't worked them out yet

* Send device list updates to all required servers

* Glue it all together
2020-08-04 11:32:14 +01:00
Neil Alexander
72b3160776
Send-to-device messages over federation (#1198)
* Initial work to send send-to-device messages over federation

* Wire up send-to-device consumer, message formatting

* Generate random message ID

* Review comments, update sytest whitelist
2020-07-14 12:33:37 +01:00
Kegsay
4675e1ddb6
Add trace logging to RoomserverInternalAPI (#1120)
This is a wrapper around whatever impl we have which then logs
the function name/request/response/error.

Also tweak when we log on kafka streams: only log on the producer
side not the consumer side: we've never had issues with comms and
having 1 message rather than N would be nice.
2020-06-12 12:10:08 +01:00
Kegsay
ec7718e7f8
Roomserver API changes (#1118)
* s/QueryBackfill/PerformBackfill/g

* OutputEvent now includes AddStateEvents which contain the full event of extra state events

* Only include adds not the current event

* Get adding state right
2020-06-11 19:50:40 +01:00
Kegsay
24d8df664c
Fix #897 and shuffle directory around (#1054)
* Fix #897 and shuffle directory around

* Update find-lint

* goimports

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-05-21 14:40:13 +01:00
Neil Alexander
a16db1c408
Improve federation sender performance, implement backoff and blacklisting, fix up invites a bit (#1007)
* Improve federation sender performance and behaviour, add backoff

* Tweaks

* Tweaks

* Tweaks

* Take copies of events before passing to destination queues

* Don't accidentally drop queued messages

* Don't take copies again

* Tidy up a bit

* Break out statistics (tracked component-wide), report success and failures from Perform actions

* Fix comment, use atomic add

* Improve logic a bit, don't block on wakeup, move idle check

* Don't retry sucessful invites, don't dispatch sendEvent, sendInvite etc

* Dedupe destinations, fix other bug hopefully

* Dispatch sends again

* Federation sender to ignore invites that are destined locally

* Loopback invite events

* Remodel a bit with channels

* Linter

* Only loopback invite event if we know the room

* We should tell other resident servers about the invite if we know about the room

* Correct invite signing

* Fix invite loopback

* Check HTTP response codes, push new invites to front of queue

* Review comments
2020-05-07 12:42:06 +01:00
Neil Alexander
e15f6676ac
Consolidation of roomserver APIs (#994)
* Consolidation of roomserver APIs

* Comment out alias tests for now, they are broken

* Wire AS API into roomserver again

* Roomserver didn't take asAPI param before so return to that

* Prevent roomserver asking AS API for alias info

* Rename some files

* Remove alias_test, incoherent tests and unwanted appservice integration

* Remove FS API inject on syncapi component
2020-05-01 10:48:17 +01:00
Neil Alexander
3ab8ebf6b8
More invite support (#979)
* Update gomatixserverlib

* Try to build invite stripped state if not given to us

* SendInvite improvements

* Transpose invite_room_state into invite_state.events for sync API

* Remove syncapi debugging output

* Use RespInviteV2

* Update gomatrixserverlib

* Send the invite event as a normal roomserver event too, for incorporating into room (should this be done by the roomserver automatically for invite inputs?)

* Federation sender use invite_room_state, room server try to insert membership state

* Check supported room versions on the invite endpoint

* Prevent roomserver query API from trying to handle requests for stub rooms

* Adding a nolint

* Replace IsRoomStub with RoomNIDExcludingStubs, fix query API to use that instead

* Review comments
2020-04-24 16:30:25 +01:00
Neil Alexander
c30b12b5a1
Fix sarama import URLs (#856)
* Fix sarama import URLs

* Update gomatrixserverlib

* Update naffka

* Update naffka

* Update in kafka-producer
2020-04-22 15:26:56 +01:00
Neil Alexander
067b875063
Invites v2 endpoint (#952)
* Start converting v1 invite endpoint to v2

* Update gomatrixserverlib

* Early federationsender code for sending invites

* Sending invites sorta happens now

* Populate invite request with stripped state

* Remodel a bit, don't reflect received invites

* Handle invite_room_state

* Handle room versions a bit better

* Update gomatrixserverlib

* Tweak order in destinationQueue.next

* Revert check in processMessage

* Tweak federation sender destination queue code a bit

* Add comments
2020-04-03 14:29:06 +01:00
Kegsay
8fbe9f4078
Implement typing over federation (#949)
Also fix a pet peeve of mine: not putting units on things!!!

Manually tested on p2p and works well, with some fudge factor delay.
2020-03-30 16:40:28 +01:00
Kegsay
11a8059bba
Rename the typing server to EDU server (#948)
* Blunt move and sed rename

* Rename common/ refs to typing

* Rename internal stuff in eduserver

* Rename docs and scripts

* Rename constants/filenames, goimports everything to re-order imports
2020-03-30 15:02:20 +01:00
Neil Alexander
05e1ae8745
Further room version wiring (#936)
* Room version 2 by default, other wiring updates, update gomatrixserverlib

* Fix nil pointer exception

* Fix some more nil pointer exceptions hopefully

* Update gomatrixserverlib

* Send all room versions when joining, not just stable ones

* Remove room version cquery

* Get room version when getting events from the roomserver database

* Reset default back to room version 2

* Don't generate event IDs unless needed

* Revert "Remove room version cquery"

This reverts commit a170d5873360dd059614460acc8b21ab2cda9767.

* Query room version in federation API, client API as needed

* Improvements to make_join send_join dance

* Make room server producers use headered events

* Lint tweaks

* Update gomatrixserverlib

* Versioned SendJoin

* Query room version in syncapi backfill

* Handle transaction marshalling/unmarshalling within Dendrite

* Sorta fix federation (kinda)

* whoops commit federation API too

* Use NewEventFromTrustedJSON when getting events from the database

* Update gomatrixserverlib

* Strip headers on federationapi endpoints

* Fix bug in clientapi profile room version query

* Update gomatrixserverlib

* Return more useful error if room version query doesn't find the room

* Update gomatrixserverlib

* Update gomatrixserverlib

* Maybe fix federation

* Fix formatting directive

* Update sytest whitelist and blacklist

* Temporarily disable room versions 3 and 4 until gmsl is fixed

* Fix count of EDUs in logging

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib

* Rely on EventBuilder in gmsl to generate the event IDs for us

* Some review comments fixed

* Move function out of common and into gmsl

* Comment in federationsender destinationqueue

* Update gomatrixserverlib
2020-03-27 16:28:22 +00:00
Neil Alexander
aebf347a79
Implement gomatrixserverlib.HeaderedEvent in roomserver Kafka output (#914)
* Use Event.Headered

* Use HeaderedEvent in roomserver kafka output

* Fix syncserver-integration-tests

* Update producers to roomserver inputs

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib
2020-03-17 11:01:25 +00:00
Neil Alexander
acb505b717
Implement gomatrixserverlib.HeaderedEvent in roomserver query API (#912)
* Implement gomatrixserverlib.HeaderedEvent, which should allow us to store room version headers along with the event across API boundaries and consumers/producers, and intercept unmarshalling to get the event structure right

* Add federationsender to previous
2020-03-16 17:29:52 +00:00
Neil Alexander
c28577ea25
Implement storage interfaces (#841)
* Implement interfaces for federationsender storage

* Implement interfaces for mediaapi storage

* Implement interfaces for publicroomsapi storage

* Implement interfaces for roomserver storage

* Implement interfaces for syncapi storage

* Implement interfaces for keydb storage

* common.PartitionStorer in publicroomsapi interface

* Update copyright notices
2020-01-03 14:07:05 +00:00
Alex Chen
f8d2860765 Replace membership and visibility values with constants (#774)
Signed-off-by: Alex Chen <minecnly@gmail.com>
2019-08-06 15:07:36 +01:00
ruben
74827428bd use go module for dependencies (#594) 2019-05-21 21:56:55 +01:00