* Add support for database migrations
Closes#1246
This PR does NOT add any migrations as an example. I have
manually tested that the library works with SQL and Go based
upgrades correctly. Documentation should be sufficient for
devs to add migrations.
* Clarifications
* Linting
* Use state from RespSendJoin post-check
* Don't create input events for invalid events
* Let's try this again
* Update gomatrixserverlib
* Update gomatrixserverlib to matrix-org/gomatrixserverlib@38f437f
* Use federation sender for backfill and getting missing events
* Fix internal URL paths
* Update go.mod/go.sum for matrix-org/gomatrixserverlib#218
* Add missing server implementations in HTTP interface
* Add FederationClient interface to federationsender
- Use a shim struct in HTTP mode to keep the same API as `FederationClient`.
- Use `federationsender` instead of `FederationClient` in `keyserver`.
* Pointers not values
* Review comments
* Fix unit tests
* Rejig backoff
* Unbreak test
* Remove debug logs
* Review comments and linting
* Initial pass at refactoring config (not finished)
* Don't forget current state and EDU servers
* More shifting around
* Update server key API tests
* Fix roomserver test
* Fix more tests
* Further tweaks
* Fix current state server test (sort of)
* Maybe fix appservices
* Fix client API test
* Include database connection string in database options
* Fix sync API build
* Update config test
* Fix unit tests
* Fix federation sender build
* Fix gobind build
* Set Listen address for all services in HTTP monolith mode
* Validate config, reinstate appservice derived in directory, tweaks
* Tweak federation API test
* Set MaxOpenConnections/MaxIdleConnections to previous values
* Update generate-config
* Add device list updater which manages updating remote device lists
- Doesn't persist stale lists to the database yet
- Doesn't have tests yet
* Mark device lists as fresh when we persist
* Add QueryDeviceMessages to serve up device keys and stream IDs
* Consume key change events in fedsender
Don't yet send them to destinations as we haven't worked them out yet
* Send device list updates to all required servers
* Glue it all together
* Perform outbound federation hits for querying/claiming E2E keys
Untested currently because we need the receiving end to work
before sytest will be happy.
* Linting
Squashed commit of the following:
commit 86c2388e13ffdbabdd50cea205652dccc40e1860
Merge: b0a3ee6c f5e7e751
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Thu Jul 16 13:47:10 2020 +0100
Merge branch 'master' into neilalexander/yggbarequic
commit b0a3ee6c5c063962384bb91c59ec753ddc8cfe5f
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Thu Jul 16 13:42:22 2020 +0100
Add support for broadcasting wake-up EDUs to known hosts
commit 8a5c2020b3a4b705b5d5686a9e71990a49e6d471
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Thu Jul 16 13:42:10 2020 +0100
Bare QUIC demo working
commit d3939b3d6568cf4262c0391486a5203873b68bfc
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Wed Jul 15 11:42:43 2020 +0100
Support bare Yggdrasil sessions with encrypted QUIC
* Add a bit more logging to the fedsender
* bugfix: continue sending PDUs if ones are added whilst sending another PDU
Without this, the queue goes back to sleep on `<-oq.notifyPDUs` which won't
fire because `pendingPDUs` is already > 0. This should fix a flakey sytest.
* Break if no txn is sent
* WIP syncapi work
* More debugging
* Bump GMSL version to pull in working Event.Redact
* Remove logging
* Make redactions work on v3+
* Fix more tests
* Emit redacted_event from the roomserver when redactions are validated
- Consume them in the currentstateserver and act accordingly.
- Add integration test for the roomserver to check that injecting
`m.room.redaction` events result in `redacted_event` being emitted.
* Linting
* Ignore events that redact themselves
* Don't return null to public directory request
* Initial support for finding public rooms in Yggdrasil demo (incomplete)
* Increase QUIC idle time to 15 minutes
* Initial QUIC work
* Update Yggdrasil demo
* Make sure that the federation sender knows how many pending events are in the database when the worker starts
* QUIC tunables
* pprof
* Don't spin
* Set build info for Yggdrasil
* Use content_value instead of membership
* Fix build
* Replace publicroomsapi with a combination of clientapi/roomserver/currentstateserver
- All public rooms paths are now handled by clientapi
- Requests to (un)publish rooms are sent to the roomserver via `PerformPublish`
which are stored in a new `published_table.go`
- Requests for public rooms are handled in clientapi by:
* Fetch all room IDs which are published using `QueryPublishedRooms` on the roomserver.
* Apply pagination parameters to the slice.
* Do a `QueryBulkStateContent` request to the currentstateserver to pull out
required state event *content* (not entire events).
* Aggregate and return the chunk.
Mostly but not fully implemented (DB queries on currentstateserver are missing)
* Fix pq query
* Make postgres work
* Make sqlite work
* Fix tests
* Unbreak pagination tests
* Linting
* Return remote errors from FS.PerformJoin
Follows the same pattern as PerformJoin on roomserver (no error return).
Also return the right format for incompatible room version errors.
Makes a bunch of tests pass!
* Handle network errors better when returning remote HTTP errors
* Linting
* Fix tests
* Update whitelist, pass network errors through in API=1 mode
* More key tweaks
* Start testing stuff
* Move responsibility for generating local keys into server key API, don't register prom in caches unless needed, start tests
* Don't store our own keys in the database
* Don't store our own keys in the database
* Don't run tests for now
* Tweak caching behaviour, update tests
* Update comments, add fixes from forward-merge
* Debug logging
* Debug logging
* Perform final comparison against original set of requests
* oops
* Fetcher timeouts
* Fetcher timeouts
* missing func
* Tweaks
* Update gomatrixserverlib
* Fix Federation API test
* Break up FetchKeys
* Add comments to caching
* Add URL check in test
* Partially revert "Move responsibility for generating local keys into server key API, don't register prom in caches unless needed, start tests"
This reverts commit d7eb54c5b30b2f6a9d6514b643e32e6ad2b602f3.
* Fix federation API test
* Fix internal cache stuff again
* Fix server key API test
* Update comments
* Update comments from review
* Fix lint
* Use MissingAuthEventHandler on performjoin to try and speed up cases where we have missing events
* Update gomatrixserverlib
* Use supplied room version
* Use AuthChainProvider
* Tweaks
* Update gomatrixserverlib
* Signature checks
* Return bad request on CS API /send if bad JSON
* Return some more M_BAD_JSON in the right places
* nolint because damnit gocyclo all I added was a type check for an error
* Update gomatrixserverlib
* Update gomatrixserverlib
* Update sytest-whitelist
* Update gomatrixserverlib
* Update sytest-whitelist
* NotJSON -> BadJSON
