We did this already for local `/keys/upload` but didn't for
remote `/users/devices`. This meant any resyncs would spam produce
events, hammering disk i/o and spamming the logs.
* Make PerformJoin send input membership event
* Invite input room events in separate goroutine
* Don't limit roomserver input events using request context
* Synchronous input room events
* Nope, that didn't work
* oops send state key to GetMembership
* Don't generate stripped state in client API more times than necessary, generate output events on receiving end of federated invite
* Commit membership updater changes
* Tweaks
Really we should never have put the AS API bits under CS API to
begin with, as it is a completely different versioned spec: split
it out (which bumps the CS API % slightly).
Also, split out Device Keys into 2 extra sections: Device Key Backup
and Cross-signing Keys. The point of AWSY is for it to be easy for
anyone to see which features are implemented in Dendrite, and bundling
three distinct features under the umbrella of Device Keys was just
unclear.
- As a last resort, query the DB when exhausting all possible remote query
endpoints, but keep the field in `failures` so clients can detect that this
is stale data.
- Unblock `DeviceListUpdater.Update` on failures rather than timing out.
- Use a mutex when writing directly to `res`, not just for failures.
* WIP: Eagerly sync device lists on /user/keys/query requests
Also notify servers when a user's device display name changes. Few
caveats:
- sytest `Device deletion propagates over federation` fails
- `populateResponseWithDeviceKeysFromDatabase` is called from multiple
goroutines and hence is unsafe.
* Handle deleted devices correctly over federation
* Add sync mechanism to block when updating device lists
With a timeout, mainly for sytest to fix the test
"Server correctly handles incoming m.device_list_update"
which is flakey because it assumes that when `/send` 200 OKs
that the server has updated the device lists in prep for
`/keys/query` which is not always true when using workers.
* Fix UT
* Add new working test
* First pass at server ACLs (not efficient)
* Use transaction origin, update whitelist
* Fix federation API test
It's sufficient for us to return nothing in response to current state, so that the server ACL check returns no ACLs.
* More efficient server ACLs - hopefully
* Fix queries
* Fix queries
* Avoid panics by nil pointers
* Bug fixes
* Fix state event type
* Fix mutex
* Update logging
* Ignore port when matching servername
* Use read mutex
* Fix bugs
* Fix sync API test
* Comments
* Add tests, tweaks to behaviour
* Fix test output
* Initial pass at refactoring config (not finished)
* Don't forget current state and EDU servers
* More shifting around
* Update server key API tests
* Fix roomserver test
* Fix more tests
* Further tweaks
* Fix current state server test (sort of)
* Maybe fix appservices
* Fix client API test
* Include database connection string in database options
* Fix sync API build
* Update config test
* Fix unit tests
* Fix federation sender build
* Fix gobind build
* Set Listen address for all services in HTTP monolith mode
* Validate config, reinstate appservice derived in directory, tweaks
* Tweak federation API test
* Set MaxOpenConnections/MaxIdleConnections to previous values
* Update generate-config
* Add tests for device list updates
* Add stale_device_lists table and use db before asking remote for device keys
* Fetch remote keys if all devices are requested
* Add display_name col to store remote device names
Few other tweaks to make `Server correctly handles incoming m.device_list_update`
pass.
* Fix sqlite otk bug
* Unbuffered channel to block /send causing sytest to not race anymore
* Linting and fix bug whereby we didn't send updated dl tokens to the client causing a tightloop on /sync sometimes
* No longer assert staleness as Update blocks on workers now
* Back out tweaks
* Bugfixes
* Add device list updater which manages updating remote device lists
- Doesn't persist stale lists to the database yet
- Doesn't have tests yet
* Mark device lists as fresh when we persist
* Add InputDeviceListUpdate
* Unbreak unit tests
* Process inbound device list updates from federation
- Persist the keys in the keyserver and produce key changes
- Does not currently fetch keys from the remote server if the prev IDs are missing
* Linting
* Add QueryDeviceMessages to serve up device keys and stream IDs
* Consume key change events in fedsender
Don't yet send them to destinations as we haven't worked them out yet
* Send device list updates to all required servers
* Glue it all together
* Fix New users appear in /keys/changes
* Create blank device keys when logging in on a new device
* Add PerformDeviceUpdate and fix a few bugs
- Correct device deletion query on sqlite
- Return no keys on /keys/query rather than an empty key
* Unbreak sqlite properly
* Use a real DB for currentstateserver integration tests
* Race fix
* Recheck device lists when join/leave events come in
* Add PerformDeviceDeletion
* Notify clients when devices are deleted
* Unbreak things
* Remove debug logging
* Add support for logs in StreamingToken
Tokens now end up looking like `s11_22|dl-0-123|ab-0-12224`
where `dl` and `ab` are log names, `0` is the partition and
`123` and `12224` are the offsets.
* Also test reserialisation
* s/|/./g so tokens url escape nicely
* Persist partition|offset|user_id in the keyserver
Required for a query API which will be used by the syncapi which
will be called when a `/sync` request comes in which will return
a list of user IDs of people who have changed their device keys
between two tokens.
* Add tests and fix maxOffset bug
* s/offset/log_offset/g because 'offset' is a reserved word in postgres
* User directory
* Fix syncapi unit test
* Make user directory only show remote users you know about from your joined rooms
* Update sytest-whitelist
* Review comments
* Modify /state/{eventType}/{stateKey} to return the event at the time the user left
Or live, depending on their current state. Hopefully fixes some sytests!
* Linting
* Set HasBeenInRoom
* Fix cases for world-readable history visibility
* Fix bug in finding the requested state event
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
