Commit Graph

1926 Commits

Author SHA1 Message Date
Neil Alexander
5dd5a41119
Tweak log levels of some federation logging (#1248)
* Tweak log levels of some federation logging

* Update go.mod/go.sum for matrix-org/util#22 and matrix-org/gomatrixserverlib#215
2020-08-07 15:00:23 +01:00
Neil Alexander
6ce7af8a3e
Update go.mod/go.sum for matrix-org/gomatrixserverlib#213 2020-08-07 13:28:09 +01:00
Neil Alexander
ee22c6e440
Update go.mod/go.sum for matrix-org/gomatrixserverlib#212 2020-08-07 10:38:06 +01:00
Kegsay
32a4565b55
Add device list updater which manages updating remote device lists (#1242)
* Add device list updater which manages updating remote device lists

- Doesn't persist stale lists to the database yet
- Doesn't have tests yet

* Mark device lists as fresh when we persist
2020-08-06 17:48:10 +01:00
Neil Alexander
b7491aae03
Yggdrasil demo updates (#1241)
* PerformServersAlive in PerformBroadcastEDU

* Don't double-pointer

* More reliable QUIC session handling

* Direct peer lookup, other tweaks

* Tweaks

* Try to wake up queues on incoming QUIC session

* Set session callbak on gobind build

* Fix incoming session storage

* Stateless reset, other tweaks

* Reset sessions when coordinates change

* Disable HTTP connection reuse, tweak timeouts
2020-08-06 16:00:42 +01:00
Kegsay
642f9cb964
Process inbound device list updates from federation (#1240)
* Add InputDeviceListUpdate

* Unbreak unit tests

* Process inbound device list updates from federation

- Persist the keys in the keyserver and produce key changes
- Does not currently fetch keys from the remote server if the prev IDs are missing

* Linting
2020-08-05 13:41:16 +01:00
Neil Alexander
15dc1f4d03
Use TransactionWriter in SQLite keyserver (#1239)
* Use TransactionWriter in SQLite keyserver

* Fix keyserver storage tests
2020-08-05 11:01:37 +01:00
Neil Alexander
22f028e141
SelectJoinedHostsForRooms should use QueryVariadic on SQLite (#1238)
* SelectJoinedHostsForRooms should use QueryVariadic on SQLite

* Fix strings.Replace

* Fix statement
2020-08-05 10:00:35 +01:00
Kegan Dougal
2197e54441 Update whitelist 2020-08-04 13:50:32 +01:00
Kegan Dougal
3c8e9d1d20 Bump to gmsl version which fixes outbound fed invites to synapse 2020-08-04 13:48:57 +01:00
Kegan Dougal
8ee4c0f71e Always set invite_room_state else synapse rejects invites 2020-08-04 12:14:18 +01:00
Kegan Dougal
4cf7c08146 Fix response format for federation /devices requests 2020-08-04 12:02:13 +01:00
Kegan Dougal
78ab33f91f Unbreak postgres 2020-08-04 11:41:48 +01:00
Kegsay
0c4e8f6d4f
Send device list updates to servers (outbound only) (#1237)
* Add QueryDeviceMessages to serve up device keys and stream IDs

* Consume key change events in fedsender

Don't yet send them to destinations as we haven't worked them out yet

* Send device list updates to all required servers

* Glue it all together
2020-08-04 11:32:14 +01:00
Kegsay
fb56bbf0b7
Generate stream IDs for locally uploaded device keys (#1236)
* Breaking: add stream_id to keyserver_device_keys table

* Add tests for stream ID generation

* Fix whitelist
2020-08-03 17:07:06 +01:00
Kegsay
ffcb6d2ea1
Produce OTK counts in /sync response (#1235)
* Add QueryOneTimeKeys for /sync extensions

* Unbreak tests

* Produce OTK counts in /sync response

* Linting
2020-08-03 12:29:58 +01:00
Kegsay
b5cb1d1534
Fix edge cases around device lists (#1234)
* Fix New users appear in /keys/changes

* Create blank device keys when logging in on a new device

* Add PerformDeviceUpdate and fix a few bugs

- Correct device deletion query on sqlite
- Return no keys on /keys/query rather than an empty key

* Unbreak sqlite properly

* Use a real DB for currentstateserver integration tests

* Race fix
2020-07-31 14:40:45 +01:00
Kegsay
a7e67e65a8
Notify clients when devices are deleted (#1233)
* Recheck device lists when join/leave events come in

* Add PerformDeviceDeletion

* Notify clients when devices are deleted

* Unbreak things

* Remove debug logging
2020-07-30 18:00:56 +01:00
Neil Alexander
292a9ddd82
Fix Yggdrasil demo builds 2020-07-30 15:47:44 +01:00
Kegsay
a2174d3294
Implement /keys/changes (#1232)
* Implement /keys/changes

And refactor QueryKeyChanges to accept a `to` offset.

* Unbreak tests

* Sort keys when serialising log tokens
2020-07-30 14:52:21 +01:00
Kegsay
9355fb5ac8
Hook up device list updates to the sync notifier (#1231)
* WIP hooking up key changes

* Fix import cycle, get tests passing and binary compiling

* Linting and update whitelist
2020-07-30 11:15:46 +01:00
Kegsay
0fdd4f14d1
Add support for logs in StreamingToken (#1229)
* Add support for logs in StreamingToken

Tokens now end up looking like `s11_22|dl-0-123|ab-0-12224`
where `dl` and `ab` are log names, `0` is the partition and
`123` and `12224` are the offsets.

* Also test reserialisation

* s/|/./g so tokens url escape nicely
2020-07-29 19:00:04 +01:00
Kegsay
9a5fb489c5
Add QueryKeyChanges (#1228)
Hook some things up to call it as well.
2020-07-28 18:25:16 +01:00
Kegsay
adf7b59294
Persist partition|offset|user_id in the keyserver (#1226)
* Persist partition|offset|user_id in the keyserver

Required for a query API which will be used by the syncapi which
will be called when a `/sync` request comes in which will return
a list of user IDs of people who have changed their device keys
between two tokens.

* Add tests and fix maxOffset bug

* s/offset/log_offset/g because 'offset' is a reserved word in postgres
2020-07-28 17:38:30 +01:00
Neil Alexander
acc8e80a51
User directory (#1225)
* User directory

* Fix syncapi unit test

* Make user directory only show remote users you know about from your joined rooms

* Update sytest-whitelist

* Review comments
2020-07-28 10:53:17 +01:00
Kegsay
c632867135
Modify /state/{eventType}/{stateKey} to return the event at the time the user left (#1222)
* Modify /state/{eventType}/{stateKey} to return the event at the time the user left

Or live, depending on their current state. Hopefully fixes some sytests!

* Linting

* Set HasBeenInRoom

* Fix cases for world-readable history visibility

* Fix bug in finding the requested state event

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-07-28 10:09:10 +01:00
Henrik Sölver
83f038e12b
Don't use more than 999 variables in SQLite querys. (#1224)
Closes #1223

Signed-off-by: Henrik Sölver <henrik.solver@gmail.com>

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-07-27 13:19:30 +01:00
Kegsay
c8d476a3cc
Return HTTP errors when trying to kick invalid users (#1221)
Room integrity was never compromised as GMSL does auth checks,
but we would incorrectly 200 OK the request instead of 403ing.
2020-07-27 09:20:09 +01:00
Kegsay
61963a74ae
Add logic for determining when device lists have changed due to membership changes (#1220)
* Flesh out structure for handling device list updates for room membership changes

* First cut untested algorithm

* Add tests for determining changed/left device lists

* Linting

* Unbreak tests

* Sigh.. linting
2020-07-27 09:19:55 +01:00
Kegan Dougal
abef9bc04f Put this in the right category 2020-07-24 17:11:16 +01:00
Kegan Dougal
b63fa7b880 Update AWSY list 2020-07-24 16:17:33 +01:00
Kegsay
af5b4d1f6b
Modify QuerySharedUsers to handle counts/include/exclude (#1219)
* Modify QuerySharedUsers to handle counts/include/exclude

We will need this functionality when working out whether to
send device list changes to users who have joined/left a room.

* Linting
2020-07-24 10:33:41 +01:00
Kegsay
98f2f09bb4
keyserver: produce key change events (#1218)
* Produce kafka events when keys are added

* Consume key changes in syncapi with TODO markers for handling them and catching up

* unbreak tests

* Linting
2020-07-23 16:41:36 +01:00
Kegsay
7b862384a7
currentstate: Add QuerySharedUsers (#1217)
This will be used to determine who to send device list updates to. It
can also be used to determine who to send presence info to.
2020-07-23 12:26:31 +01:00
Neil Alexander
cfeb1b2f42
Add UNIQUE constraint to blacklist table (#1216) 2020-07-23 10:22:23 +01:00
Kegan Dougal
9684efebd7 Fix gobind 2020-07-22 17:06:19 +01:00
Kegsay
541a23f712
Handle inbound federation E2E key queries/claims (#1215)
* Handle inbound /keys/claim and /keys/query requests

* Add display names to device key responses

* Linting
2020-07-22 17:04:57 +01:00
Neil Alexander
1e71fd645e
Persistent federation sender blacklist (#1214)
* Initial persistence of blacklists

* Move statistics folder

* Make MaxFederationRetries configurable

* Set lower failure thresholds for Yggdrasil demos

* Still write events into database for blacklisted hosts (they can be tidied up later)

* Review comments
2020-07-22 17:01:29 +01:00
Kegsay
470933789b
Perform outbound federation hits for querying/claiming E2E keys (#1212)
* Perform outbound federation hits for querying/claiming E2E keys

Untested currently because we need the receiving end to work
before sytest will be happy.

* Linting
2020-07-21 17:46:47 +01:00
Neil Alexander
f54689f862
Fix lint error 2020-07-21 17:21:48 +01:00
Neil Alexander
b6bc132485
Use TransactionWriter in other component SQLite (#1209)
* Use TransactionWriter on other component SQLites

* Fix sync API tests

* Fix panic in media API

* Fix a couple of transactions

* Fix wrong query, add some logging output

* Add debug logging into StoreEvent

* Adjust InsertRoomNID

* Update logging
2020-07-21 15:48:21 +01:00
Kegsay
1d72ce8b7a
Implement claiming one-time keys locally (#1210)
* Add API shape for claiming keys

* Implement claiming one-time keys locally

Fairly boring, nothing too special going on.
2020-07-21 14:47:53 +01:00
Neil Alexander
d76eb1b994
Use TransactionWriter in roomserver SQLite (#1208) 2020-07-21 10:48:49 +01:00
Neil Alexander
489f34fed7
Remove debug lines 2020-07-20 17:03:20 +01:00
Neil Alexander
11a39fe3b5
Deduplicate FS database, EDU persistence table (#1207)
* Deduplicate FS database, add some EDU persistence groundwork

* Extend TransactionWriter to use optional existing transaction, use that for FS SQLite database writes

* Fix build due to bad keyserver import

* Working EDU persistence

* gocyclo, unsurprisingly

* Remove unused

* Update copyright notices
2020-07-20 16:55:20 +01:00
Creed Haymond
f3c482b078
Adding the e2ekey database to the create_db script and docs (#1206)
Signed-off-by: Creed Haymond <creedasaurus@gmail.com>
2020-07-20 09:18:48 +01:00
Neil Alexander
986498f5a5
Don't block Yggdrasil switch 2020-07-16 15:19:26 +01:00
Neil Alexander
fbf139f20e
Small fix to gobind build 2020-07-16 14:26:04 +01:00
Neil Alexander
e5208c2ec9
Yggdrasil demo updates ("Bare QUIC")
Squashed commit of the following:

commit 86c2388e13ffdbabdd50cea205652dccc40e1860
Merge: b0a3ee6c f5e7e751
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Jul 16 13:47:10 2020 +0100

    Merge branch 'master' into neilalexander/yggbarequic

commit b0a3ee6c5c063962384bb91c59ec753ddc8cfe5f
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Jul 16 13:42:22 2020 +0100

    Add support for broadcasting wake-up EDUs to known hosts

commit 8a5c2020b3a4b705b5d5686a9e71990a49e6d471
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Jul 16 13:42:10 2020 +0100

    Bare QUIC demo working

commit d3939b3d6568cf4262c0391486a5203873b68bfc
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Jul 15 11:42:43 2020 +0100

    Support bare Yggdrasil sessions with encrypted QUIC
2020-07-16 13:52:08 +01:00
Kegsay
f5e7e7513c
Implement /keys/query locally (#1204)
* Implement /keys/query locally

* Fix sqlite tests and close rows
2020-07-15 18:40:41 +01:00