Commit Graph

81 Commits

Author SHA1 Message Date
Neil Alexander
4b09f445c9
Configuration format v1 (#1230)
* Initial pass at refactoring config (not finished)

* Don't forget current state and EDU servers

* More shifting around

* Update server key API tests

* Fix roomserver test

* Fix more tests

* Further tweaks

* Fix current state server test (sort of)

* Maybe fix appservices

* Fix client API test

* Include database connection string in database options

* Fix sync API build

* Update config test

* Fix unit tests

* Fix federation sender build

* Fix gobind build

* Set Listen address for all services in HTTP monolith mode

* Validate config, reinstate appservice derived in directory, tweaks

* Tweak federation API test

* Set MaxOpenConnections/MaxIdleConnections to previous values

* Update generate-config
2020-08-10 14:18:04 +01:00
Kegan Dougal
366fd975c8 Fix unit tests 2020-08-07 17:58:44 +01:00
Kegsay
f371783da7
Finish inbound E2E device lists (#1243)
* Add tests for device list updates

* Add stale_device_lists table and use db before asking remote for device keys

* Fetch remote keys if all devices are requested

* Add display_name col to store remote device names

Few other tweaks to make `Server correctly handles incoming m.device_list_update`
pass.

* Fix sqlite otk bug

* Unbuffered channel to block /send causing sytest to not race anymore

* Linting and fix bug whereby we didn't send updated dl tokens to the client causing a tightloop on /sync sometimes

* No longer assert staleness as Update blocks on workers now

* Back out tweaks

* Bugfixes
2020-08-07 17:32:13 +01:00
Kegsay
32a4565b55
Add device list updater which manages updating remote device lists (#1242)
* Add device list updater which manages updating remote device lists

- Doesn't persist stale lists to the database yet
- Doesn't have tests yet

* Mark device lists as fresh when we persist
2020-08-06 17:48:10 +01:00
Kegsay
642f9cb964
Process inbound device list updates from federation (#1240)
* Add InputDeviceListUpdate

* Unbreak unit tests

* Process inbound device list updates from federation

- Persist the keys in the keyserver and produce key changes
- Does not currently fetch keys from the remote server if the prev IDs are missing

* Linting
2020-08-05 13:41:16 +01:00
Neil Alexander
15dc1f4d03
Use TransactionWriter in SQLite keyserver (#1239)
* Use TransactionWriter in SQLite keyserver

* Fix keyserver storage tests
2020-08-05 11:01:37 +01:00
Kegsay
0c4e8f6d4f
Send device list updates to servers (outbound only) (#1237)
* Add QueryDeviceMessages to serve up device keys and stream IDs

* Consume key change events in fedsender

Don't yet send them to destinations as we haven't worked them out yet

* Send device list updates to all required servers

* Glue it all together
2020-08-04 11:32:14 +01:00
Kegsay
fb56bbf0b7
Generate stream IDs for locally uploaded device keys (#1236)
* Breaking: add stream_id to keyserver_device_keys table

* Add tests for stream ID generation

* Fix whitelist
2020-08-03 17:07:06 +01:00
Kegsay
ffcb6d2ea1
Produce OTK counts in /sync response (#1235)
* Add QueryOneTimeKeys for /sync extensions

* Unbreak tests

* Produce OTK counts in /sync response

* Linting
2020-08-03 12:29:58 +01:00
Kegsay
b5cb1d1534
Fix edge cases around device lists (#1234)
* Fix New users appear in /keys/changes

* Create blank device keys when logging in on a new device

* Add PerformDeviceUpdate and fix a few bugs

- Correct device deletion query on sqlite
- Return no keys on /keys/query rather than an empty key

* Unbreak sqlite properly

* Use a real DB for currentstateserver integration tests

* Race fix
2020-07-31 14:40:45 +01:00
Kegsay
a7e67e65a8
Notify clients when devices are deleted (#1233)
* Recheck device lists when join/leave events come in

* Add PerformDeviceDeletion

* Notify clients when devices are deleted

* Unbreak things

* Remove debug logging
2020-07-30 18:00:56 +01:00
Kegsay
a2174d3294
Implement /keys/changes (#1232)
* Implement /keys/changes

And refactor QueryKeyChanges to accept a `to` offset.

* Unbreak tests

* Sort keys when serialising log tokens
2020-07-30 14:52:21 +01:00
Kegsay
9355fb5ac8
Hook up device list updates to the sync notifier (#1231)
* WIP hooking up key changes

* Fix import cycle, get tests passing and binary compiling

* Linting and update whitelist
2020-07-30 11:15:46 +01:00
Kegsay
9a5fb489c5
Add QueryKeyChanges (#1228)
Hook some things up to call it as well.
2020-07-28 18:25:16 +01:00
Kegsay
adf7b59294
Persist partition|offset|user_id in the keyserver (#1226)
* Persist partition|offset|user_id in the keyserver

Required for a query API which will be used by the syncapi which
will be called when a `/sync` request comes in which will return
a list of user IDs of people who have changed their device keys
between two tokens.

* Add tests and fix maxOffset bug

* s/offset/log_offset/g because 'offset' is a reserved word in postgres
2020-07-28 17:38:30 +01:00
Kegsay
98f2f09bb4
keyserver: produce key change events (#1218)
* Produce kafka events when keys are added

* Consume key changes in syncapi with TODO markers for handling them and catching up

* unbreak tests

* Linting
2020-07-23 16:41:36 +01:00
Kegsay
541a23f712
Handle inbound federation E2E key queries/claims (#1215)
* Handle inbound /keys/claim and /keys/query requests

* Add display names to device key responses

* Linting
2020-07-22 17:04:57 +01:00
Kegsay
470933789b
Perform outbound federation hits for querying/claiming E2E keys (#1212)
* Perform outbound federation hits for querying/claiming E2E keys

Untested currently because we need the receiving end to work
before sytest will be happy.

* Linting
2020-07-21 17:46:47 +01:00
Kegsay
1d72ce8b7a
Implement claiming one-time keys locally (#1210)
* Add API shape for claiming keys

* Implement claiming one-time keys locally

Fairly boring, nothing too special going on.
2020-07-21 14:47:53 +01:00
Neil Alexander
11a39fe3b5
Deduplicate FS database, EDU persistence table (#1207)
* Deduplicate FS database, add some EDU persistence groundwork

* Extend TransactionWriter to use optional existing transaction, use that for FS SQLite database writes

* Fix build due to bad keyserver import

* Working EDU persistence

* gocyclo, unsurprisingly

* Remove unused

* Update copyright notices
2020-07-20 16:55:20 +01:00
Kegsay
f5e7e7513c
Implement /keys/query locally (#1204)
* Implement /keys/query locally

* Fix sqlite tests and close rows
2020-07-15 18:40:41 +01:00
Kegsay
9dd2ed7f65
Implement key uploads (#1202)
* Add storage layer for postgres/sqlite

* Return OTK counts when inserting new keys

* Hook up the key DB and make a test pass

* Convert postgres queries to be sqlite queries

* Blacklist test due to requiring rejected events

* Unbreak tests

* Update blacklist
2020-07-15 12:02:34 +01:00
Kegsay
7daa3bf098
Implement logic for key uploads (#1197)
* begin work on storing keys

* Finish rough impl of the internal key API

* Linting
2020-07-14 12:59:07 +01:00
Kegsay
396219ef53
Add boilerplate for key server APIs (#1196)
Also add a README which outilnes how things will work.
2020-07-13 16:02:35 +01:00
Neil Alexander
ddf1c8adf1
Hacks for supporting Riot iOS (#1148)
* Join room body is optional

* Support deprecated login by user/password

* Implement dummy key upload endpoint

* Make a very determinate end to /messages if we hit the create event in back-pagination

* Linting
2020-06-17 17:41:45 +01:00
Kegsay
9c77022513
Make userapi responsible for checking access tokens (#1133)
* Make userapi responsible for checking access tokens

There's still plenty of dependencies on account/device DBs, but this
is a start. This is a breaking change as it adds a required config
value `listen.user_api`.

* Cleanup

* Review comments and test fix
2020-06-16 14:10:55 +01:00
Kegsay
ecd7accbad
Rehuffle where things are in the internal package (#1122)
renamed:    internal/eventcontent.go -> internal/eventutil/eventcontent.go
	renamed:    internal/events.go -> internal/eventutil/events.go
	renamed:    internal/types.go -> internal/eventutil/types.go
	renamed:    internal/http/http.go -> internal/httputil/http.go
	renamed:    internal/httpapi.go -> internal/httputil/httpapi.go
	renamed:    internal/httpapi_test.go -> internal/httputil/httpapi_test.go
	renamed:    internal/httpapis/paths.go -> internal/httputil/paths.go
	renamed:    internal/routing.go -> internal/httputil/routing.go
	renamed:    internal/basecomponent/base.go -> internal/setup/base.go
	renamed:    internal/basecomponent/flags.go -> internal/setup/flags.go
	renamed:    internal/partition_offset_table.go -> internal/sqlutil/partition_offset_table.go
	renamed:    internal/postgres.go -> internal/sqlutil/postgres.go
	renamed:    internal/postgres_wasm.go -> internal/sqlutil/postgres_wasm.go
	renamed:    internal/sql.go -> internal/sqlutil/sql.go
2020-06-12 14:55:57 +01:00
Kegsay
4f171c56a8
Split out SetupFooComponent (#1106)
* Split out adding HTTP routes from making internal APIs for clarity

* Split out more components

* Split out more things

* Finish converting

* internal mux for internal routes
2020-06-08 15:51:07 +01:00
Neil Alexander
fe82e1f725
Separate muxes for public and internal APIs (#1056)
* Separate muxes for public and internal APIs

* Update client-api-proxy and federation-api-proxy so they don't add /api to the path

* Tidy up

* Consistent HTTP setup

* Set up prefixes properly
2020-05-22 11:43:17 +01:00
Kegsay
24d8df664c
Fix #897 and shuffle directory around (#1054)
* Fix #897 and shuffle directory around

* Update find-lint

* goimports

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-05-21 14:40:13 +01:00
Neil Alexander
8adc128225
Keyserver skeleton (#1032)
* Keyserver skeleton

* Indentation
2020-05-14 14:05:14 +01:00