Commit Graph

972 Commits

Author SHA1 Message Date
Kegsay
0dc4ceaa2d
Minor perf/debugging improvements (#1121)
* Minor perf/debugging improvements

- publicroomsapi: Don't call QueryEventsByID with no event IDs
- appservice: Consume only if there are 1 or more ASes
- roomserver: don't keep a copy of the request "for debugging" - we trace now

* fedsender: return early if we have no destinations

* Unbreak tests
2020-06-12 15:11:33 +01:00
Kegsay
ecd7accbad
Rehuffle where things are in the internal package (#1122)
renamed:    internal/eventcontent.go -> internal/eventutil/eventcontent.go
	renamed:    internal/events.go -> internal/eventutil/events.go
	renamed:    internal/types.go -> internal/eventutil/types.go
	renamed:    internal/http/http.go -> internal/httputil/http.go
	renamed:    internal/httpapi.go -> internal/httputil/httpapi.go
	renamed:    internal/httpapi_test.go -> internal/httputil/httpapi_test.go
	renamed:    internal/httpapis/paths.go -> internal/httputil/paths.go
	renamed:    internal/routing.go -> internal/httputil/routing.go
	renamed:    internal/basecomponent/base.go -> internal/setup/base.go
	renamed:    internal/basecomponent/flags.go -> internal/setup/flags.go
	renamed:    internal/partition_offset_table.go -> internal/sqlutil/partition_offset_table.go
	renamed:    internal/postgres.go -> internal/sqlutil/postgres.go
	renamed:    internal/postgres_wasm.go -> internal/sqlutil/postgres_wasm.go
	renamed:    internal/sql.go -> internal/sqlutil/sql.go
2020-06-12 14:55:57 +01:00
Kegsay
4675e1ddb6
Add trace logging to RoomserverInternalAPI (#1120)
This is a wrapper around whatever impl we have which then logs
the function name/request/response/error.

Also tweak when we log on kafka streams: only log on the producer
side not the consumer side: we've never had issues with comms and
having 1 message rather than N would be nice.
2020-06-12 12:10:08 +01:00
Neil Alexander
079d8fe8fb
More key tweaks (#1116) 2020-06-12 11:07:26 +01:00
Kegsay
ec7718e7f8
Roomserver API changes (#1118)
* s/QueryBackfill/PerformBackfill/g

* OutputEvent now includes AddStateEvents which contain the full event of extra state events

* Only include adds not the current event

* Get adding state right
2020-06-11 19:50:40 +01:00
Kegsay
25cd2dd1c9
Remove unused internal APIs (#1117) 2020-06-11 15:07:16 +01:00
Neil Alexander
89d61c4877 Fix tiebreak in Yggdrasil demo (uses wrong key) 2020-06-11 13:08:11 +01:00
Neil Alexander
ce2517b9fb Support embedding Riot Web (#1113)
* Embed

    * Support for embedding is better now

    * HTTP on localhost for Ygg demo

    * Move embed into Yggdrasil demo
2020-06-11 10:16:46 +01:00
Neil Alexander
b7d0ca6855 Update Yggdrasil demo timeouts again 2020-06-11 09:50:54 +01:00
Neil Alexander
63a24e81c4 Yggdrasil demo tweaks 2020-06-10 17:18:37 +01:00
Kegsay
399b6ae334
Remove federationsender producer, which in fact was not a producer (#1115)
* Remove federationsender producer, which in fact was not a producer

* Set the signing struct
2020-06-10 16:54:43 +01:00
Neil Alexander
3b4be90000
Improvements to Yggdrasil demo (#1114)
* Improvements to Yggdrasil demo

* Fix missing copyright

* Fix tie-break
2020-06-10 16:29:02 +01:00
Kegsay
90a0aa9b3e
Remove public appservice routes (#1112) 2020-06-10 13:28:32 +01:00
Kegsay
b7187a9a35
Remove clientapi producers which aren't actually producers (#1111)
* Remove clientapi producers which aren't actually producers

They are actually just convenience wrappers around the internal APIs
for roomserver/eduserver. Move their logic to their respective `api`
packages and call them directly.

* Remove TODO

* unbreak ygg
2020-06-10 12:17:54 +01:00
Neil Alexander
d9d6f4568c
Use ServerKeyAPI as keyring (#1110)
* Use ServerKeyAPI as keyring

* Update fetcher name for debug logs
2020-06-10 11:02:03 +01:00
Neil Alexander
464718c3e6
Yggdrasil-based P2P demo (#1108)
* Initial work on Yggdrasil demo

* Muxing?

* Yamux

* Updates to yamux

* Updates

* Comments

* Update to use monolith stuff

* Update go.mod/go.sum

* Set defaults

* Tweaks

* Update yggdrasil

* Update config

* MarshalIndent

* Change default instance name/port

* add -peer switch

* gocyclo, for a change

* Determinate yamux roles

* Fix copyright notices

* Remove HTTP API checks as always false, remove unused topic
2020-06-10 10:54:06 +01:00
Kegsay
98cb0705ea
Remove unused UserUpdates producer (#1109) 2020-06-10 10:34:22 +01:00
Kegsay
85ac8a3f5b
Factor out how monolith routes get added (#1107)
Previously we had 3 monoliths:
 - dendrite-monolith-server
 - dendrite-demo-libp2p
 - dendritejs

which all had their own of setting up public routes. Factor this
out into a new `setup.Monolith` struct which gets all dependencies
set as fields. This is different to `basecomponent.Base` which
doesn't provide any way to set configured deps (e.g public rooms db)

Part of a larger process to clean up how we initialise Dendrite.
2020-06-09 12:07:33 +01:00
Kegsay
4f171c56a8
Split out SetupFooComponent (#1106)
* Split out adding HTTP routes from making internal APIs for clarity

* Split out more components

* Split out more things

* Finish converting

* internal mux for internal routes
2020-06-08 15:51:07 +01:00
Neil Alexander
cdb9a11571 Update gomatrixserverlib 2020-06-08 13:56:03 +01:00
Neil Alexander
67784ecb56
Nuke old Docker stuff (#1104)
* Remove old Docker stuff

* Move hub stuff to upper level docker directory

* Build monolith images

* Update README.md

* Update paths

* Update README.md
2020-06-05 19:00:30 +01:00
Neil Alexander
e7b19d2c70
More flexible caching (#1101) 2020-06-05 16:42:01 +01:00
Neil Alexander
76ff47c052
Use AuthChainProvider to try and speed up federated joins (#1100)
* Use MissingAuthEventHandler on performjoin to try and speed up cases where we have missing events

* Update gomatrixserverlib

* Use supplied room version

* Use AuthChainProvider

* Tweaks

* Update gomatrixserverlib

* Signature checks
2020-06-05 11:48:52 +01:00
Kegsay
29a20d1da7
General cleanup when making components (#1098)
* Remove ParseMonolith/LoadMonolith

* cleanup which components need to be made
2020-06-05 09:28:15 +01:00
Kegsay
2bd12f635c
Convert serverkeys to inthttp (#1097) 2020-06-04 16:26:35 +01:00
Kegsay
9834ac97db
Convert everything but serverkeyapi to inthttp (#1096)
* Convert roomserver to new inthttp format

* Convert eduserver to new inthttp format

* Convert appservice to new inthttp format
2020-06-04 15:43:07 +01:00
Neil Alexander
d785ad82b9
Add key validity fetching to server key API (#1094)
* Add key validity checks

* Store fetched keys

* Don't double-cache key results

* Perform server key API operations using new context

* Revert "Perform server key API operations using new context"

This reverts commit 02172223f5cb7850b0852c6ed6836ad82508ea76.

* Perform server key API operations using new context
2020-06-04 15:40:23 +01:00
Kegsay
f4c676ccdd
Refactor how federationsender gets created (#1095)
* Refactor how federationsender gets created

* s/httpint/inthttp/ for alphabetical niceness with internal package
2020-06-04 14:27:10 +01:00
Neil Alexander
f7025d3499 Update gomatrixserverlib 2020-06-04 13:30:08 +01:00
Kegsay
097cdf879c
s/Base64String/Base64Bytes/g (#1093)
* s/Base64String/Base64Bytes/g

Requires https://github.com/matrix-org/gomatrixserverlib/pull/203 to land first

* update gmsl
2020-06-04 11:50:57 +01:00
Kegsay
e7d1ac84c3
Add ParseFileURI and use it when dealing with file URIs (#1088)
* Add ParseFileURI and use it when dealing with file URIs

Fixes #1059

* Missing file

* Linting
2020-06-04 11:18:08 +01:00
Kegsay
d4f9a4bb97
Fix #632 and send spec-compliant transactions to the AS (#1091) 2020-06-04 11:17:37 +01:00
Kegsay
feb32ba365
Encode v3 event IDs correctly (#1090) 2020-06-04 11:14:08 +01:00
Neil Alexander
8c3f51d624
Update are-we-synapse-yet (#1092) 2020-06-04 11:13:40 +01:00
Neil Alexander
225b72bd42
Don't reset counters before successful outgoing federation request (#1089)
* Don't reset counters before successful outgoing federation request on incoming federation request

* Comments
2020-06-04 10:54:10 +01:00
Neil Alexander
8a6152ca70
Enable room version 6 (#1087)
* Return bad request on CS API /send if bad JSON

* Return some more M_BAD_JSON in the right places

* nolint because damnit gocyclo all I added was a type check for an error

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update sytest-whitelist

* Update gomatrixserverlib

* Update sytest-whitelist

* NotJSON -> BadJSON
2020-06-04 10:53:39 +01:00
Kegsay
e21d7d4baf
Update DendriteJS.Dockerfile 2020-06-03 16:03:54 +01:00
Aiden McClelland
17c92ad10e
Adds support for adding a proxy to the HTTP Client from the config (#1055)
* adds support for defining an proxy for the http client within the config

* alphabetize imports

* goimports

* comments
2020-06-02 21:02:24 +01:00
Kegsay
dc3338d1f2
Remove ENV to avoid confusion 2020-06-02 16:35:07 +01:00
Neil Alexander
e598e80d76
Delegate responsibility for marking room versions as supported/stable to gomatrixserverlib (#1082) 2020-06-02 16:20:50 +01:00
Kegsay
e37720be44
Sort public rooms again by member count (#1083) 2020-06-02 16:18:01 +01:00
Kegsay
02b150fd13
Only store our own aliases in publicroomsapi (#1081)
Otherwise we just store the latest aliases submitted from a server,
which is not what we want.
2020-06-02 15:01:13 +01:00
Neil Alexander
794c63e757
Reset backoff on incoming federation (#1080)
* Reset backoffs in response to incoming federation requests

* Federation wakeups no more than once per minute per origin
2020-06-02 12:42:36 +01:00
Neil Alexander
484b6f694c
Use gomatrixserverlib device structs (#1079) 2020-06-02 11:29:47 +01:00
Kegan Dougal
353a5d6fc2 Remove p2p log lines 2020-06-02 11:03:01 +01:00
Neil Alexander
a07fc0ef35 Return correct error to LookupRoomAlias 2020-06-02 09:34:36 +01:00
Kegan Dougal
42e797de5e Unbreak dendritejs 2020-06-01 18:41:58 +01:00
S7evinK
895c8f03c0
Fix pg user already exists (#1076)
* Return newly created error if user already exists (#1002)

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Rename variable

* Remove check for account and use returned error

* Return ErrUserExists

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* State that CreateAccount will return err ErrUserExists if the user exists

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Also check sqlite for constraint error

* Revert "Also check sqlite for constraint error"

This reverts commit 7d310514

* Check for sqlite3 constraint error

* Add documentation to CreateAccount

* Move ErrUserExists to accounts package

* Revert "Move ErrUserExists to accounts package"
Import Cycle..

This reverts commit be3d4cda

Co-authored-by: Kegsay <kegan@matrix.org>
2020-06-01 18:34:29 +01:00
Kegsay
cfc137652e
Add a way to force federationsender to retry sending transactions (#1077)
* Add a way to force federationsender to retry sending transactions

And use it in P2P mode when we pick up new nodes.

* Linting

* Use atomic bool to stop us blocking on the channel
2020-06-01 18:34:08 +01:00
Neil Alexander
a5d822004d
Send-to-device support (#1072)
* Groundwork for send-to-device messaging

* Update sample config

* Add unstable routing for now

* Send to device consumer in sync API

* Start the send-to-device consumer

* fix indentation in dendrite-config.yaml

* Create send-to-device database tables, other tweaks

* Add some logic for send-to-device messages, add them into sync stream

* Handle incoming send-to-device messages, count them with EDU stream pos

* Undo changes to test

* pq.Array

* Fix sync

* Logging

* Fix a couple of transaction things, fix client API

* Add send-to-device test, hopefully fix bugs

* Comments

* Refactor a bit

* Fix schema

* Fix queries

* Debug logging

* Fix storing and retrieving of send-to-device messages

* Try to avoid database locks

* Update sync position

* Use latest sync position

* Jiggle about sync a bit

* Fix tests

* Break out the retrieval from the update/delete behaviour

* Comments

* nolint on getResponseWithPDUsForCompleteSync

* Try to line up sync tokens again

* Implement wildcard

* Add all send-to-device tests to whitelist, what could possibly go wrong?

* Only care about wildcard when targeted locally

* Deduplicate transactions

* Handle tokens properly, return immediately if waiting send-to-device messages

* Fix sync

* Update sytest-whitelist

* Fix copyright notice (need to do more of this)

* Comments, copyrights

* Return errors from Do, fix dendritejs

* Review comments

* Comments

* Constructor for TransactionWriter

* defletions

* Update gomatrixserverlib, sytest-blacklist
2020-06-01 17:50:19 +01:00