Commit Graph

275 Commits

Author SHA1 Message Date
Neil Alexander
d7cc187ec0
Prevent JetStream from handling OS signals, allow running as a Windows service (#2385)
* Prevent JetStream from handling OS signals, allow running as a Windows service (fixes #2374)

* Remove double import
2022-04-27 13:36:40 +01:00
Neil Alexander
144c060fa7
Remove libp2p demos (#2337)
* Remove libp2p demos

* Put it back
2022-04-21 13:31:43 +01:00
Neil Alexander
a9f0a390c6
Update to NATS Server v2.8.0 and nats.go v1.14.0 (#2359) 2022-04-20 15:13:04 +01:00
Neil Alexander
711e377b9c
Update go-sqlite3-js to matrix-org/go-sqlite3-js#2 (SQLite 3.36.0) 2022-04-19 10:34:33 +01:00
Till
69f2ff7c82
Correctly use provided filters (#2339)
* Apply filters correctly

* Fix issues; Use prepareWithFilters

* Update gmsl & tests

* go.mod..

* PR comments
2022-04-11 09:05:23 +02:00
Neil Alexander
986d27a128
Update to matrix-org/pinecone@2999ea2 2022-04-08 16:39:09 +01:00
Neil Alexander
9bd9f2beba
Update to matrix-org/pinecone@9b3963248c 2022-04-08 16:23:50 +01:00
Neil Alexander
4312b87ab0
Update dependencies (#2336) 2022-04-08 12:24:49 +01:00
Neil Alexander
4d9d9cc9b1
Update to matrix-org/gomatrixserverlib#300 2022-04-05 14:43:44 +01:00
Neil Alexander
562d742240
Update to matrix-org/gomatrixserverlib#299 2022-04-04 19:04:17 +01:00
Neil Alexander
c69159bda7
Update to matrix-org/pinecone@e526fa8 2022-04-04 15:16:02 +01:00
Neil Alexander
8213b2ba30
Update Pinecone P2P demo 2022-03-30 15:01:22 +01:00
Neil Alexander
6c95f9b535
Update to matrix-org/pinecone@6fb0773 2022-03-23 14:45:22 +00:00
Neil Alexander
c54d88aecf
Update to matrix-org/pinecone@8687b0d 2022-03-18 15:32:14 +00:00
Neil Alexander
0fb94fc781
Update to matrix-org/gomatrixserverlib@0980b7f 2022-03-17 16:48:41 +00:00
Neil Alexander
1894f34868
Update to matrix-org/gomatrixserverlib@a62d9f3 2022-03-17 09:58:44 +00:00
Neil Alexander
047147a3af
Update Yggdrasil P2P demo (also remove QUIC dependency) 2022-03-16 16:49:10 +00:00
Neil Alexander
0ad665fabc
Update to NATS Server 2.7.4 (#2282)
* Update to NATS Server v2.7.4-beta2

* Bump to release v2.7.4

* Listen for TCP

* Revert "Listen for TCP"

This reverts commit 28dff0ab4d807417a9eda8feab9f8efdf13837c7.
2022-03-16 10:47:18 +00:00
Neil Alexander
2696f36738
Federation client improvements (HTTP/2, reaping federation trippers, TLS client session cache, keepalive tweaks, update to matrix-org/gomatrixserverlib#297) 2022-03-16 10:27:16 +00:00
Neil Alexander
e1881627d1
Implement MSC3383 (update to matrix-org/gomatrixserverlib#296) 2022-03-10 12:44:31 +00:00
Neil Alexander
030b995636
Update to matrix-org/pinecone@cfde1f8 2022-03-09 10:19:49 +00:00
Neil Alexander
0297929b76
Revert NATS server upgrade in 00b3545b14 2022-03-04 15:56:31 +00:00
Neil Alexander
00b3545b14
Update NATS Server to v2.7.3 2022-03-02 10:36:20 +00:00
Kegan Dougal
1a79060b46 Bump GMSL version 2022-03-01 14:16:47 +00:00
kegsay
f1b92de017
MSC2946: Spaces Summary (round 2) (#2232)
* Initial cut at fixing up MSC2946 to work with latest spec

* bugfix: send response back correctly

* Initial working version of MSC2946

* msc2946: handle suggested_only; remove custom database

As the MSC doesn't require reverse lookups, we can just pull
the room state and inspect via the roomserver database. To
handle this, expand QueryCurrentState to support wildcards.

Use all this and handle `?suggested_only`.

* Sort child rooms

* msc2946: Make TestClientSpacesSummary pass

* msc2946: allow invited rooms to be spidered

* msc2946: support basic federation requests

* fix up go mod
2022-03-01 13:40:07 +00:00
Neil Alexander
a23fda6626
Update Events call-sites which now don't return an error, update parsedRespState to sort (#2227)
* Topologically sort with `SendEventWithState`, so that earlier events should satisfy auth for later ones

* Revert "Topologically sort with `SendEventWithState`, so that earlier events should satisfy auth for later ones"

This reverts commit b0cd706012b4c9b6724b11e16f19c4cb732ab286.

* Update to matrix-org/gomatrixserverlib#293

* `Events` no longer returns an error, other tweaks

* Make sure `Events` is sorted for `parsedRespState` too
2022-02-28 14:51:40 +00:00
Neil Alexander
4c07374c42
Reduce allocations significantly in state res v2, which should help to keep memory down when joining rooms too (update to matrix-org/gomatrixserverlib@f6ab9c5) 2022-02-24 17:05:49 +00:00
Neil Alexander
b8a97b6ee0
Update to matrix-org/pinecone@0f0afd1a46 2022-02-23 10:45:07 +00:00
Neil Alexander
934491eda5
Update NATS Server to v2.7.2 (#2193)
* Update NATS JetStream to v2.7.2

* Remove deprecated option
2022-02-17 13:15:35 +00:00
Neil Alexander
4c8c53244e
Update prometheus 2022-02-15 16:27:22 +00:00
Neil Alexander
e22e87c012
Update to matrix-org/gomatrixserverlib@20632dd 2022-02-15 11:14:43 +00:00
kegsay
aa5c3b88de
Unmarshal events at the Dendrite level not GMSL level (#2164)
* Use new event json types in gmsl

* Fix EventJSON to actually unmarshal events

* Update GMSL

* Bump GMSL and improve error messages

* Send back the correct RespState

* Update GMSL
2022-02-09 20:31:24 +00:00
Neil Alexander
a2b4777ae5
Update to matrix-org/gomatrixserverlib@a05e156fd8 2022-02-08 09:30:21 +00:00
S7evinK
9de7efa0b0
Remove sarama/saramajetstream dependencies (#2138)
* Remove dependency on saramajetstream & sarama

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Remove internal.ContinualConsumer from federationapi

* Remove internal.ContinualConsumer from syncapi

* Remove internal.ContinualConsumer from keyserver

* Move to new Prepare function

* Remove saramajetstream & sarama dependency

* Delete unneeded file

* Remove duplicate import

* Log error instead of silently irgnoring it

* Move `OffsetNewest` and `OffsetOldest` into keyserver types, change them to be more sane values

* Fix comments

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-02-04 13:08:13 +00:00
Neil Alexander
532f445c4e
Remove roomserver input deadlines (#2144)
It isn't really clear that the deadlines actually help in any way. Currently we can use up our 2 minutes doing something, run out of context time and then return an error which causes the transaction to rollback and forgetting everything we've done. If the message came to us from NATS then we probably will end up retrying just to be in the same situation. We'd be really a lot better if we just spent the time reconciling the problem in the first place, and then we're much less likely to need to fetch those missing auth or prev events in the future.

Also includes matrix-org/gomatrixserverlib#287 so we don't wait so long for servers that are obviously dead.
2022-02-04 12:13:07 +00:00
Neil Alexander
567fd04428
Update to matrix-org/gomatrixserverlib#286 2022-01-31 14:29:13 +00:00
Neil Alexander
dac762d025
Revert Prometheus client upgrades altogether 2022-01-31 11:47:07 +00:00
Neil Alexander
2995f73ae0
Update prometheus client 2022-01-31 11:16:21 +00:00
Neil Alexander
5367e7ed2c
Update to matrix-org/gomatrixserverlib@801c51af9f 2022-01-31 10:51:01 +00:00
Neil Alexander
ba1a9b98b7
Tweak some logging (#2130)
* Modify some log levels

* Update gomatrixserverlib to matrix-org/gomatrixserverlib@336334f

* Update gomatrixserverlib to matrix-org/gomatrixserverlib@cde7ac8

* Demote warning about key change producer

* Add more useful roomserver logging

* Further tweaking
2022-01-31 10:48:28 +00:00
Neil Alexander
4281976df9
Update Sarama to fix 32-bit builds (#2120) 2022-01-28 17:31:54 +00:00
Neil Alexander
2f72688e9a
Upgrade dependencies (#2074)
* Upgrade dependencies

* Revert gjson/sjson due to panics

* Revert sarama as it requires Go 1.16

* Revert quic-go as it requires Go 1.16

* Revert sarama again
2022-01-28 10:27:09 +00:00
Neil Alexander
b626143694
Update gomatrixserverlib 2022-01-28 10:01:20 +00:00
Neil Alexander
a763cbb0e1
Roomserver/federation input refactor (#2104)
* Put federation client functions into their own file

* Look for missing auth events in RS input

* Remove retrieveMissingAuthEvents from federation API

* Logging

* Sorta transplanted the code over

* Use event origin failing all else

* Don't get stuck on mutexes:

* Add verifier

* Don't mark state events with zero snapshot NID as not existing

* Check missing state if not an outlier before storing the event

* Reject instead of soft-fail, don't copy roominfo so much

* Use synchronous contexts, limit time to fetch missing events

* Clean up some commented out bits

* Simplify `/send` endpoint significantly

* Submit async

* Report errors on sending to RS input

* Set max payload in NATS to 16MB

* Tweak metrics

* Add `workerForRoom` for tidiness

* Try skipping unmarshalling errors for RespMissingEvents

* Track missing prev events separately to avoid calculating state when not possible

* Tweak logic around checking missing state

* Care about state when checking missing prev events

* Don't check missing state for create events

* Try that again

* Handle create events better

* Send create room events as new

* Use given event kind when sending auth/state events

* Revert "Use given event kind when sending auth/state events"

This reverts commit 089d64d271b5fca8c104e1554711187420dbebca.

* Only search for missing prev events or state for new events

* Tweaks

* We only have missing prev if we don't supply state

* Room version tweaks

* Allow async inputs again

* Apply backpressure to consumers/synchronous requests to hopefully stop things being overwhelmed

* Set timeouts on roomserver input tasks (need to decide what timeout makes sense)

* Use work queue policy, deliver all on restart

* Reduce chance of duplicates being sent by NATS

* Limit the number of servers we attempt to reduce backpressure

* Some review comment fixes

* Tidy up a couple things

* Don't limit servers, randomise order using map

* Some context refactoring

* Update gmsl

* Don't resend create events

* Set stateIDs length correctly or else the roomserver thinks there are missing events when there aren't

* Exclude our own servername

* Try backing off servers

* Make excluding self behaviour optional

* Exclude self from g_m_e

* Update sytest-whitelist

* Update consumers for the roomserver output stream

* Remember to send outliers for state returned from /gme

* Make full HTTP tests less upsetti

* Remove 'If a device list update goes missing, the server resyncs on the next one' from the sytest blacklist

* Remove debugging test

* Fix blacklist again, remove unnecessary duplicate context

* Clearer contexts, don't use background in case there's something happening there

* Don't queue up events more than once in memory

* Correctly identify create events when checking for state

* Fill in gaps again in /gme code

* Remove `AuthEventIDs` from `InputRoomEvent`

* Remove stray field

Co-authored-by: Kegan Dougal <kegan@matrix.org>
2022-01-27 14:29:14 +00:00
Neil Alexander
a4601bb698
Update to matrix-org/gomatrixserverlib@f3e2ef8 (matrix-org/matrix-doc#3667) 2022-01-24 10:28:23 +00:00
S7evinK
161f145176
Add NATS JetStream support (#1866)
* Add NATS JetStream support
Update shopify/sarama

* Fix addresses

* Don't change Addresses in Defaults

* Update saramajetstream

* Add missing error check

Keep typing events for at least one minute

* Use all configured NATS addresses

* Update saramajetstream

* Try setting up with NATS

* Make sure NATS uses own persistent directory (TODO: make this configurable)

* Update go.mod/go.sum

* Jetstream package

* Various other refactoring

* Build fixes

* Config tweaks, make random jetstream storage path for CI

* Disable interest policies

* Try to sane default on jetstream base path

* Try to use in-memory for CI

* Restore storage/retention

* Update nats.go dependency

* Adapt changes to config

* Remove unneeded TopicFor

* Dep update

* Revert "Remove unneeded TopicFor"

This reverts commit f5a4e4a339b6f94ec215778dca22204adaa893d1.

* Revert changes made to streams

* Fix build problems

* Update nats-server

* Update go.mod/go.sum

* Roomserver input API queuing using NATS

* Fix topic naming

* Prometheus metrics

* More refactoring to remove saramajetstream

* Add missing topic

* Don't try to populate map that doesn't exist

* Roomserver output topic

* Update go.mod/go.sum

* Message acknowledgements

* Ack tweaks

* Try to resume transaction re-sends

* Try to resume transaction re-sends

* Update to matrix-org/gomatrixserverlib@91dadfb

* Remove internal.PartitionStorer from components that don't consume keychanges

* Try to reduce re-allocations a bit in resolveConflictsV2

* Tweak delivery options on RS input

* Publish send-to-device messages into correct JetStream subject

* Async and sync roomserver input

* Update dendrite-config.yaml

* Remove roomserver tests for now (they need rewriting)

* Remove roomserver test again (was merged back in)

* Update documentation

* Docker updates

* More Docker updates

* Update Docker readme again

* Fix lint issues

* Send final event in `processEvent` synchronously (since this might stop Sytest from being so upset)

* Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that

* Go 1.16 instead of Go 1.13 for upgrade tests and Complement

* Revert "Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that"

This reverts commit 368675283fc44501f227639811bdb16dd5deef8c.

* Don't report any errors on `/send` to see what fun that creates

* Fix panics on closed channel sends

* Enforce state key matches sender

* Do the same for leave

* Various tweaks to make tests happier

Squashed commit of the following:

commit 13f9028e7a63662759ce7c55504a9d2423058668
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 15:47:14 2022 +0000

    Do the same for leave

commit e6be7f05c349fafbdddfe818337a17a60c867be1
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 15:33:42 2022 +0000

    Enforce state key matches sender

commit 85ede6d64bf10ce9b91cdd6d80f87350ee55242f
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 14:07:04 2022 +0000

    Fix panics on closed channel sends

commit 9755494a98bed62450f8001d8128e40481d27e15
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 13:38:22 2022 +0000

    Don't report any errors on `/send` to see what fun that creates

commit 3bb4f87b5dd56882febb4db5621db484c8789b7c
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 13:00:26 2022 +0000

    Revert "Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that"

    This reverts commit 368675283fc44501f227639811bdb16dd5deef8c.

commit fe2673ed7be9559eaca134424e403a4faca100b0
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 12:09:34 2022 +0000

    Go 1.16 instead of Go 1.13 for upgrade tests and Complement

commit 368675283fc44501f227639811bdb16dd5deef8c
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 11:51:45 2022 +0000

    Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that

commit b028dfc08577bcf52e6cb498026e15fa5d46d07c
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 10:29:08 2022 +0000

    Send final event in `processEvent` synchronously (since this might stop Sytest from being so upset)

* Merge in NATS Server v2.6.6 and nats.go v1.13 into the in-process connection fork

* Add `jetstream.WithJetStreamMessage` to make ack/nak-ing less messy, use process context in consumers

* Fix consumer component name in  federation API

* Add comment explaining where streams are defined

* Tweaks to roomserver input with comments

* Finish that sentence that I apparently forgot to finish in INSTALL.md

* Bump version number of config to 2

* Add comments around asynchronous sends to roomserver in processEventWithMissingState

* More useful error message when the config version does not match

* Set version in generate-config

* Fix version in config.Defaults

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-01-05 17:44:49 +00:00
Martin Ashby
e5a5be9f54
Added std type to logging config. (#2060)
* Added std type to logging level.

Added demux-ing of logging to stdout/err working around
https://github.com/sirupsen/logrus/issues/403

Turned off the default logging in favour of using hooks always.

Added fallback, if no stdout/err logger was configured, add one
automatically. This prevents unexpected lack of logging after an
upgrade in case the user's configuration file is not updated.

Fixes: https://github.com/matrix-org/dendrite/issues/2054
Signed-off-by: Martin Ashby <martin@ashbysoft.com>

* Fix build on Windows - revert function rename SetupHookLogging

Fixes https://github.com/matrix-org/dendrite/pull/2060/files#r761692681

* Revert logging formatter changes to log.go, base.go.

The same formatting should be applied on both windows and unix, so it
makes sense to keep it in files which are shared between both platforms.

Fixes
https://github.com/matrix-org/dendrite/pull/2060#discussion_r762557679

* Remove unnecessary startup logging about info level logger.

Fixes
https://github.com/matrix-org/dendrite/pull/2060#discussion_r763959474

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-12-16 15:37:01 +00:00
Neil Alexander
6b574ac537
Update to matrix-org/pinecone@aa2808d 2021-12-13 13:29:01 +00:00
Neil Alexander
310edd737c
Fix P2P demo builds 2021-11-25 10:48:44 +00:00
Neil Alexander
8f2a838452
Update to matrix-org/pinecone@0cc483b 2021-11-25 09:49:00 +00:00