Commit Graph

2133 Commits

Author SHA1 Message Date
kegsay
db7d9cba8a
BREAKING: Remove Partitioned Stream Positions (#2096)
* go mod tidy

* Break complement to check it fails CI

* Remove partitioned stream positions

This was used by the device list stream position. The device list position
now corresponds to the `Offset`, and the partition is always 0, in prep
for removing reliance on Kafka topics for device list changes.

* Linting

* Migrate old style tokens to new style because element-web doesn't soft-logoout on 4xx errors on /sync
2022-01-20 15:26:45 +00:00
Neil Alexander
16035b9737
NATS JetStream tweaks (#2086)
* Use named NATS durable consumers

* Build fixes

* Remove dupe call to SetFederationAPI

* Use namespaced consumer name

* Fix namespacing

* Fix unit tests hopefully
2022-01-07 17:31:57 +00:00
Neil Alexander
a422321435
Fix panic at startup if roomserver was not given federation API reference by the time NATS consumes an event, tweak backpressure metrics 2022-01-07 13:41:53 +00:00
kegsay
173b1e8d3e
Fix #2084 - incorrect /event_auth response (#2085)
* Fix #2084

* Return early

* Linting
2022-01-06 17:13:34 +00:00
S7evinK
161f145176
Add NATS JetStream support (#1866)
* Add NATS JetStream support
Update shopify/sarama

* Fix addresses

* Don't change Addresses in Defaults

* Update saramajetstream

* Add missing error check

Keep typing events for at least one minute

* Use all configured NATS addresses

* Update saramajetstream

* Try setting up with NATS

* Make sure NATS uses own persistent directory (TODO: make this configurable)

* Update go.mod/go.sum

* Jetstream package

* Various other refactoring

* Build fixes

* Config tweaks, make random jetstream storage path for CI

* Disable interest policies

* Try to sane default on jetstream base path

* Try to use in-memory for CI

* Restore storage/retention

* Update nats.go dependency

* Adapt changes to config

* Remove unneeded TopicFor

* Dep update

* Revert "Remove unneeded TopicFor"

This reverts commit f5a4e4a339b6f94ec215778dca22204adaa893d1.

* Revert changes made to streams

* Fix build problems

* Update nats-server

* Update go.mod/go.sum

* Roomserver input API queuing using NATS

* Fix topic naming

* Prometheus metrics

* More refactoring to remove saramajetstream

* Add missing topic

* Don't try to populate map that doesn't exist

* Roomserver output topic

* Update go.mod/go.sum

* Message acknowledgements

* Ack tweaks

* Try to resume transaction re-sends

* Try to resume transaction re-sends

* Update to matrix-org/gomatrixserverlib@91dadfb

* Remove internal.PartitionStorer from components that don't consume keychanges

* Try to reduce re-allocations a bit in resolveConflictsV2

* Tweak delivery options on RS input

* Publish send-to-device messages into correct JetStream subject

* Async and sync roomserver input

* Update dendrite-config.yaml

* Remove roomserver tests for now (they need rewriting)

* Remove roomserver test again (was merged back in)

* Update documentation

* Docker updates

* More Docker updates

* Update Docker readme again

* Fix lint issues

* Send final event in `processEvent` synchronously (since this might stop Sytest from being so upset)

* Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that

* Go 1.16 instead of Go 1.13 for upgrade tests and Complement

* Revert "Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that"

This reverts commit 368675283fc44501f227639811bdb16dd5deef8c.

* Don't report any errors on `/send` to see what fun that creates

* Fix panics on closed channel sends

* Enforce state key matches sender

* Do the same for leave

* Various tweaks to make tests happier

Squashed commit of the following:

commit 13f9028e7a63662759ce7c55504a9d2423058668
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 15:47:14 2022 +0000

    Do the same for leave

commit e6be7f05c349fafbdddfe818337a17a60c867be1
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 15:33:42 2022 +0000

    Enforce state key matches sender

commit 85ede6d64bf10ce9b91cdd6d80f87350ee55242f
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 14:07:04 2022 +0000

    Fix panics on closed channel sends

commit 9755494a98bed62450f8001d8128e40481d27e15
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 13:38:22 2022 +0000

    Don't report any errors on `/send` to see what fun that creates

commit 3bb4f87b5dd56882febb4db5621db484c8789b7c
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 13:00:26 2022 +0000

    Revert "Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that"

    This reverts commit 368675283fc44501f227639811bdb16dd5deef8c.

commit fe2673ed7be9559eaca134424e403a4faca100b0
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 12:09:34 2022 +0000

    Go 1.16 instead of Go 1.13 for upgrade tests and Complement

commit 368675283fc44501f227639811bdb16dd5deef8c
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 11:51:45 2022 +0000

    Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that

commit b028dfc08577bcf52e6cb498026e15fa5d46d07c
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 10:29:08 2022 +0000

    Send final event in `processEvent` synchronously (since this might stop Sytest from being so upset)

* Merge in NATS Server v2.6.6 and nats.go v1.13 into the in-process connection fork

* Add `jetstream.WithJetStreamMessage` to make ack/nak-ing less messy, use process context in consumers

* Fix consumer component name in  federation API

* Add comment explaining where streams are defined

* Tweaks to roomserver input with comments

* Finish that sentence that I apparently forgot to finish in INSTALL.md

* Bump version number of config to 2

* Add comments around asynchronous sends to roomserver in processEventWithMissingState

* More useful error message when the config version does not match

* Set version in generate-config

* Fix version in config.Defaults

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-01-05 17:44:49 +00:00
S7evinK
a47b12dc7d
Use non http API as fsAPI to avoid network requests (#2075)
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
2021-12-20 10:27:58 +00:00
S7evinK
640ba425e2
Allow partial test runs (#2076)
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
2021-12-20 10:26:23 +00:00
Martin Ashby
e5a5be9f54
Added std type to logging config. (#2060)
* Added std type to logging level.

Added demux-ing of logging to stdout/err working around
https://github.com/sirupsen/logrus/issues/403

Turned off the default logging in favour of using hooks always.

Added fallback, if no stdout/err logger was configured, add one
automatically. This prevents unexpected lack of logging after an
upgrade in case the user's configuration file is not updated.

Fixes: https://github.com/matrix-org/dendrite/issues/2054
Signed-off-by: Martin Ashby <martin@ashbysoft.com>

* Fix build on Windows - revert function rename SetupHookLogging

Fixes https://github.com/matrix-org/dendrite/pull/2060/files#r761692681

* Revert logging formatter changes to log.go, base.go.

The same formatting should be applied on both windows and unix, so it
makes sense to keep it in files which are shared between both platforms.

Fixes
https://github.com/matrix-org/dendrite/pull/2060#discussion_r762557679

* Remove unnecessary startup logging about info level logger.

Fixes
https://github.com/matrix-org/dendrite/pull/2060#discussion_r763959474

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-12-16 15:37:01 +00:00
Neil Alexander
002c3e0a5f
Fix build error 2021-12-13 13:29:43 +00:00
Neil Alexander
6b574ac537
Update to matrix-org/pinecone@aa2808d 2021-12-13 13:29:01 +00:00
Neil Alexander
3113210f17
Fix keyring regressions in previous P2P demo 2021-12-13 13:24:49 +00:00
Neil Alexander
c3dda0779d
Return event NID from StoreEvent, match PSQL vs SQLite behaviour, tweak backfill persistence (#2071) 2021-12-09 15:03:26 +00:00
Neil Alexander
172bd7c478
Set @matrix-org/dendrite-core as repository code owner 2021-12-09 13:29:49 +00:00
Neil Alexander
61406a6747
Cherry-pick removal of unused HTTP client from #2061
Co-authored-by: Tommie Gannert <tommie@gannert.se>
2021-12-03 17:30:14 +00:00
Neil Alexander
b7f09f78b0
Cherry-pick typing fix from #2061
Co-authored-by: Tommie Gannert <tommie@gannert.se>
2021-12-03 17:26:30 +00:00
S7evinK
08a0278760
Add missing HTTP mode for userapi (#1982)
* Add missing internal api endpoint

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Add missing performKeyBackup endpoint

* Add missing http mode for userapi

* Fix failing tests

* Add error checks

* Fix sytest

* Update startup logic for HTTP mode

* Use userImpl for AS (annoying)

* Don't send device list updates for appservice devices

* Fix build

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-12-03 17:18:35 +00:00
S7evinK
f9bac2f78a
Add missing internal routes (#2064)
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
2021-12-03 09:49:14 +00:00
nymori
388d7a1974
Squash username to lowercase at login (#2065)
Signed-off-by: Bernard Zhao <bernard.zhao.us@gmail.com>
2021-12-03 09:48:49 +00:00
Ben Yanke
83b9cb4d79
Updating example to Postgres v14 (#2062)
See issue #2052
2021-11-29 10:17:16 +00:00
Neil Alexander
11f588b0d6
Fixes for multiple Pinecone peers 2021-11-25 11:18:01 +00:00
Neil Alexander
310edd737c
Fix P2P demo builds 2021-11-25 10:48:44 +00:00
Neil Alexander
8f2a838452
Update to matrix-org/pinecone@0cc483b 2021-11-25 09:49:00 +00:00
Neil Alexander
9bc1c36ff6
Support connecting to multiple Pinecone static peers in the P2P demos (supply a comma-separated list) 2021-11-25 09:46:26 +00:00
S7evinK
25dcf80180
Ratelimit requests to /media/r0/download|upload (#2020)
* Add /media/r0/config handler

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Add rate limiting to media api

* Rename variable

* Add passing tests

* Don't send multiple headers

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-11-24 12:55:44 +00:00
The Stranjer
17227f8e98
Don't let things added to the media_store/ directory find their way into the repo (#2057) 2021-11-24 12:55:13 +00:00
Neil Alexander
c9419e51af
Don't populate config defaults where it doesn't make sense (#2058)
* Don't populate config defaults where it doesn't make sense

* Fix dendritejs builds
2021-11-24 11:57:39 +00:00
Neil Alexander
ec716793eb
Merge federationapi, federationsender, signingkeyserver components (#2055)
* Initial federation sender -> federation API refactoring

* Move base into own package, avoids import cycle

* Fix build errors

* Fix tests

* Add signing key server tables

* Try to fold signing key server into federation API

* Fix dendritejs builds

* Update embedded interfaces

* Fix panic, fix lint error

* Update configs, docker

* Rename some things

* Reuse same keyring on the implementing side

* Fix federation tests, `NewBaseDendrite` can accept freeform options

* Fix build

* Update create_db, configs

* Name tables back

* Don't rename federationsender consumer for now
2021-11-24 10:45:23 +00:00
Neil Alexander
6e93531e94
Don't persist transaction IDs in the roomserver (#2048) 2021-11-22 09:13:12 +00:00
Neil Alexander
403498a85b
Only return non-stub rooms from GetKnownRooms (#2049)
* Only return non-stub rooms from `GetKnownRooms`

This should stop a bunch of errors at startup with invalid server ACLs.

* Fix query
2021-11-18 11:34:19 +00:00
Neil Alexander
e81ef1649b
Version 0.5.1 (#2047)
* Version bump, changelog

* Update to matrix-org/pinecone@8c7a1ad

* Update to matrix-org/pinecone@febf350
2021-11-16 11:37:03 +00:00
Neil Alexander
a9e715b5c5
Guard in all key consumers 2021-11-16 09:27:49 +00:00
Neil Alexander
ddbef7c3ff
Fix panic in federationsender consumer 2021-11-16 09:25:36 +00:00
Neil Alexander
58b831ad36
Mark room versions 8 and 9 as unstable (update to matrix-org/gomatrixserverlib@15a64d2) 2021-11-15 19:29:03 +00:00
Neil Alexander
5565e40395
Fix default power levels (update to matrix-org/gomatrixserverlib@af2616b) 2021-11-12 15:16:22 +00:00
Neil Alexander
59cf8e936e
Fix sytest 2021-11-08 10:32:21 +00:00
Neil Alexander
9cd4a1f97b
Update to matrix-org/gomatrixserverlib@ed70208 (hopefully will fix sytest) 2021-11-08 09:45:39 +00:00
Neil Alexander
ee35c15467
Update sytest-whitelist 2021-11-08 09:25:01 +00:00
Neil Alexander
323a6fb54f
Resume federation sends (#2039)
* Resume federation sends

* Review comments

* Fix build error
2021-11-08 09:24:16 +00:00
Neil Alexander
b9a575919a
Try to reduce re-allocations a bit in resolveConflictsV2 2021-11-04 10:55:07 +00:00
Neil Alexander
77096898f0
Further state resolution v2 optimisations 2021-11-04 10:35:23 +00:00
Neil Alexander
3c049c068f
State resolution v2 performance improvements (reduce allocs in Kahn's algorithm, update to matrix-org/gomatrixserverlib@91dadfb) 2021-11-04 09:09:26 +00:00
Neil Alexander
837f50ac89
Reduce CPU usage of SelectStateInRange (#2038) 2021-11-03 09:53:37 +00:00
Neil Alexander
b4a007ecce
Support joining version 8 and 9 rooms (update to matrix-org/gomatrixserverlib@13366e7) 2021-11-02 17:15:06 +00:00
S7evinK
0c2872828b
Fix building on Windows (#2008)
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
2021-11-02 16:50:59 +00:00
PiotrKozimor
dec05c3347
Run gofmt on dendrite - apply go 1.17 preferred build tags (#2021) 2021-11-02 16:48:48 +00:00
Neil Alexander
b99f594a93
Fix #2028 (#2036) 2021-11-02 16:47:39 +00:00
database64128
5b969d172b
0️⃣ Media API: Fix maxFileSizeBytes=0 handling (#1993) 2021-11-02 16:35:15 +00:00
hacktivista
1ec5b0709b
Support WebP based thumbnails (#2033)
Co-authored-by: Felix Freeman <libsys@hacktivista.org>
2021-11-02 10:15:13 +00:00
Neil Alexander
fbd1a0ab13
Update to matrix-org/gomatrixserverlib@5e02b64 2021-11-02 10:13:38 +00:00
kegsay
42973e7d65
Add part of #1322 tests (#2034)
* Add part of #1322 tests

* Fix #1322
2021-11-01 15:05:20 +00:00