Commit Graph

353 Commits

Author SHA1 Message Date
devonh
241d5c47df
Refactor Federation Destination Queues (#2807)
This is a refactor of the federation destination queues.
It fixes a few things, namely:
- actually retry outgoing events with backoff behaviour
- obtain enough events from the database to fill messages as much as
possible
- minimize the amount of running goroutines
  - use pure timers for backoff
  - don't restart queue unless necessary
  - close the background task when backing off
- increase max edus in a transaction to match the spec
- cleanup timers more aggresively to reduce memory usage
- add jitter to backoff timers to reduce resource spikes
- add a bunch of tests (with real and fake databases) to ensure
everything is working
2022-10-19 11:03:16 +01:00
Neil Alexander
9c189b1b80
Try to make AddEvent less expensive (update to matrix-org/gomatrixserverlib@a72a83f) 2022-10-18 09:51:31 +01:00
Neil Alexander
eac5678449
Update dependency now that it is fixed 2022-10-14 14:53:53 +01:00
Neil Alexander
f76969831e
Update direct dependencies (#2794)
This updates a number of Dendrite's dependencies.
2022-10-14 10:59:32 +01:00
Neil Alexander
82d1d434c5
Update to NATS Server v2.9.3 and nats.go v1.18.0 2022-10-14 10:10:25 +01:00
Till
088ad1dd21
Fix outliers whose auth_events are in a different room are correctly rejected (#2791)
Fixes `outliers whose auth_events are in a different room are correctly
rejected`, by validating that auth events are all from the same room and
not using rejected events for event auth.
2022-10-14 09:14:54 +02:00
Neil Alexander
f3be4b3185
Revert "Federation backoff fixes and tests (#2792)"
This reverts commit dcedd1b6bf.
2022-10-13 16:06:50 +01:00
devonh
dcedd1b6bf
Federation backoff fixes and tests (#2792)
This fixes some edge cases where federation queue backoffs and
blacklisting weren't behaving as expected.
It also adds new tests for the federation queues to ensure their
behaviour continues to work correctly.
2022-10-13 14:38:13 +00:00
Neil Alexander
23a3e04579
Event relations (#2790)
This adds support for tracking `m.relates_to`, as well as adding support
for the various `/room/{roomID}/relations/...` endpoints to the CS API.
2022-10-13 14:50:52 +01:00
Neil Alexander
60ec9180e6
Update NATS Server to v2.9.2 2022-09-30 16:10:44 +01:00
Neil Alexander
e45ba35e97
Enable knocking on room versions 8 and 9 (update to matrix-org/gomatrixserverlib@91d455c) 2022-09-29 20:05:05 +01:00
devonh
90f1985bf3
Add network interface registration to pinecone demo (#2750)
### Pull Request Checklist

<!-- Please read docs/CONTRIBUTING.md before submitting your pull
request -->

No additional tests have been added as this simply extends the API for
the embedded pinecone demo.

Signed-off-by: `Devon Hudson <devonhudson@librem.one>`
2022-09-29 16:05:16 +00:00
Neil Alexander
b28bbadeb0
Update to matrix-org/gomatrixserverlib@377b320 2022-09-29 16:55:03 +01:00
Neil Alexander
e1bf709eb3
Improve device list update parsing (update to matrix-org/gomatrixserverlib#342) 2022-09-29 16:08:41 +01:00
Neil Alexander
68d6eb0a6f
P2P demo fixes 2022-09-29 13:06:55 +01:00
Neil Alexander
f18bce93cc
Pinecone hybrid routing (update to matrix-org/pinecone#67) 2022-09-27 11:15:49 +01:00
Neil Alexander
f022fc1397
Remove origin field from PDUs (#2737)
This nukes the `origin` field from PDUs as per
matrix-org/matrix-spec#998, matrix-org/gomatrixserverlib#341.
2022-09-26 17:35:35 +01:00
Neil Alexander
d5978d98fd
Update to matrix-org/pinecone@0900fceecb 2022-09-23 16:21:12 +01:00
Neil Alexander
a50556dcf0
Tweak mainline ordering (update to matrix-org/gomatrixserverlib@2217f6c) 2022-09-23 12:59:00 +01:00
Neil Alexander
852d856db8
Update embedded NATS Server to v2.9.1-beta1 (as this includes a fix that prevents high CPU usage after Dendrite startup) 2022-09-22 13:55:40 +01:00
Neil Alexander
bd39748b5c
Update dependencies (#2729)
This updates Dendrite dependencies.
2022-09-20 15:01:19 +01:00
Neil Alexander
e6960d0b15
Update to matrix-org/pinecone@608215eb1b 2022-09-14 14:25:25 +01:00
Neil Alexander
62afb936a5
Update to matrix-org/gomatrixserverlib@7b96db4 2022-09-12 15:27:45 +01:00
Neil Alexander
7595fbf58c
Update to NATS Server v2.9.0 2022-09-12 13:24:03 +01:00
Neil Alexander
e9af30b3fe
Update to matrix-org/pinecone@b215925 2022-09-12 10:35:04 +01:00
Neil Alexander
5997c32452
Update to matrix-org/gomatrixserverlib@dec87dbaa4 2022-09-11 15:39:11 +01:00
Till
d5876abbe9
Fulltext implementation incl. config (#2480)
This adds the main component of the fulltext search.
This PR doesn't do anything yet, besides creating an empty fulltextindex
folder if enabled. Indexing events is done in a separate PR.
2022-09-07 18:15:54 +02:00
Neil Alexander
847032df36
Update to matrix-org/gomatrixserverlib@637a173 2022-09-07 09:11:36 +01:00
Neil Alexander
70cd9a902c
State resolution fixes in applyAndAuthEvent (update to matrix-org/gomatrixserverlib#336) 2022-09-06 17:19:21 +01:00
Neil Alexander
bea3dbe77b
State and auth tweaks (update to matrix-org/gomatrixserverlib#335) 2022-09-05 17:00:54 +01:00
Neil Alexander
fea869b41f
Update P2P demos 2022-09-01 17:12:27 +01:00
Neil Alexander
02ec00b1bb
Check for "creator" field in m.room.create during event auth (update to matrix-org/gomatrixserverlib@c71e518) 2022-08-30 17:40:46 +01:00
Neil Alexander
bbb3ade4a2
Fix the Cat Overflow bug (update to matrix-org/gomatrixserverlib@2020bdc) 2022-08-30 11:44:26 +01:00
Till Faelligen
aba171d9bc
Default to limit of 20 state events 2022-08-29 07:23:46 +02:00
Neil Alexander
a3eb4e5e98
Enforce type, sender, state_key and room_id lengths using codepoints rather than bytes (update to matrix-org/gomatrixserverlib@5f66df0, matrix-org/matrix-spec#1001) 2022-08-28 09:52:03 +01:00
Till
07dd9bd995
SyncAPI tweaks/fixes (#2671)
- Reverts 9dc57122d9 as it was causing issues https://github.com/matrix-org/dendrite/issues/2660
- Updates the GMSL `DefaultStateFilter` to use a limit of 20 events
- Uses the timeline events to determine the new position instead of the state events
2022-08-25 13:42:47 +01:00
Neil Alexander
bcdbd5c00a
Version 0.9.3 2022-08-15 10:56:54 +01:00
Neil Alexander
9fa30f5d3c
Update NATS Server and nats.go libraries 2022-08-12 13:29:09 +01:00
Till
05cafbd197
Implement history visibility on /messages, /context, /sync (#2511)
* Add possibility to set history_visibility and user AccountType

* Add new DB queries

* Add actual history_visibility changes for /messages

* Add passing tests

* Extract check function

* Cleanup

* Cleanup

* Fix build on 386

* Move ApplyHistoryVisibilityFilter to internal

* Move queries to topology table

* Add filtering to /sync and /context
Some cleanup

* Add passing tests; Remove failing tests :(

* Re-add passing tests

* Move filtering to own function to avoid duplication

* Re-add passing test

* Use newly added GMSL HistoryVisibility

* Update gomatrixserverlib

* Set the visibility when creating events

* Default to shared history visibility

* Remove unused query

* Update history visibility checks to use gmsl
Update tests

* Remove unused statement

* Update migrations to set "correct" history visibility

* Add method to fetch the membership at a given event

* Tweaks and logging

* Use actual internal rsAPI, default to shared visibility in tests

* Revert "Move queries to topology table"

This reverts commit 4f0d41be9c194a46379796435ce73e79203edbd6.

* Remove noise/unneeded code

* More cleanup

* Try to optimize database requests

* Fix imports

* PR peview fixes/changes

* Move setting history visibility to own migration, be more restrictive

* Fix unit tests

* Lint

* Fix missing entries

* Tweaks for incremental syncs

* Adapt generic changes

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: kegsay <kegan@matrix.org>
2022-08-11 18:23:35 +02:00
Neil Alexander
376391d1c7
Update Pinecone 2022-08-03 10:38:36 +01:00
Neil Alexander
645f31ae24
Update NATS Server and nats.go to use upstream 2022-08-01 09:51:13 +01:00
Neil Alexander
ccea23cd40
Remove room_id field from MSC2946 stripped events (closes #2588) 2022-07-25 11:42:22 +01:00
Till
081f5e7226
Update database migrations, remove goose (#2264)
* Add new db migration

* Update migrations
Remove goose

* Add possibility to test direct upgrades

* Try to fix WASM test

* Add checks for specific migrations

* Remove AddMigration
Use WithTransaction
Add Dendrite version to table

* Fix linter issues

* Update tests

* Update comments, outdent if

* Namespace migrations

* Add direct upgrade tests, skipping over one version

* Split migrations

* Update go version in CI

* Fix copy&paste mistake

* Use contexts in migrations

Co-authored-by: kegsay <kegan@matrix.org>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-07-25 10:39:22 +01:00
Till
a7e92f8cb9
History visibility database changes (#2533)
* Add new history_visibility column

* Update SQL queries to include history_visibility

* Store the history visibilty calculated by the roomserver

* Update GMSL

* Update migrations

* Fix migration

* Update GMSL

* Fix `go.sum`

* Update GMSL to use sql.Scanner & sql.Valuer

* Re-order migration/table creation

* Update gomatrixserverlib

* Add history_visibility column to current_room_state

* Fix migrations

* Return error instead of Fatal log

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-07-18 14:46:15 +02:00
Neil Alexander
9cd8e9d4b9
Use /v3 to request media from remote servers (update to matrix-org/gomatrixserverlib#324) 2022-07-13 09:32:03 +01:00
Neil Alexander
3ea21273bc
Ristretto cache (#2563)
* Try Ristretto cache

* Tweak

* It's beautiful

* Update GMSL

* More strict keyable interface

* Fix that some more

* Make less panicky

* Don't enforce mutability checks for now

* Determine mutability using deep equality

* Tweaks

* Namespace keys

* Make federation caches mutable

* Update cost estimation, add metric

* Update GMSL

* Estimate cost for metrics better

* Reduce counters a bit

* Try caching events

* Some guards

* Try again

* Try this

* Use separate caches for hopefully better hash distribution

* Fix bug with admitting events into cache

* Try to fix bugs

* Check nil

* Try that again

* Preserve order jeezo this is messy

* thanks VS Code for doing exactly the wrong thing

* Try this again

* Be more specific

* aaaaargh

* One more time

* That might be better

* Stronger sorting

* Cache expiries, async publishing of EDUs

* Put it back

* Use a shared cache again

* Cost estimation fixes

* Update ristretto

* Reduce counters a bit

* Clean up a bit

* Update GMSL

* 1GB

* Configurable cache sizees

* Tweaks

* Add `config.DataUnit` for specifying friendly cache sizes

* Various tweaks

* Update GMSL

* Add back some lazy loading caching

* Include key in cost

* Include key in cost

* Tweak max age handling, config key name

* Only register prometheus metrics if requested

* Review comments @S7evinK

* Don't return errors when creating caches (it is better just to crash since otherwise we'll `nil`-pointer exception everywhere)

* Review comments

* Update sample configs

* Update GHA Workflow

* Update Complement images to Go 1.18

* Remove the cache test from the federation API as we no longer guarantee immediate cache admission

* Don't check the caches in the renewal test

* Possibly fix the upgrade tests

* Update to matrix-org/gomatrixserverlib#322

* Update documentation to refer to Go 1.18
2022-07-11 14:31:31 +01:00
Neil Alexander
eb8dc50a97
Update Pinecone to matrix-org/pinecone@1ce778f 2022-07-08 14:54:08 +01:00
Till
89cd0e8fc1
Try to fix backfilling (#2548)
* Try to fix backfilling

* Return start/end to not confuse clients

* Update GMSL

* Update GMSL
2022-07-01 11:49:26 +02:00
Jean Lucas
920a20821b
Fix nats.go commit (#2540)
Signed-off-by: Jean Lucas <jean@4ray.co>
2022-06-27 09:15:19 +01:00
Neil Alexander
4c2a10f1a6
Handle state before, send history visibility in output (#2532)
* Check state before event

* Tweaks

* Refactor a bit, include in output events

* Don't waste time if soft failed either

* Tweak control flow, comments, use GMSL history visibility type
2022-06-13 15:11:10 +01:00