Commit Graph

337 Commits

Author SHA1 Message Date
Neil Alexander
f022fc1397
Remove origin field from PDUs (#2737)
This nukes the `origin` field from PDUs as per
matrix-org/matrix-spec#998, matrix-org/gomatrixserverlib#341.
2022-09-26 17:35:35 +01:00
Neil Alexander
d5978d98fd
Update to matrix-org/pinecone@0900fceecb 2022-09-23 16:21:12 +01:00
Neil Alexander
a50556dcf0
Tweak mainline ordering (update to matrix-org/gomatrixserverlib@2217f6c) 2022-09-23 12:59:00 +01:00
Neil Alexander
852d856db8
Update embedded NATS Server to v2.9.1-beta1 (as this includes a fix that prevents high CPU usage after Dendrite startup) 2022-09-22 13:55:40 +01:00
Neil Alexander
bd39748b5c
Update dependencies (#2729)
This updates Dendrite dependencies.
2022-09-20 15:01:19 +01:00
Neil Alexander
e6960d0b15
Update to matrix-org/pinecone@608215eb1b 2022-09-14 14:25:25 +01:00
Neil Alexander
62afb936a5
Update to matrix-org/gomatrixserverlib@7b96db4 2022-09-12 15:27:45 +01:00
Neil Alexander
7595fbf58c
Update to NATS Server v2.9.0 2022-09-12 13:24:03 +01:00
Neil Alexander
e9af30b3fe
Update to matrix-org/pinecone@b215925 2022-09-12 10:35:04 +01:00
Neil Alexander
5997c32452
Update to matrix-org/gomatrixserverlib@dec87dbaa4 2022-09-11 15:39:11 +01:00
Till
d5876abbe9
Fulltext implementation incl. config (#2480)
This adds the main component of the fulltext search.
This PR doesn't do anything yet, besides creating an empty fulltextindex
folder if enabled. Indexing events is done in a separate PR.
2022-09-07 18:15:54 +02:00
Neil Alexander
847032df36
Update to matrix-org/gomatrixserverlib@637a173 2022-09-07 09:11:36 +01:00
Neil Alexander
70cd9a902c
State resolution fixes in applyAndAuthEvent (update to matrix-org/gomatrixserverlib#336) 2022-09-06 17:19:21 +01:00
Neil Alexander
bea3dbe77b
State and auth tweaks (update to matrix-org/gomatrixserverlib#335) 2022-09-05 17:00:54 +01:00
Neil Alexander
fea869b41f
Update P2P demos 2022-09-01 17:12:27 +01:00
Neil Alexander
02ec00b1bb
Check for "creator" field in m.room.create during event auth (update to matrix-org/gomatrixserverlib@c71e518) 2022-08-30 17:40:46 +01:00
Neil Alexander
bbb3ade4a2
Fix the Cat Overflow bug (update to matrix-org/gomatrixserverlib@2020bdc) 2022-08-30 11:44:26 +01:00
Till Faelligen
aba171d9bc
Default to limit of 20 state events 2022-08-29 07:23:46 +02:00
Neil Alexander
a3eb4e5e98
Enforce type, sender, state_key and room_id lengths using codepoints rather than bytes (update to matrix-org/gomatrixserverlib@5f66df0, matrix-org/matrix-spec#1001) 2022-08-28 09:52:03 +01:00
Till
07dd9bd995
SyncAPI tweaks/fixes (#2671)
- Reverts 9dc57122d9 as it was causing issues https://github.com/matrix-org/dendrite/issues/2660
- Updates the GMSL `DefaultStateFilter` to use a limit of 20 events
- Uses the timeline events to determine the new position instead of the state events
2022-08-25 13:42:47 +01:00
Neil Alexander
bcdbd5c00a
Version 0.9.3 2022-08-15 10:56:54 +01:00
Neil Alexander
9fa30f5d3c
Update NATS Server and nats.go libraries 2022-08-12 13:29:09 +01:00
Till
05cafbd197
Implement history visibility on /messages, /context, /sync (#2511)
* Add possibility to set history_visibility and user AccountType

* Add new DB queries

* Add actual history_visibility changes for /messages

* Add passing tests

* Extract check function

* Cleanup

* Cleanup

* Fix build on 386

* Move ApplyHistoryVisibilityFilter to internal

* Move queries to topology table

* Add filtering to /sync and /context
Some cleanup

* Add passing tests; Remove failing tests :(

* Re-add passing tests

* Move filtering to own function to avoid duplication

* Re-add passing test

* Use newly added GMSL HistoryVisibility

* Update gomatrixserverlib

* Set the visibility when creating events

* Default to shared history visibility

* Remove unused query

* Update history visibility checks to use gmsl
Update tests

* Remove unused statement

* Update migrations to set "correct" history visibility

* Add method to fetch the membership at a given event

* Tweaks and logging

* Use actual internal rsAPI, default to shared visibility in tests

* Revert "Move queries to topology table"

This reverts commit 4f0d41be9c194a46379796435ce73e79203edbd6.

* Remove noise/unneeded code

* More cleanup

* Try to optimize database requests

* Fix imports

* PR peview fixes/changes

* Move setting history visibility to own migration, be more restrictive

* Fix unit tests

* Lint

* Fix missing entries

* Tweaks for incremental syncs

* Adapt generic changes

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: kegsay <kegan@matrix.org>
2022-08-11 18:23:35 +02:00
Neil Alexander
376391d1c7
Update Pinecone 2022-08-03 10:38:36 +01:00
Neil Alexander
645f31ae24
Update NATS Server and nats.go to use upstream 2022-08-01 09:51:13 +01:00
Neil Alexander
ccea23cd40
Remove room_id field from MSC2946 stripped events (closes #2588) 2022-07-25 11:42:22 +01:00
Till
081f5e7226
Update database migrations, remove goose (#2264)
* Add new db migration

* Update migrations
Remove goose

* Add possibility to test direct upgrades

* Try to fix WASM test

* Add checks for specific migrations

* Remove AddMigration
Use WithTransaction
Add Dendrite version to table

* Fix linter issues

* Update tests

* Update comments, outdent if

* Namespace migrations

* Add direct upgrade tests, skipping over one version

* Split migrations

* Update go version in CI

* Fix copy&paste mistake

* Use contexts in migrations

Co-authored-by: kegsay <kegan@matrix.org>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-07-25 10:39:22 +01:00
Till
a7e92f8cb9
History visibility database changes (#2533)
* Add new history_visibility column

* Update SQL queries to include history_visibility

* Store the history visibilty calculated by the roomserver

* Update GMSL

* Update migrations

* Fix migration

* Update GMSL

* Fix `go.sum`

* Update GMSL to use sql.Scanner & sql.Valuer

* Re-order migration/table creation

* Update gomatrixserverlib

* Add history_visibility column to current_room_state

* Fix migrations

* Return error instead of Fatal log

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-07-18 14:46:15 +02:00
Neil Alexander
9cd8e9d4b9
Use /v3 to request media from remote servers (update to matrix-org/gomatrixserverlib#324) 2022-07-13 09:32:03 +01:00
Neil Alexander
3ea21273bc
Ristretto cache (#2563)
* Try Ristretto cache

* Tweak

* It's beautiful

* Update GMSL

* More strict keyable interface

* Fix that some more

* Make less panicky

* Don't enforce mutability checks for now

* Determine mutability using deep equality

* Tweaks

* Namespace keys

* Make federation caches mutable

* Update cost estimation, add metric

* Update GMSL

* Estimate cost for metrics better

* Reduce counters a bit

* Try caching events

* Some guards

* Try again

* Try this

* Use separate caches for hopefully better hash distribution

* Fix bug with admitting events into cache

* Try to fix bugs

* Check nil

* Try that again

* Preserve order jeezo this is messy

* thanks VS Code for doing exactly the wrong thing

* Try this again

* Be more specific

* aaaaargh

* One more time

* That might be better

* Stronger sorting

* Cache expiries, async publishing of EDUs

* Put it back

* Use a shared cache again

* Cost estimation fixes

* Update ristretto

* Reduce counters a bit

* Clean up a bit

* Update GMSL

* 1GB

* Configurable cache sizees

* Tweaks

* Add `config.DataUnit` for specifying friendly cache sizes

* Various tweaks

* Update GMSL

* Add back some lazy loading caching

* Include key in cost

* Include key in cost

* Tweak max age handling, config key name

* Only register prometheus metrics if requested

* Review comments @S7evinK

* Don't return errors when creating caches (it is better just to crash since otherwise we'll `nil`-pointer exception everywhere)

* Review comments

* Update sample configs

* Update GHA Workflow

* Update Complement images to Go 1.18

* Remove the cache test from the federation API as we no longer guarantee immediate cache admission

* Don't check the caches in the renewal test

* Possibly fix the upgrade tests

* Update to matrix-org/gomatrixserverlib#322

* Update documentation to refer to Go 1.18
2022-07-11 14:31:31 +01:00
Neil Alexander
eb8dc50a97
Update Pinecone to matrix-org/pinecone@1ce778f 2022-07-08 14:54:08 +01:00
Till
89cd0e8fc1
Try to fix backfilling (#2548)
* Try to fix backfilling

* Return start/end to not confuse clients

* Update GMSL

* Update GMSL
2022-07-01 11:49:26 +02:00
Jean Lucas
920a20821b
Fix nats.go commit (#2540)
Signed-off-by: Jean Lucas <jean@4ray.co>
2022-06-27 09:15:19 +01:00
Neil Alexander
4c2a10f1a6
Handle state before, send history visibility in output (#2532)
* Check state before event

* Tweaks

* Refactor a bit, include in output events

* Don't waste time if soft failed either

* Tweak control flow, comments, use GMSL history visibility type
2022-06-13 15:11:10 +01:00
Neil Alexander
c4df6d7723
Optimise state res v2 by only updating the allower when necessary (update to matrix-org/gomatrixserverlib@e55d796) 2022-06-07 15:34:58 +01:00
Neil Alexander
b21a2223ef
Optimise state res v2 by parsing power level content less often (update to matrix-org/gomatrixserverlib@e938c07) 2022-06-07 15:04:00 +01:00
Neil Alexander
3d9fe20748
Fix bugs related to state resolution (#2507)
* Fix bugs related to state resolution

* Clean up `resolve-state`

* Don't panic when entries can't be found

* Ensure we have state entries for the auth events

* Revert "Ensure we have state entries for the auth events"

This reverts commit 9b13b7ed37f40ce6d1301d9cb423a27b0db9c897.

* Revert "Revert "Ensure we have state entries for the auth events""

This reverts commit d86db197e3e317f7d64ec6722cc60533872f4617.

* Fix bug

* Try that again

* Update gomatrixserverlib

* Remove recursion from `loadAuthEvents`
2022-06-01 09:46:21 +01:00
Neil Alexander
500124dd18
Add support for room version 10 (update to matrix-org/gomatrixserverlib#315) 2022-05-30 09:50:23 +01:00
Neil Alexander
b541f3043f
Add support for MSC3787 and org.matrix.msc3787 room version (update to matrix-org/gomatrixserverlib#310) 2022-05-26 15:08:17 +01:00
Neil Alexander
9e46d5025a
Upgrade various dependencies (#2496)
* Upgrade various dependencies

* Yaml dependency too
2022-05-26 12:34:43 +01:00
Neil Alexander
5a878b6e14
Update to matrix-org/gomatrixserverlib#314 2022-05-26 11:52:39 +01:00
Neil Alexander
ff53398635
Mark room versions 8 and 9 as stable 2022-05-25 16:39:51 +01:00
Neil Alexander
ac4d0072cf
Fix using the remote provided event from /send_join (#2490) 2022-05-25 14:31:07 +01:00
Neil Alexander
d621dd2986
Allow authing invited user in rooms with "restricted" join rule (update to matrix-org/gomatrixserverlib#313) 2022-05-24 11:08:28 +01:00
Neil Alexander
8a7567c652
Fix control flow related to joining restricted v8/v9 rooms (update to matrix-org/gomatrixserverlib#312) 2022-05-23 17:49:48 +01:00
Neil Alexander
028963af1c
Update to matrix-org/gomatrixserverlib#311 (#2479)
* Update to matrix-org/gomatrixserverlib#311

* Update to squash commit
2022-05-23 14:26:56 +01:00
Neil Alexander
be9be2553f
Resolve over old and new extremities (#2457)
* Feed existing state into state res when calculating state from new extremities

* Remove duplicates

* Fix bug

* Sort and unique

* Update to matrix-org/gomatrixserverlib#308

* Trim the slice properly

* Update gomatrixserverlib again

* Update to matrix-org/gomatrixserverlib#308
2022-05-13 11:52:04 +01:00
Neil Alexander
cafc2d2c10
Update NATS Server to version 2.8.2 (#2460) 2022-05-13 11:36:04 +01:00
Till
f69ebc6af2
Add roomserver tests (1/?) (#2434)
* Add EventJSONTable tests

* Add eventJSON tests

* Add EventStateKeysTable tests

* Add EventTypesTable tests

* Add Events Table tests
Move variable declaration outside loops
Switch to testify/assert for tests

* Move variable declaration outside loop

* Remove random data

* Fix issue where the EventReferenceSHA256 is not set

* Add more tests

* Revert "Fix issue where the EventReferenceSHA256 is not set"

This reverts commit 8ae34c4e5f78584f0edb479f5a893556d2b95d19.

* Update GMSL

* Add tests for duplicate entries

* Test what happens if we select non-existing NIDs

* Add test for non-existing eventType

* Really update GMSL
2022-05-09 15:30:32 +02:00
Neil Alexander
79e2fbc663
Update to matrix-org/gomatrixserverlib#307 2022-05-09 13:53:51 +01:00