Commit Graph

397 Commits

Author SHA1 Message Date
kegsay
72285b2659
refactor: update GMSL (#3058)
Sister PR to https://github.com/matrix-org/gomatrixserverlib/pull/364

Read this commit by commit to avoid going insane.
2023-04-19 15:50:33 +01:00
Till
9fa39263c0
Add sync API db tests (#3043)
Co-authored-by: kegsay <kegan@matrix.org>
2023-04-17 10:25:33 +01:00
kegsay
3691423626
Move GMSL client types to Dendrite (#3045)
GMSL is intended for Federation only. Sister PR to
https://github.com/matrix-org/gomatrixserverlib/pull/357
2023-04-04 17:16:53 +00:00
Till
e8b2162a01
Add /search tests (#3025) 2023-03-27 11:26:52 +02:00
Till
234ed603e6
Move every db.Prepare to sqlutil.Statementlist, remove trace driver (#3026)
Doesn't buy us much, but makes everything a bit more consistent.

Also removes the SQL trace driver, as it is unused and the output is
hard to read anyway.
2023-03-23 13:52:53 +01:00
Till Faelligen
6948d16527
Update Go, use go tool covdata for coverage files? 2023-03-22 14:50:21 +01:00
Till
5e85a00cb3
Remove BaseDendrite (#3023)
Removes `BaseDendrite` to, hopefully, make testing and composing of
components easier in the future.
2023-03-22 09:21:32 +01:00
Till
5579121c6f
Preparations for removing BaseDendrite (#3016)
Preparations to actually remove/replace `BaseDendrite`.
Quite a few changes:
- SyncAPI accepts an `fulltext.Indexer` interface (fulltext is removed
from `BaseDendrite`)
- Caches are removed from `BaseDendrite`
- Introduces a `Router` struct (likely to change)
  - also fixes #2903
- Introduces a `sqlutil.ConnectionManager`, which should remove
`base.DatabaseConnection` later on
- probably more
2023-03-17 11:09:45 +00:00
Till
7d83f8b633
Add tests for UpdateRelations (#2999)
This also fixes an issue regarding updates to relations for invalid
events, which could result in us retrying said event over and over
again, if we fail to unmarshal the event to
`gomatrixserverlib.RelationContent`, this was discovered by
`@sleroq:virto.community`
2023-03-06 12:43:59 +01:00
Till
6c20f8f742
Refactor StoreEvent, add MaybeRedactEvent, create an EventDatabase (#2989)
This PR changes the following:
- `StoreEvent` now only stores an event (and possibly prev event),
instead of also doing redactions
- Adds a `MaybeRedactEvent` (pulled out from `StoreEvent`), which should
be called after storing events
- a few other things
2023-03-01 17:06:47 +01:00
Till
4594233f89
Merge keyserver & userapi (#2972)
As discussed yesterday, a first draft of merging the keyserver and the
userapi.
2023-02-20 14:58:03 +01:00
Till
11d9b9db0e
Remove polylith/API mode (#2967)
This removes most of the code used for polylith/API mode.

This removes the `/api` internal endpoints entirely. 

Binary size change roughly 5%: 
```
51437560 Feb 13 10:15 dendrite-monolith-server # old
48759008 Feb 13 10:15 dendrite-monolith-server # new
```
2023-02-14 12:47:47 +01:00
Till
eb29a31550
Optimize /sync and history visibility (#2961)
Should fix the following issues or make a lot less worse when using
Postgres:

The main issue behind #2911: The client gives up after a certain time,
causing a cascade of context errors, because the response couldn't be
built up fast enough. This mostly happens on accounts with many rooms,
due to the inefficient way we're getting recent events and current state

For #2777: The queries for getting the membership events for history
visibility were being executed for each room (I think 185?), resulting
in a whooping 2k queries for membership events. (Getting the
statesnapshot -> block nids -> actual wanted membership event)

Both should now be better by:
- Using a LATERAL join to get all recent events for all joined rooms in
one go (TODO: maybe do the same for room summary and current state etc)
- If we're lazy loading on initial syncs, we're now not getting the
whole current state, just to drop the majority of it because we're lazy
loading members - we add a filter to exclude membership events on the
first call to `CurrentState`.
- Using an optimized query to get the membership events needed to
calculate history visibility

---------

Co-authored-by: kegsay <kegan@matrix.org>
2023-02-07 14:31:23 +01:00
Till
baf118b08c
Add Sytest/Complement coverage to scheduled runs (#2962)
This adds Sytest and Complement coverage reporting to the nightly
scheduled CI runs.

Fixes a few API mode related issues as well, since we seemingly never
really ran them with Complement.

Also fixes a bug related to device list changes: When we pass in an
empty `newlyLeftRooms` slice, we got a list of all currently joined
rooms with the corresponding members. When we then got the
`newlyJoinedRooms`, we wouldn't update the `changed` slice, because we
already got the user from the `newlyLeftRooms` query. This is fixed by
simply ignoring empty `newlyLeftRooms`.
2023-02-03 13:42:35 +01:00
Neil
738686ae68
Add /_dendrite/admin/purgeRoom/{roomID} (#2662)
This adds a new admin endpoint `/_dendrite/admin/purgeRoom/{roomID}`. It
completely erases all database entries for a given room ID.

The roomserver will start by clearing all data for that room and then
will generate an output event to notify downstream components (i.e. the
sync API and federation API) to do the same.

It does not currently clear media and it is currently not implemented
for SQLite since it relies on SQL array operations right now.

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: Till Faelligen <2353100+S7evinK@users.noreply.github.com>
2023-01-19 21:02:32 +01:00
Till
0d0280cf5f
/sync performance optimizations (#2927)
Since #2849 there is no limit for the current state we fetch to
calculate history visibility. In large rooms this can cause us to fetch
thousands of membership events we don't really care about.
This now only gets the state event types and senders in our timeline,
which should significantly reduce the amount of events we fetch from the
database.

Also removes `MaxTopologicalPosition`, as it is an unnecessary DB call,
given we use the result in `topological_position < $1` calls.
2023-01-17 10:08:23 +01:00
Umar Getagazov
8582c7520a
Omit state field from /messages response if empty (#2940)
The field type is `[ClientEvent]` in the
[spec](https://spec.matrix.org/v1.5/client-server-api/#get_matrixclientv3roomsroomidmessages),
but right now `null` can also be returned. Omit the field completely if
it's empty. Some clients (rightfully) assume it's either not present at
all or it's of the right type (see
https://github.com/matrix-org/matrix-react-sdk/pull/9913).

### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [x] I have added Go unit tests or [Complement integration
tests](https://github.com/matrix-org/complement) for this PR _or_ I have
justified why this PR doesn't need tests
    * The PR is a simple struct tag fix
* [x] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)
_or_ I have already signed off privately

Signed-off-by: `Umar Getagazov <umar@handlerug.me>`

Signed-off-by: Umar Getagazov <umar@handlerug.me>
2023-01-17 09:07:42 +01:00
Till
0491a8e343
Fix room summary returning wrong heroes (#2930)
This should fix #2910.
Probably makes Sytest/Complement a bit upset, since this not using
`sort.Strings` anymore.
2023-01-12 10:06:03 +01:00
Till
c136a450d5
Fix newly joined users presence (#2854)
Fixes #2803 
Also refactors the presence stream to not hit the database for every
user, instead queries all users at once now.
2022-12-08 08:25:03 +01:00
Neil Alexander
ba2ffb7da9
Repeatable reads for /sync (#2783)
This puts repeatable reads into all sync streams.

Co-authored-by: kegsay <kegan@matrix.org>
2022-12-06 18:16:17 +00:00
Till
f8d1dc521d
Fix m.receipts causing notifications (#2893)
Fixes https://github.com/matrix-org/dendrite/issues/2353
2022-11-29 15:46:28 +01:00
Erik Johnston
31f56ac3f4
Never filter out a user's own membership when using LL (#2887) 2022-11-22 21:38:27 +00:00
Neil Alexander
6650712a1c
Federation fixes for virtual hosting 2022-11-15 15:05:23 +00:00
Till
d35a5642e8
Deny guest access on several endpoints (#2873)
Second part for guest access, this adds a `WithAllowGuests()` option to
`MakeAuthAPI`, allowing guests to access the specified endpoints.
Endpoints taken from the
[spec](https://spec.matrix.org/v1.4/client-server-api/#client-behaviour-14)
and by checking Synapse endpoints for `allow_guest=true`.
2022-11-11 10:52:08 +01:00
Till
efe28db631
Update latestPosition when getting reversed room delta (#2860)
Regression test added in
https://github.com/matrix-org/complement/pull/551
Should fix https://github.com/matrix-org/dendrite/issues/2514?
2022-11-04 15:39:09 +01:00
Neil Alexander
98d3f88bfb
Move prev_batch calculation (#2856)
This might help #2847.
2022-11-03 16:56:21 +00:00
Neil Alexander
8704e84898
Tweak removeDuplicates calls to use events instead of recentEvents (#2853)
... since `events` is *after* history visibility filtering, not before
it.
2022-11-03 10:19:37 +00:00
Tak Wai Wong
75a508cc27
Fix issue where a member is forced to leave a room when the invite is marked deleted (#2839)
Proposed fix for issue:
https://github.com/matrix-org/dendrite/issues/2838

Suppose bob received invites to spaceA and spaceB.
When Bob joins spaceA, we add an OutputEvent event to retire the invite.
This sets the invite to "deleted" in the database. This makes sense.

The bug is in stream_invites.go. Triggered when bob received a new
invite for spaceB, and does a client sync.

In the block (line 76)
`for roomID := range retiredInvites

   if _, ok := req.Response.Rooms.Invite[roomID]; ok {
	continue
   }

   if _, ok := req.Response.Rooms.Join[roomID]; ok {
	continue
  }
...
` 

Bob is not in either maps even though he had just accepted the invite
for spaceA. Consequently, the spaceA invite is treated as a retired
invite, and a membership Leave event is generated. What bob sees is that
after accepting the invite to spaceB, he lose access to spaceA.


### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [ ] I have added tests for PR _or_ I have justified why this PR
doesn't need tests.
* [x ] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)
_or_ I have already signed off privately

Signed-off-by: `Tak Wai Wong <tak@hntlabs.com>`

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-11-02 10:02:23 +00:00
Neil Alexander
3db9e98456
Don't limit "state" (#2849)
This is apparently some incorrect behaviour that we built as a result of
a spec bug (matrix-org/matrix-spec#1314) where we were applying a filter
to the `"state"` section of the `/sync` response incorrectly. The client
then has no way to know that the state was limited.

This PR removes the state limiting, which probably also helps #2842.
2022-11-02 09:34:19 +00:00
ash lea
5aaa60227a
return required room_id field in /members (#2846)
### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [ ] I have added tests for PR _or_ I have justified why this PR
doesn't need tests.
* [x] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)
_or_ I have already signed off privately

Signed-off-by: `ash lea <example@thisismyactual.email>`
2022-11-01 16:42:07 +00:00
Neil Alexander
0b21cb78aa
Try to fix a panic in the sync API PDU stream 2022-11-01 14:45:15 +00:00
Neil Alexander
7307701a24
Tweak "state" and "timeline" filtering (#2844)
This should stop state events disappearing down a gap where we'd try to
separate out the sections *before* applying history visibility instead
of after.

This may be a better approach than #2843 but I hope @tak-hntlabs will
shout if it isn't.
2022-10-31 15:14:08 +00:00
Till
69aff372f3
Limit recent events when going backwards (#2840)
If we're going backwards, we were selecting potentially thousands of
events, which in turn were fed to history visibility checks, resulting
in bad sync performance.
2022-10-28 13:40:51 +02:00
Till Faelligen
f6035822e7
Simplify error checking and check the correct error 2022-10-28 08:17:40 +02:00
Till
a169a9121a
Fix /members (#2837)
Fixes a bug introduced in #2827, where the SyncAPI might not have all
requested eventIDs, resulting in too few members returned.
2022-10-27 14:18:22 +02:00
Till
c62ac3d6ad
Fix Current state appears in timeline in private history with many messages after (#2830)
The problem was that we weren't getting enough recent events, as most of
them were removed by the history visibility filter. Now we're getting
all events between the given input range and re-slice the returned
values after applying history visibility.
2022-10-25 15:15:24 +02:00
Till Faelligen
8b7bf5e7d7
Return forbidden if not a member anymore (fix #2802) 2022-10-25 15:00:52 +02:00
Till
313cb3fd19
Filter /members, return members at given point (#2827)
Makes the tests
```
Can get rooms/{roomId}/members at a given point
Can filter rooms/{roomId}/members
```
pass, by moving `/members` and `/joined_members` to the SyncAPI.
2022-10-25 12:39:10 +02:00
Till
7506e3303e
Get messages from before user left the room (#2824)
This is going to make `Can get rooms/{roomId}/messages for a departed
room (SPEC-216)` pass, since we now only grep events from before the
user left the room.
2022-10-24 17:03:04 +02:00
Till
3cf42a1d64
Add syncapi_memberships table tests (#2805) 2022-10-21 12:53:04 +02:00
Till
40cfb9a4ea
Fix invite -> leave -> join dance when accepting invites (#2817)
As mentioned in
https://github.com/matrix-org/dendrite/issues/2361#issuecomment-1139394565
and observed by ourselves, this should fix the odd `invite -> leave ->
join` dance when accepting invites.
2022-10-21 09:26:22 +01:00
Till
e79bfd8fd5
Get state deltas without filters (#2810)
This makes the following changes:
- get state deltas without the user supplied filter, so we can actually
"calculate" state transitions
- closes `stmt` when using SQLite
- Adds presence for users who newly joined a room, even if the syncing
user already knows about the presence status (should fix
https://github.com/matrix-org/complement/pull/516)
2022-10-19 14:05:39 +02:00
Till
a8bc558a60
Always add UnreadNotifications to joined room reponses (#2793)
Fixes a minor bug, where we failed to add `UnreadNotifications` to the
join response, if it wasn't in
`GetUserUnreadNotificationCountsForRooms`.
2022-10-14 10:38:12 +02:00
Neil Alexander
23a3e04579
Event relations (#2790)
This adds support for tracking `m.relates_to`, as well as adding support
for the various `/room/{roomID}/relations/...` endpoints to the CS API.
2022-10-13 14:50:52 +01:00
Neil Alexander
0a9aebdf01
Private read receipts (#2789)
Implement behaviours for `m.read.private` receipts.
2022-10-11 12:27:21 +01:00
Neil Alexander
3920b9f9b6
Tweak GetStateDeltas behaviour (#2788)
Improves the control flow of `GetStateDeltas` for clarity and possibly
also fixes a bug where duplicate state delta entries could be inserted
with different memberships instead of being correctly overridden by
`join`.
2022-10-11 10:58:34 +01:00
Till
0f09e9d196
Move /event to the SyncAPI (#2782)
This allows us to apply history visibility without having to recalculate
it in the roomserver.
Unblocks https://github.com/matrix-org/complement/pull/495, fix missing
part of https://github.com/matrix-org/dendrite/issues/617
2022-10-10 12:19:16 +02:00
Neil Alexander
1b5460a920
Ensure we only wake up a given user once (#2775)
This ensures that the sync API notifier only wakes up a given user once
for a given stream position.
2022-10-07 13:42:35 +01:00
Till
8c5b166784
Use the stream positions of the notifier (#2768)
Use the stream positions of the notifier, which might have advanced
since setting it at the beginning of the loop. This possibly helps in
reducing roundtrips to the SyncAPI, just because we didn't fetch the
latest data.
Also fixes a minor oversight in the receipts stream.
2022-10-06 11:57:13 +01:00
Till
0f777d421c
Remove empty fields from /sync response (#2755)
First attempt at removing empty fields from `/sync` responses. Needs
https://github.com/matrix-org/sytest/pull/1298 to keep Sytest happy.

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-10-05 13:47:13 +01:00