Commit Graph

97 Commits

Author SHA1 Message Date
Joakim Recht
00217a69d1
Only fetch events once for all rooms (#3311)
This refactors `PDUStreamProvider` a bit so that it doesn't trigger a
database query per room, but instead utilizes the fact that it's
possible to bulk query. This improves sync performance significantly
when you have 1000s of rooms.

### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [x] I have added Go unit tests or [Complement integration
tests](https://github.com/matrix-org/complement) for this PR _or_ I have
justified why this PR doesn't need tests
* [x] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)
_or_ I have already signed off privately

Signed-off-by: `Joakim Recht <joakim@beyondwork.ai>`
2024-01-25 20:10:46 +01:00
devonh
db83789654
Move pseudoID ClientEvent hotswapping to a common location (#3199)
Fixes a variety of issues where clients were receiving pseudoIDs in
places that should be userIDs.
This change makes pseudoIDs work with sliding sync & element x.

---------

Co-authored-by: Till <2353100+S7evinK@users.noreply.github.com>
2023-09-15 15:25:09 +00:00
devonh
8245b24100
Update gmsl to use new validated RoomID on PDUs (#3200)
GMSL returns a `spec.RoomID` when calling `PDU.RoomID()`
2023-09-15 14:39:06 +00:00
Sam Wedgwood
058081e68e
[pseudoIDs] changing event ID fix (#3195)
power levels events in pseudo IDs sometimes changed event IDs (this was
already fixed earlier, but one of the edgecases was not covered, and is
now covered)

Signed-off-by: `Sam Wedgwood <sam@wedgwood.dev>`
2023-09-12 16:32:24 +01:00
devonh
bb2ab62cbf
Handle event_format federation in /sync responses (#3192) 2023-08-31 15:33:38 +00:00
Sam Wedgwood
9a12420428
[pseudoID] More pseudo ID fixes (#3167)
Signed-off-by: `Sam Wedgwood <sam@wedgwood.dev>`
2023-08-15 12:37:04 +01:00
Sam Wedgwood
af13fa1c75
[pseudoIDs] Fixes for room alias tests (#3159)
Some (deceptively) simple fixes for some bugs that caused room alias
tests to fail (sytext `tests/30rooms/05aliases.pl`). Each commit has
details about what it fixes.

Sytest results:

- Sytest before (79d4a0e):
https://gist.github.com/swedgwood/972ac4ef93edd130d3db0930703d6c82
- Sytest after (4b09bed):
https://gist.github.com/swedgwood/504b00ac4ee892acb757b7fac55fa28a

Room aliases go from `8/15` to `15/15`, but looks like these fixes also
managed to fix about `4` other tests, which is a nice bonus :)

Signed-off-by: `Sam Wedgwood <sam@wedgwood.dev>`
2023-07-31 14:39:41 +01:00
Till
74a5ab6c24
Fix issues reported by Sentry (#3143)
This should fix a few issues reported by Sentry
2023-07-07 22:00:10 +02:00
Till
23cd7877a1
Add MXIDMapping for pseudoID rooms (#3112)
Add `MXIDMapping` on membership events when
creating/joining rooms.
2023-06-28 20:29:49 +02:00
devonh
e4665979bf
Merge SenderID & Per Room User Key work (#3109) 2023-06-14 14:23:46 +00:00
devonh
77d9e4e93d
Cleanup remaining statekey usage for senderIDs (#3106) 2023-06-12 11:19:25 +00:00
devonh
8ea1a11105
Use SenderID Type (#3105) 2023-06-07 17:14:35 +00:00
devonh
7a1fd7f512
PDU Sender split (#3100)
Initial cut of splitting PDU Sender into SenderID & looking up UserID where required.
2023-06-06 20:55:18 +00:00
Till
d11da6ec7c
Fix newly found linter issues (#3099)
Fixes the issues found in
https://github.com/matrix-org/dendrite/actions/runs/5155539352/jobs/9285342056#step:5:22.
Only naked returns in longer functions.
2023-06-02 15:48:04 +02:00
kegsay
1432743d1a
Use PDU in more places (#3072) 2023-04-28 16:00:22 +01:00
kegsay
6171310307
Use PDU interface (#3070)
We only use it in a few places currently, enough to get things to
compile and run. We should be using it in much more places.

Similarly, in some places we cast []PDU back to []*Event, we need to not
do that. Likewise, in some places we cast PDU to *Event, we need to not
do that. For now though, hopefully this is a start.
2023-04-27 16:35:19 +01:00
kegsay
b189edf4f4
Remove gmsl.HeaderedEvent (#3068)
Replaced with types.HeaderedEvent _for now_. In reality we want to move
them all to gmsl.Event and only use HeaderedEvent when we _need_ to
bundle the version/event ID with the event (seriailsation boundaries,
and even then only when we don't have the room version).

Requires https://github.com/matrix-org/gomatrixserverlib/pull/373
2023-04-27 12:54:20 +01:00
kegsay
72285b2659
refactor: update GMSL (#3058)
Sister PR to https://github.com/matrix-org/gomatrixserverlib/pull/364

Read this commit by commit to avoid going insane.
2023-04-19 15:50:33 +01:00
kegsay
3691423626
Move GMSL client types to Dendrite (#3045)
GMSL is intended for Federation only. Sister PR to
https://github.com/matrix-org/gomatrixserverlib/pull/357
2023-04-04 17:16:53 +00:00
Till
4594233f89
Merge keyserver & userapi (#2972)
As discussed yesterday, a first draft of merging the keyserver and the
userapi.
2023-02-20 14:58:03 +01:00
Till
eb29a31550
Optimize /sync and history visibility (#2961)
Should fix the following issues or make a lot less worse when using
Postgres:

The main issue behind #2911: The client gives up after a certain time,
causing a cascade of context errors, because the response couldn't be
built up fast enough. This mostly happens on accounts with many rooms,
due to the inefficient way we're getting recent events and current state

For #2777: The queries for getting the membership events for history
visibility were being executed for each room (I think 185?), resulting
in a whooping 2k queries for membership events. (Getting the
statesnapshot -> block nids -> actual wanted membership event)

Both should now be better by:
- Using a LATERAL join to get all recent events for all joined rooms in
one go (TODO: maybe do the same for room summary and current state etc)
- If we're lazy loading on initial syncs, we're now not getting the
whole current state, just to drop the majority of it because we're lazy
loading members - we add a filter to exclude membership events on the
first call to `CurrentState`.
- Using an optimized query to get the membership events needed to
calculate history visibility

---------

Co-authored-by: kegsay <kegan@matrix.org>
2023-02-07 14:31:23 +01:00
Till
0d0280cf5f
/sync performance optimizations (#2927)
Since #2849 there is no limit for the current state we fetch to
calculate history visibility. In large rooms this can cause us to fetch
thousands of membership events we don't really care about.
This now only gets the state event types and senders in our timeline,
which should significantly reduce the amount of events we fetch from the
database.

Also removes `MaxTopologicalPosition`, as it is an unnecessary DB call,
given we use the result in `topological_position < $1` calls.
2023-01-17 10:08:23 +01:00
Till
0491a8e343
Fix room summary returning wrong heroes (#2930)
This should fix #2910.
Probably makes Sytest/Complement a bit upset, since this not using
`sort.Strings` anymore.
2023-01-12 10:06:03 +01:00
Till
c136a450d5
Fix newly joined users presence (#2854)
Fixes #2803 
Also refactors the presence stream to not hit the database for every
user, instead queries all users at once now.
2022-12-08 08:25:03 +01:00
Till
f8d1dc521d
Fix m.receipts causing notifications (#2893)
Fixes https://github.com/matrix-org/dendrite/issues/2353
2022-11-29 15:46:28 +01:00
Erik Johnston
31f56ac3f4
Never filter out a user's own membership when using LL (#2887) 2022-11-22 21:38:27 +00:00
Till
efe28db631
Update latestPosition when getting reversed room delta (#2860)
Regression test added in
https://github.com/matrix-org/complement/pull/551
Should fix https://github.com/matrix-org/dendrite/issues/2514?
2022-11-04 15:39:09 +01:00
Neil Alexander
98d3f88bfb
Move prev_batch calculation (#2856)
This might help #2847.
2022-11-03 16:56:21 +00:00
Neil Alexander
8704e84898
Tweak removeDuplicates calls to use events instead of recentEvents (#2853)
... since `events` is *after* history visibility filtering, not before
it.
2022-11-03 10:19:37 +00:00
Tak Wai Wong
75a508cc27
Fix issue where a member is forced to leave a room when the invite is marked deleted (#2839)
Proposed fix for issue:
https://github.com/matrix-org/dendrite/issues/2838

Suppose bob received invites to spaceA and spaceB.
When Bob joins spaceA, we add an OutputEvent event to retire the invite.
This sets the invite to "deleted" in the database. This makes sense.

The bug is in stream_invites.go. Triggered when bob received a new
invite for spaceB, and does a client sync.

In the block (line 76)
`for roomID := range retiredInvites

   if _, ok := req.Response.Rooms.Invite[roomID]; ok {
	continue
   }

   if _, ok := req.Response.Rooms.Join[roomID]; ok {
	continue
  }
...
` 

Bob is not in either maps even though he had just accepted the invite
for spaceA. Consequently, the spaceA invite is treated as a retired
invite, and a membership Leave event is generated. What bob sees is that
after accepting the invite to spaceB, he lose access to spaceA.


### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [ ] I have added tests for PR _or_ I have justified why this PR
doesn't need tests.
* [x ] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)
_or_ I have already signed off privately

Signed-off-by: `Tak Wai Wong <tak@hntlabs.com>`

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-11-02 10:02:23 +00:00
Neil Alexander
3db9e98456
Don't limit "state" (#2849)
This is apparently some incorrect behaviour that we built as a result of
a spec bug (matrix-org/matrix-spec#1314) where we were applying a filter
to the `"state"` section of the `/sync` response incorrectly. The client
then has no way to know that the state was limited.

This PR removes the state limiting, which probably also helps #2842.
2022-11-02 09:34:19 +00:00
Neil Alexander
0b21cb78aa
Try to fix a panic in the sync API PDU stream 2022-11-01 14:45:15 +00:00
Neil Alexander
7307701a24
Tweak "state" and "timeline" filtering (#2844)
This should stop state events disappearing down a gap where we'd try to
separate out the sections *before* applying history visibility instead
of after.

This may be a better approach than #2843 but I hope @tak-hntlabs will
shout if it isn't.
2022-10-31 15:14:08 +00:00
Till
69aff372f3
Limit recent events when going backwards (#2840)
If we're going backwards, we were selecting potentially thousands of
events, which in turn were fed to history visibility checks, resulting
in bad sync performance.
2022-10-28 13:40:51 +02:00
Till Faelligen
f6035822e7
Simplify error checking and check the correct error 2022-10-28 08:17:40 +02:00
Till
c62ac3d6ad
Fix Current state appears in timeline in private history with many messages after (#2830)
The problem was that we weren't getting enough recent events, as most of
them were removed by the history visibility filter. Now we're getting
all events between the given input range and re-slice the returned
values after applying history visibility.
2022-10-25 15:15:24 +02:00
Till
313cb3fd19
Filter /members, return members at given point (#2827)
Makes the tests
```
Can get rooms/{roomId}/members at a given point
Can filter rooms/{roomId}/members
```
pass, by moving `/members` and `/joined_members` to the SyncAPI.
2022-10-25 12:39:10 +02:00
Till
40cfb9a4ea
Fix invite -> leave -> join dance when accepting invites (#2817)
As mentioned in
https://github.com/matrix-org/dendrite/issues/2361#issuecomment-1139394565
and observed by ourselves, this should fix the odd `invite -> leave ->
join` dance when accepting invites.
2022-10-21 09:26:22 +01:00
Till
e79bfd8fd5
Get state deltas without filters (#2810)
This makes the following changes:
- get state deltas without the user supplied filter, so we can actually
"calculate" state transitions
- closes `stmt` when using SQLite
- Adds presence for users who newly joined a room, even if the syncing
user already knows about the presence status (should fix
https://github.com/matrix-org/complement/pull/516)
2022-10-19 14:05:39 +02:00
Till
a8bc558a60
Always add UnreadNotifications to joined room reponses (#2793)
Fixes a minor bug, where we failed to add `UnreadNotifications` to the
join response, if it wasn't in
`GetUserUnreadNotificationCountsForRooms`.
2022-10-14 10:38:12 +02:00
Neil Alexander
0a9aebdf01
Private read receipts (#2789)
Implement behaviours for `m.read.private` receipts.
2022-10-11 12:27:21 +01:00
Till
8c5b166784
Use the stream positions of the notifier (#2768)
Use the stream positions of the notifier, which might have advanced
since setting it at the beginning of the loop. This possibly helps in
reducing roundtrips to the SyncAPI, just because we didn't fetch the
latest data.
Also fixes a minor oversight in the receipts stream.
2022-10-06 11:57:13 +01:00
Till
0f777d421c
Remove empty fields from /sync response (#2755)
First attempt at removing empty fields from `/sync` responses. Needs
https://github.com/matrix-org/sytest/pull/1298 to keep Sytest happy.

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-10-05 13:47:13 +01:00
Neil Alexander
d32f60249d
Modify sync transaction behaviour (#2758)
This now uses a transaction per stream, so that errors in one stream
don't propagate to another, and we therefore no longer need to do hacks
to reopen a new transaction after aborting a failed one.
2022-10-03 11:38:20 +01:00
Neil Alexander
0116db79c6
Reset transaction after a failure 2022-09-30 17:07:37 +01:00
Neil Alexander
16048be236
Handle case when applying history visibility failed 2022-09-30 16:55:10 +01:00
Neil Alexander
7d9545ceea
Another /sync fix 2022-09-30 16:34:06 +01:00
Neil Alexander
ee40a29e55
Fix broken /sync due to transaction error 2022-09-30 16:07:18 +01:00
Neil Alexander
6348486a13
Transactional isolation for /sync (#2745)
This should transactional snapshot isolation for `/sync` etc requests.

For now we don't use repeatable read due to some odd test failures with
invites.
2022-09-30 12:48:10 +01:00
Till
249b32c4f3
Refactor notifications (#2688)
This PR changes the handling of notifications
- removes the `StreamEvent` and `ReadUpdate` stream
- listens on the `OutputRoomEvent` stream in the UserAPI to inform the
SyncAPI about unread notifications
- listens on the `OutputReceiptEvent` stream in the UserAPI to set
receipts/update notifications
- sets the `read_markers` directly from within the internal UserAPI

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-09-27 15:01:34 +02:00