Commit Graph

2346 Commits

Author SHA1 Message Date
Neil Alexander
3f9e38e80a
Consistent *sql.Tx usage across sync API (#2744)
This tidies up the `storage` package so that everything takes a
transaction parameter instead of something things that do and some that
don't.
2022-09-28 10:18:03 +01:00
texuf
a574ed5369
Fix for `sql: converting argument $1 type: unsupported type []interfa… (#2743)
…ce {}, a slice of interface` in new notifications select

The sqlite3 version was just not working, original pr here:
https://github.com/matrix-org/dendrite/pull/2688

signed off by: austin ellis <austin@hntlabs.com>

This doesn't fix the notification counts, they still only work about 1
out of every 5 times in my tests. I will stick with my other fix locally
for reliable notification delivery:
https://github.com/matrix-org/dendrite/pull/2701
2022-09-28 06:19:34 +02:00
Neil Alexander
083ae01520
Promote reindexing log level 2022-09-27 17:30:40 +01:00
Neil Alexander
34993717fd
Update search docs 2022-09-27 17:10:47 +01:00
Till
87be32ca26
Fulltext implementation using Bleve (#2675)
Based on #2480

This actually indexes events based on their event type. They are removed
from the index if we receive a `m.room.redaction` event on the
`OutputRoomEvent` stream.
An admin endpoint is added to reindex all existing events.


Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-09-27 18:06:49 +02:00
Neil Alexander
6c67552bf9
Return M_UNRECOGNIZED for unknown CS API endpoints/actions (#2740)
Fixes #2739.
2022-09-27 15:50:22 +01:00
Till
249b32c4f3
Refactor notifications (#2688)
This PR changes the handling of notifications
- removes the `StreamEvent` and `ReadUpdate` stream
- listens on the `OutputRoomEvent` stream in the UserAPI to inform the
SyncAPI about unread notifications
- listens on the `OutputReceiptEvent` stream in the UserAPI to set
receipts/update notifications
- sets the `read_markers` directly from within the internal UserAPI

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-09-27 15:01:34 +02:00
Neil Alexander
f18bce93cc
Pinecone hybrid routing (update to matrix-org/pinecone#67) 2022-09-27 11:15:49 +01:00
Neil Alexander
d531202b0e
Build Docker images using Go 1.19 (related to #2714) 2022-09-27 10:52:03 +01:00
Dov Alperin
b5bfff47d2
Use /usr/bin/env bash in shebangs to make them universal (#2735)
Some systems (like nixos) don't have bash living at `/bin/bash` so using
`/usr/bin/env bash` we can make these scripts universal.

### Pull Request Checklist

<!-- Please read docs/CONTRIBUTING.md before submitting your pull
request -->

* [X] I have added added tests for PR _or_ I have justified why this PR
doesn't need tests.
* [x] Pull request includes a [sign
off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off)

Signed-off-by: `Dov Alperin <git@dov.dev>`

Signed-off-by: `Dov Alperin <git@dov.dev>`
2022-09-27 09:42:08 +01:00
PiotrKozimor
12649ccedd
Improve selectRoomIDsWithAnyMembershipSQL performance (#2738)
Recently I have observed that dendrite spends a lot of time (~390s) in
`selectRoomIDsWithAnyMembershipSQL` query

```
dendrite_syncapi=# select total_exec_time, left(query,100) from pg_stat_statements order by total_exec_time desc limit 5 ;
  total_exec_time   |                                                 left
--------------------+------------------------------------------------------------------------------------------------------
  747826.5800519128 | SELECT event_id, id, headered_event_json, session_id, exclude_from_sync, transaction_id, history_vis
  389130.5490339942 | SELECT DISTINCT room_id, membership FROM syncapi_current_room_state WHERE type = $2 AND state_key =
 376104.17514700035 | SELECT psd.datname, xact_commit, xact_rollback, blks_read, blks_hit, tup_returned, tup_fetched, tup_
   363644.164092031 | SELECT event_type_nid, event_state_key_nid, event_nid FROM roomserver_events WHERE event_nid = ANY($
  58570.48104699995 | SELECT event_id, headered_event_json FROM syncapi_current_room_state WHERE room_id = $1 AND ( $2::te
(5 rows)
```

Explain analyze showed correct usage of `syncapi_room_state_unique`
index:

```
dendrite_syncapi=#
explain analyze SELECT distinct room_id, membership FROM syncapi_current_room_state WHERE type = 'm.room.member' AND state_key = '@qjfl:dendrite.stg.globekeeper.com';
                                                                               QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Unique  (cost=2749.38..2749.56 rows=24 width=52) (actual time=2.933..2.956 rows=65 loops=1)
   ->  Sort  (cost=2749.38..2749.44 rows=24 width=52) (actual time=2.932..2.937 rows=65 loops=1)
         Sort Key: room_id, membership
         Sort Method: quicksort  Memory: 34kB
         ->  Index Scan using syncapi_room_state_unique on syncapi_current_room_state  (cost=0.41..2748.83 rows=24 width=52) (actual time=0.030..2.890 rows=65 loops=1)
               Index Cond: ((type = 'm.room.member'::text) AND (state_key = '@qjfl:dendrite.stg.globekeeper.com'::text))
 Planning Time: 0.140 ms
 Execution Time: 2.990 ms
(8 rows)
```

Multi-column indexes in Postgres shall perform well for leftmost
columns, but I gave it a try and created
`syncapi_current_room_state_type_state_key_idx` index. I could observe
significant performance improvement. Execution time dropped from 2.9 ms
to 0.24 ms:

```
explain analyze SELECT distinct room_id, membership FROM syncapi_current_room_state WHERE type = 'm.room.member' AND state_key = '@qjfl:dendrite.stg.globekeeper.com';
                                                                             QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Unique  (cost=96.46..96.64 rows=24 width=52) (actual time=0.199..0.218 rows=65 loops=1)
   ->  Sort  (cost=96.46..96.52 rows=24 width=52) (actual time=0.199..0.202 rows=65 loops=1)
         Sort Key: room_id, membership
         Sort Method: quicksort  Memory: 34kB
         ->  Bitmap Heap Scan on syncapi_current_room_state  (cost=4.53..95.91 rows=24 width=52) (actual time=0.048..0.139 rows=65 loops=1)
               Recheck Cond: ((type = 'm.room.member'::text) AND (state_key = '@qjfl:dendrite.stg.globekeeper.com'::text))
               Heap Blocks: exact=59
               ->  Bitmap Index Scan on syncapi_current_room_state_type_state_key_idx  (cost=0.00..4.53 rows=24 width=0) (actual time=0.037..0.037 rows=65 loops=1)
                     Index Cond: ((type = 'm.room.member'::text) AND (state_key = '@qjfl:dendrite.stg.globekeeper.com'::text))
 Planning Time: 0.236 ms
 Execution Time: 0.242 ms
(11 rows)
```

Next improvement is skipping DISTINCT and rely on map assignment in
`SelectRoomIDsWithAnyMembership`. Execution time drops by almost half:

```
explain analyze SELECT room_id, membership FROM syncapi_current_room_state WHERE type = 'm.room.member' AND state_key = '@qjfl:dendrite.stg.globekeeper.com';
                                                                       QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on syncapi_current_room_state  (cost=4.53..95.91 rows=24 width=52) (actual time=0.032..0.113 rows=65 loops=1)
   Recheck Cond: ((type = 'm.room.member'::text) AND (state_key = '@qjfl:dendrite.stg.globekeeper.com'::text))
   Heap Blocks: exact=59
   ->  Bitmap Index Scan on syncapi_current_room_state_type_state_key_idx  (cost=0.00..4.53 rows=24 width=0) (actual time=0.021..0.021 rows=65 loops=1)
         Index Cond: ((type = 'm.room.member'::text) AND (state_key = '@qjfl:dendrite.stg.globekeeper.com'::text))
 Planning Time: 0.087 ms
 Execution Time: 0.136 ms
(7 rows)
```

In our env we spend only 1s on inserting to table, so the write penalty
of creating an index should be small.
```
dendrite_syncapi=# select total_exec_time, left(query,100) from pg_stat_statements where query like '%INSERT%syncapi_current_room_state%' order by total_exec_time desc;
  total_exec_time   |                                                 left
--------------------+------------------------------------------------------------------------------------------------------
 1139.9057619999971 | INSERT INTO syncapi_current_room_state (room_id, event_id, type, sender, contains_url, state_key, he
(1 row)
``` 

This PR does not require test modifications.

### Pull Request Checklist

<!-- Please read docs/CONTRIBUTING.md before submitting your pull
request -->

* [x] I have added added tests for PR _or_ I have justified why this PR
doesn't need tests.
* [x] Pull request includes a [sign
off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off)

Signed-off-by: `Piotr Kozimor <p1996k@gmail.com>`
2022-09-27 09:41:36 +01:00
networkException
40fec70d13
Add pinecone demo container image (#2710)
This pull request adds the configuration and CI steps to build and
publish a container wrapping the `dendrite-demo-pinecone` command as
well as fixes a sentence structure issue in the pull request template.

As this does not touch any go source code no tests have been added

### Pull Request Checklist

<!-- Please read docs/CONTRIBUTING.md before submitting your pull
request -->

* [x] I have added tests for PR _or_ I have justified why this PR
doesn't need tests.
* [x] Pull request includes a [sign
off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off)

Signed-off-by: networkException <git@nwex.de> (by private sign-off)
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-09-27 09:39:39 +01:00
Neil Alexander
f022fc1397
Remove origin field from PDUs (#2737)
This nukes the `origin` field from PDUs as per
matrix-org/matrix-spec#998, matrix-org/gomatrixserverlib#341.
2022-09-26 17:35:35 +01:00
Neil Alexander
3e87096a21
Use TxStmt in SQLite pusher table 2022-09-26 09:54:54 +01:00
Till Faelligen
3c416517b0
Fix possible "Database is locked" issue 2022-09-26 10:45:35 +02:00
Neil Alexander
8d64c24b23
Update documentation to state that Dendrite requires PostgreSQL UTF-8 encoding 2022-09-26 09:33:34 +01:00
Neil Alexander
d5978d98fd
Update to matrix-org/pinecone@0900fceecb 2022-09-23 16:21:12 +01:00
Neil Alexander
14b7322003
Add -dir option to dendrite-demo-pinecone and dendrite-demo-yggdrasil 2022-09-23 15:44:21 +01:00
Neil Alexander
a50556dcf0
Tweak mainline ordering (update to matrix-org/gomatrixserverlib@2217f6c) 2022-09-23 12:59:00 +01:00
Till
c53f284fdb
Get the DeviceListPosition before anything else in complete syncs (#2733)
This should hopefully unflake `Can query remote device keys using POST`
in Complement.
2022-09-22 17:49:35 +02:00
Neil Alexander
f40e280327
Version 0.9.9 (#2732)
Changelog and version bump.
2022-09-22 14:54:25 +01:00
Till
0ddfb0cad4
Tweak InsertMigration to avoid logging (#2720)
We'd still produce logs in Postgres when trying to insert a migration we
already ran. This should stop us from creating those log entries.
2022-09-22 15:53:15 +02:00
Neil Alexander
852d856db8
Update embedded NATS Server to v2.9.1-beta1 (as this includes a fix that prevents high CPU usage after Dendrite startup) 2022-09-22 13:55:40 +01:00
dependabot[bot]
61a34d7cfb
Bump commonmarker from 0.23.4 to 0.23.6 in /docs (#2731)
Bumps [commonmarker](https://github.com/gjtorikian/commonmarker) from
0.23.4 to 0.23.6.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/gjtorikian/commonmarker/releases">commonmarker's
releases</a>.</em></p>
<blockquote>
<h2>v0.23.6</h2>
<h2>What's Changed</h2>
<p>This release includes two updates from the upstream
<code>cmark-gfm</code> library, namely:</p>
<ul>
<li><a href="https://github.com/github/cmark-gfm/releases">DoS
vulnerability in autolink extension</a> per <a
href="https://github.com/github/cmark-gfm/security/advisories/GHSA-cgh3-p57x-9q7q">GHSA-cgh3-p57x-9q7q</a></li>
<li><a
href="https://github.com/github/cmark-gfm/releases/tag/0.29.0.gfm.5">Added
<code>xmpp:</code> and <code>mailto:</code> support to the autolink
extension</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/gjtorikian/commonmarker/blob/main/CHANGELOG.md">commonmarker's
changelog</a>.</em></p>
<blockquote>
<h1>Changelog</h1>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="a8f8d76fbc"><code>a8f8d76</code></a>
Merge pull request <a
href="https://github-redirect.dependabot.com/gjtorikian/commonmarker/issues/190">#190</a>
from anticomputer/main</li>
<li><a
href="ac91634631"><code>ac91634</code></a>
💎 release 0.23.6</li>
<li><a
href="777fd3054b"><code>777fd30</code></a>
Update cmark-upstream to <a
href="https://github.com/github/cmark-gfm/commit/9d57d8a23">https://github.com/github/cmark-gfm/commit/9d57d8a23</a>...</li>
<li><a
href="7aaeb37e97"><code>7aaeb37</code></a>
Merge pull request <a
href="https://github-redirect.dependabot.com/gjtorikian/commonmarker/issues/188">#188</a>
from stevenlaidlaw/update-to-0290gfm5</li>
<li><a
href="795e628a40"><code>795e628</code></a>
Update cmark-upstream to <a
href="https://github.com/github/cmark-gfm/commit/0578e1e4f">https://github.com/github/cmark-gfm/commit/0578e1e4f</a>...</li>
<li><a
href="39d19d6530"><code>39d19d6</code></a>
Update cmark-upstream to <a
href="https://github.com/github/cmark-gfm/commit/766f161ef">https://github.com/github/cmark-gfm/commit/766f161ef</a>...</li>
<li><a
href="63b7bf89ee"><code>63b7bf8</code></a>
Update FUNDING.yml</li>
<li><a
href="558c7275b1"><code>558c727</code></a>
Bump to 0.23.5</li>
<li><a
href="41eee7265f"><code>41eee72</code></a>
lint</li>
<li><a
href="897e8ed07d"><code>897e8ed</code></a>
Merge pull request <a
href="https://github-redirect.dependabot.com/gjtorikian/commonmarker/issues/180">#180</a>
from lumaxis/main</li>
<li>Additional commits viewable in <a
href="https://github.com/gjtorikian/commonmarker/compare/v0.23.4...v0.23.6">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=commonmarker&package-manager=bundler&previous-version=0.23.4&new-version=0.23.6)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the
default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as
the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as
the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the
default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/matrix-org/dendrite/network/alerts).

</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-22 11:21:43 +01:00
Neil Alexander
d8b19c857f
HTTP connection keepalives (#2730)
Beforehand we disabled HTTP keepalives to prevent ambient system
resources from being used by excess idle connections. Now that we've
fixed some bugs in the federation API and device list updater, this
situation is now much better and we don't open so many remote
connections anyway.

Keepalives allow us to not have to handshake TLS so often (which is
quite expensive) and reusing an idle connection is much faster than
having to open a new one. This can help with response times when talking
to remote federated servers.

This PR also adds a new option to disable keepalives if needed:

```
  # Disable HTTP keepalives, which also prevents connection reuse. Dendrite will typically
  # keep HTTP connections open to remote hosts for 5 minutes as they can be reused much
  # more quickly than opening new connections each time. Disabling keepalives will close
  # HTTP connections immediately after a successful request but may result in more CPU and
  # memory being used on TLS handshakes for each new connection instead.
  disable_http_keepalives: false
```
2022-09-20 17:17:44 +01:00
Neil Alexander
bd39748b5c
Update dependencies (#2729)
This updates Dendrite dependencies.
2022-09-20 15:01:19 +01:00
Neil Alexander
201ac05943
Update readme 2022-09-20 14:10:30 +01:00
Neil Alexander
97d7cf2232
Remove deleted state logging lines from sync API (they are pointless) 2022-09-20 11:25:18 +01:00
Till
e007b8038f
Mark device list as stale, if we don't have the requesting device (#2728)
This hopefully makes E2EE chats a little bit more reliable by re-syncing
devices if we don't have the `requesting_device_id` in our database. (As
seen in
[Synapse](c52abc1cfd/synapse/handlers/devicemessage.py (L157-L201)))
2022-09-20 11:32:03 +02:00
Neil Alexander
47af4bff5b
Update database documentation 2022-09-20 09:52:22 +01:00
Neil Alexander
cf01d29277
Update contributing documentation 2022-09-20 09:47:57 +01:00
Tak Wai Wong
99f6b6a952
Bug fix #2718 appservice txnid should be different for each batch of events (#2719)
See issue: [#2718](https://github.com/matrix-org/dendrite/issues/2718)
for more details.

The fix assumes that if the number of transaction items are different,
then the txnid should be different.

txnid := OriginalServerTS()_len(transactions) 

The case that it doesn't address is if the txnid generated this way is
the same for 2 different batches of events which have the same
OriginalServerTS and the same array length.

Another option:

txnid := OriginalServerTS()_hash(transactions)

Would love to hear other ideas and ways to fix this.

### Pull Request Checklist

* [x ] I have added added tests for PR _or_ I have justified why this PR
doesn't need tests.
* [x ] Pull request includes a [sign
off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off)

Signed-off-by: `Tak Wai Wong <tak@hntlabs.com>`

Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
2022-09-19 18:39:06 +02:00
Till Faelligen
7bfc3074d1
Fix origin on device list update EDUs 2022-09-16 13:30:20 +02:00
Neil Alexander
fc1d8e479b
Ensure that all state event IDs are included in the added section when rewriting state (#2725)
This should hopefully fix an entire class of problems where components
downstream from the roomserver (i.e. the sync API) could just lose a
whole bunch of state after a rewrite operation like a federated join.

The root of the bug is that we set `RewritesState` in the output event
which instructs downstream components to purge their copy of any room
state, but then didn't send the entire state snapshot in
`adds_state_event_ids` so the downstream state ends up being incomplete
as a result.
2022-09-16 10:35:32 +01:00
Till
a5f8c07184
Hopefully fix upgrade-tests (#2717)
Wait for events to come down `/sync` before ending the test.
2022-09-15 07:26:26 +02:00
Neil Alexander
0ea948c705
Fix Pinecone demo build errors after Pinecone update 2022-09-14 14:26:24 +01:00
Neil Alexander
e6960d0b15
Update to matrix-org/pinecone@608215eb1b 2022-09-14 14:25:25 +01:00
Neil Alexander
7f89fed1e4
Revert 482914aef4 2022-09-14 09:55:50 +01:00
Neil Alexander
482914aef4
Use AckNone on the ephemeral room input consumer 2022-09-13 15:25:02 +01:00
Neil Alexander
b05e028f7d
Tweak LoadMembershipAtEvent behaviour when state not known (#2716)
Previously `LoadMembershipAtEvent` would fail if the state before one of
the events was not known, i.e. because it was an outlier. This modifies
it so that it gracefully handles not knowing the state and returns no
memberships instead, so that history visibility doesn't freak out and
kill `/sync` requests dead.
2022-09-13 12:52:09 +01:00
Neil Alexander
3e55856254
Always resolve state in QueryStateAfterEvents 2022-09-13 09:37:38 +01:00
Till
c366ccdfca
Send-to-device consumer/producer tweaks (#2713)
Some tweaks for the send-to-device consumers/producers:
- use `json.RawMessage` without marshalling it first
- try further devices (if available) if we failed to `PublishMsg` in the
producers
- some logging changes (to better debug E2EE issues)
2022-09-13 09:35:45 +02:00
Till
100fa9b235
Check unique constraint errors when manually inserting migrations (#2712)
This should avoid unnecessary logging on startup if the migration (were
we need `InsertMigration`) was already executed.
This now checks for "unique constraint errors" for SQLite and Postgres
and fails the startup process if the migration couldn't be manually
inserted for some other reason.
2022-09-13 08:07:43 +02:00
Neil Alexander
62afb936a5
Update to matrix-org/gomatrixserverlib@7b96db4 2022-09-12 15:27:45 +01:00
Neil Alexander
47b2a5d6b8
Version 0.9.8 2022-09-12 13:41:07 +01:00
Neil Alexander
2792d0490f
Fix missing signature check on the /get_missing_events response 2022-09-12 13:30:51 +01:00
Neil Alexander
7595fbf58c
Update to NATS Server v2.9.0 2022-09-12 13:24:03 +01:00
Neil Alexander
e9af30b3fe
Update to matrix-org/pinecone@b215925 2022-09-12 10:35:04 +01:00
Neil Alexander
3a9dde28fd
P2P demo tweaks 2022-09-12 10:19:02 +01:00
Neil Alexander
5997c32452
Update to matrix-org/gomatrixserverlib@dec87dbaa4 2022-09-11 15:39:11 +01:00