Commit Graph

2582 Commits

Author SHA1 Message Date
Till Faelligen
3c416517b0
Fix possible "Database is locked" issue 2022-09-26 10:45:35 +02:00
Neil Alexander
8d64c24b23
Update documentation to state that Dendrite requires PostgreSQL UTF-8 encoding 2022-09-26 09:33:34 +01:00
Neil Alexander
d5978d98fd
Update to matrix-org/pinecone@0900fceecb 2022-09-23 16:21:12 +01:00
Neil Alexander
14b7322003
Add -dir option to dendrite-demo-pinecone and dendrite-demo-yggdrasil 2022-09-23 15:44:21 +01:00
Neil Alexander
a50556dcf0
Tweak mainline ordering (update to matrix-org/gomatrixserverlib@2217f6c) 2022-09-23 12:59:00 +01:00
Till
c53f284fdb
Get the DeviceListPosition before anything else in complete syncs (#2733)
This should hopefully unflake `Can query remote device keys using POST`
in Complement.
2022-09-22 17:49:35 +02:00
Neil Alexander
f40e280327
Version 0.9.9 (#2732)
Changelog and version bump.
2022-09-22 14:54:25 +01:00
Till
0ddfb0cad4
Tweak InsertMigration to avoid logging (#2720)
We'd still produce logs in Postgres when trying to insert a migration we
already ran. This should stop us from creating those log entries.
2022-09-22 15:53:15 +02:00
Neil Alexander
852d856db8
Update embedded NATS Server to v2.9.1-beta1 (as this includes a fix that prevents high CPU usage after Dendrite startup) 2022-09-22 13:55:40 +01:00
dependabot[bot]
61a34d7cfb
Bump commonmarker from 0.23.4 to 0.23.6 in /docs (#2731)
Bumps [commonmarker](https://github.com/gjtorikian/commonmarker) from
0.23.4 to 0.23.6.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/gjtorikian/commonmarker/releases">commonmarker's
releases</a>.</em></p>
<blockquote>
<h2>v0.23.6</h2>
<h2>What's Changed</h2>
<p>This release includes two updates from the upstream
<code>cmark-gfm</code> library, namely:</p>
<ul>
<li><a href="https://github.com/github/cmark-gfm/releases">DoS
vulnerability in autolink extension</a> per <a
href="https://github.com/github/cmark-gfm/security/advisories/GHSA-cgh3-p57x-9q7q">GHSA-cgh3-p57x-9q7q</a></li>
<li><a
href="https://github.com/github/cmark-gfm/releases/tag/0.29.0.gfm.5">Added
<code>xmpp:</code> and <code>mailto:</code> support to the autolink
extension</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/gjtorikian/commonmarker/blob/main/CHANGELOG.md">commonmarker's
changelog</a>.</em></p>
<blockquote>
<h1>Changelog</h1>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="a8f8d76fbc"><code>a8f8d76</code></a>
Merge pull request <a
href="https://github-redirect.dependabot.com/gjtorikian/commonmarker/issues/190">#190</a>
from anticomputer/main</li>
<li><a
href="ac91634631"><code>ac91634</code></a>
💎 release 0.23.6</li>
<li><a
href="777fd3054b"><code>777fd30</code></a>
Update cmark-upstream to <a
href="https://github.com/github/cmark-gfm/commit/9d57d8a23">https://github.com/github/cmark-gfm/commit/9d57d8a23</a>...</li>
<li><a
href="7aaeb37e97"><code>7aaeb37</code></a>
Merge pull request <a
href="https://github-redirect.dependabot.com/gjtorikian/commonmarker/issues/188">#188</a>
from stevenlaidlaw/update-to-0290gfm5</li>
<li><a
href="795e628a40"><code>795e628</code></a>
Update cmark-upstream to <a
href="https://github.com/github/cmark-gfm/commit/0578e1e4f">https://github.com/github/cmark-gfm/commit/0578e1e4f</a>...</li>
<li><a
href="39d19d6530"><code>39d19d6</code></a>
Update cmark-upstream to <a
href="https://github.com/github/cmark-gfm/commit/766f161ef">https://github.com/github/cmark-gfm/commit/766f161ef</a>...</li>
<li><a
href="63b7bf89ee"><code>63b7bf8</code></a>
Update FUNDING.yml</li>
<li><a
href="558c7275b1"><code>558c727</code></a>
Bump to 0.23.5</li>
<li><a
href="41eee7265f"><code>41eee72</code></a>
lint</li>
<li><a
href="897e8ed07d"><code>897e8ed</code></a>
Merge pull request <a
href="https://github-redirect.dependabot.com/gjtorikian/commonmarker/issues/180">#180</a>
from lumaxis/main</li>
<li>Additional commits viewable in <a
href="https://github.com/gjtorikian/commonmarker/compare/v0.23.4...v0.23.6">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=commonmarker&package-manager=bundler&previous-version=0.23.4&new-version=0.23.6)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the
default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as
the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as
the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the
default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/matrix-org/dendrite/network/alerts).

</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-22 11:21:43 +01:00
Neil Alexander
d8b19c857f
HTTP connection keepalives (#2730)
Beforehand we disabled HTTP keepalives to prevent ambient system
resources from being used by excess idle connections. Now that we've
fixed some bugs in the federation API and device list updater, this
situation is now much better and we don't open so many remote
connections anyway.

Keepalives allow us to not have to handshake TLS so often (which is
quite expensive) and reusing an idle connection is much faster than
having to open a new one. This can help with response times when talking
to remote federated servers.

This PR also adds a new option to disable keepalives if needed:

```
  # Disable HTTP keepalives, which also prevents connection reuse. Dendrite will typically
  # keep HTTP connections open to remote hosts for 5 minutes as they can be reused much
  # more quickly than opening new connections each time. Disabling keepalives will close
  # HTTP connections immediately after a successful request but may result in more CPU and
  # memory being used on TLS handshakes for each new connection instead.
  disable_http_keepalives: false
```
2022-09-20 17:17:44 +01:00
Neil Alexander
bd39748b5c
Update dependencies (#2729)
This updates Dendrite dependencies.
2022-09-20 15:01:19 +01:00
Neil Alexander
201ac05943
Update readme 2022-09-20 14:10:30 +01:00
Neil Alexander
97d7cf2232
Remove deleted state logging lines from sync API (they are pointless) 2022-09-20 11:25:18 +01:00
Till
e007b8038f
Mark device list as stale, if we don't have the requesting device (#2728)
This hopefully makes E2EE chats a little bit more reliable by re-syncing
devices if we don't have the `requesting_device_id` in our database. (As
seen in
[Synapse](c52abc1cfd/synapse/handlers/devicemessage.py (L157-L201)))
2022-09-20 11:32:03 +02:00
Neil Alexander
47af4bff5b
Update database documentation 2022-09-20 09:52:22 +01:00
Neil Alexander
cf01d29277
Update contributing documentation 2022-09-20 09:47:57 +01:00
Tak Wai Wong
99f6b6a952
Bug fix #2718 appservice txnid should be different for each batch of events (#2719)
See issue: [#2718](https://github.com/matrix-org/dendrite/issues/2718)
for more details.

The fix assumes that if the number of transaction items are different,
then the txnid should be different.

txnid := OriginalServerTS()_len(transactions) 

The case that it doesn't address is if the txnid generated this way is
the same for 2 different batches of events which have the same
OriginalServerTS and the same array length.

Another option:

txnid := OriginalServerTS()_hash(transactions)

Would love to hear other ideas and ways to fix this.

### Pull Request Checklist

* [x ] I have added added tests for PR _or_ I have justified why this PR
doesn't need tests.
* [x ] Pull request includes a [sign
off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off)

Signed-off-by: `Tak Wai Wong <tak@hntlabs.com>`

Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
2022-09-19 18:39:06 +02:00
Till Faelligen
7bfc3074d1
Fix origin on device list update EDUs 2022-09-16 13:30:20 +02:00
Neil Alexander
fc1d8e479b
Ensure that all state event IDs are included in the added section when rewriting state (#2725)
This should hopefully fix an entire class of problems where components
downstream from the roomserver (i.e. the sync API) could just lose a
whole bunch of state after a rewrite operation like a federated join.

The root of the bug is that we set `RewritesState` in the output event
which instructs downstream components to purge their copy of any room
state, but then didn't send the entire state snapshot in
`adds_state_event_ids` so the downstream state ends up being incomplete
as a result.
2022-09-16 10:35:32 +01:00
Till
a5f8c07184
Hopefully fix upgrade-tests (#2717)
Wait for events to come down `/sync` before ending the test.
2022-09-15 07:26:26 +02:00
Neil Alexander
0ea948c705
Fix Pinecone demo build errors after Pinecone update 2022-09-14 14:26:24 +01:00
Neil Alexander
e6960d0b15
Update to matrix-org/pinecone@608215eb1b 2022-09-14 14:25:25 +01:00
Neil Alexander
7f89fed1e4
Revert 482914aef4 2022-09-14 09:55:50 +01:00
Neil Alexander
482914aef4
Use AckNone on the ephemeral room input consumer 2022-09-13 15:25:02 +01:00
Neil Alexander
b05e028f7d
Tweak LoadMembershipAtEvent behaviour when state not known (#2716)
Previously `LoadMembershipAtEvent` would fail if the state before one of
the events was not known, i.e. because it was an outlier. This modifies
it so that it gracefully handles not knowing the state and returns no
memberships instead, so that history visibility doesn't freak out and
kill `/sync` requests dead.
2022-09-13 12:52:09 +01:00
Neil Alexander
3e55856254
Always resolve state in QueryStateAfterEvents 2022-09-13 09:37:38 +01:00
Till
c366ccdfca
Send-to-device consumer/producer tweaks (#2713)
Some tweaks for the send-to-device consumers/producers:
- use `json.RawMessage` without marshalling it first
- try further devices (if available) if we failed to `PublishMsg` in the
producers
- some logging changes (to better debug E2EE issues)
2022-09-13 09:35:45 +02:00
Till
100fa9b235
Check unique constraint errors when manually inserting migrations (#2712)
This should avoid unnecessary logging on startup if the migration (were
we need `InsertMigration`) was already executed.
This now checks for "unique constraint errors" for SQLite and Postgres
and fails the startup process if the migration couldn't be manually
inserted for some other reason.
2022-09-13 08:07:43 +02:00
Neil Alexander
62afb936a5
Update to matrix-org/gomatrixserverlib@7b96db4 2022-09-12 15:27:45 +01:00
Neil Alexander
47b2a5d6b8
Version 0.9.8 2022-09-12 13:41:07 +01:00
Neil Alexander
2792d0490f
Fix missing signature check on the /get_missing_events response 2022-09-12 13:30:51 +01:00
Neil Alexander
7595fbf58c
Update to NATS Server v2.9.0 2022-09-12 13:24:03 +01:00
Neil Alexander
e9af30b3fe
Update to matrix-org/pinecone@b215925 2022-09-12 10:35:04 +01:00
Neil Alexander
3a9dde28fd
P2P demo tweaks 2022-09-12 10:19:02 +01:00
Neil Alexander
5997c32452
Update to matrix-org/gomatrixserverlib@dec87dbaa4 2022-09-11 15:39:11 +01:00
Till Faelligen
af9a204cc0 Only verify if fulltext indexing is enabled 2022-09-09 17:19:29 +02:00
Neil Alexander
e8687f6f82
Version 0.9.7 (#2707)
Changelog and version bump.
2022-09-09 15:01:13 +01:00
Neil Alexander
955e69a3b7
Optimise SharedUsers again by using complete composite index 2022-09-09 14:18:45 +01:00
Neil Alexander
6ee758df63
Optimise shared users query in Synx API slightly by removing a potential sort 2022-09-09 13:50:50 +01:00
Neil Alexander
e1bc4f6a1e
Fix database transaction for keyserver DeleteDeviceKeys 2022-09-09 13:31:55 +01:00
Neil Alexander
c0e17bbe1b
Fix transactions around assigning NIDs 2022-09-09 13:30:09 +01:00
Till
8196b29657
Change detection of already executed migrations (#2665)
This changes the detection of already executed migrations for the
roomserver state block and keychange refactor. It now uses schema tables
provided by the database engine to check if the column was already
removed. We now also store the migration in the migrations table.

This should stop e.g. Postgres from logging errors like `ERROR: column
"event_nid" does not exist at character 8`.
2022-09-09 13:14:52 +01:00
Neil Alexander
646de03d60
More writer fixes in the Sync API 2022-09-09 13:06:42 +01:00
Neil Alexander
34e1dc210b
Use database writer in user API DeleteOldNotifications 2022-09-09 12:59:28 +01:00
Till
64472d9aab
Update getting pushrules, add tests, tweak pushrules (#2705)
This PR
- adds tests for `evaluatePushrules`
- removes the need for the UserAPI on the `OutputStreamEventConsumer`
(for easier testing)
- adds a method to get the pushrules from the database
- adds a new default pushrule for `m.reaction` events (and some other
tweaks)
2022-09-09 13:56:33 +02:00
Till
42a82091a8
Fix issue with stale device lists (#2702)
We were only sending the last entry to the worker, so most likely missed
updates.
2022-09-08 12:03:44 +02:00
Till
d5876abbe9
Fulltext implementation incl. config (#2480)
This adds the main component of the fulltext search.
This PR doesn't do anything yet, besides creating an empty fulltextindex
folder if enabled. Indexing events is done in a separate PR.
2022-09-07 18:15:54 +02:00
Neil Alexander
31f4ae8997
Use a single context instead 2022-09-07 16:24:43 +01:00
Neil Alexander
5014b35bd7
Update state reset capture to Sentry 2022-09-07 16:23:22 +01:00