Commit Graph

2926 Commits

Author SHA1 Message Date
Alex Kirk
05f72fc4be
Update docs and sample config for the relay_api (#3011)
This adds an empty `relay_api` section to the sample configuration. For
SQLite environments, or others where a `database.connection_string` is
needed for each section, there should be an entry in the configuration
sample as a basis.

This PR also changes the "Configuring Dendrite" documentation in that
respect.

The requirement was introduced in #2917. When upgrading dendrite, it
will complain about `relay_api.database.connection_string` not being
configured.

### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [x] I have added Go unit tests or [Complement integration
tests](https://github.com/matrix-org/complement) for this PR _or_ I have
justified why this PR doesn't need tests
* [x] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)
_or_ I have already signed off privately

Signed-off-by: `Alex Kirk <akirk@users.noreply.github.com>`
2023-03-27 10:55:36 +02:00
Till
234ed603e6
Move every db.Prepare to sqlutil.Statementlist, remove trace driver (#3026)
Doesn't buy us much, but makes everything a bit more consistent.

Also removes the SQL trace driver, as it is unused and the output is
hard to read anyway.
2023-03-23 13:52:53 +01:00
Till Faelligen
cb18ba0230
Upload covdatafiles for each server 2023-03-22 17:36:33 +01:00
Till Faelligen
14085d30ac
Update workflow to not use commas when joining names 2023-03-22 16:01:12 +01:00
Till Faelligen
a4400bdd76
Sytest coverage file 2023-03-22 14:58:36 +01:00
Till Faelligen
b741d38e10
Update Workflow 2023-03-22 14:51:18 +01:00
Till Faelligen
6948d16527
Update Go, use go tool covdata for coverage files? 2023-03-22 14:50:21 +01:00
Till
5e85a00cb3
Remove BaseDendrite (#3023)
Removes `BaseDendrite` to, hopefully, make testing and composing of
components easier in the future.
2023-03-22 09:21:32 +01:00
Till Faelligen
ec6879e5ae
Update GMSL to fix #3013 2023-03-21 16:04:51 +01:00
Till Faelligen
0459d2b9e5
Make "m.upload.size" optional 2023-03-20 09:24:00 +01:00
Till
5579121c6f
Preparations for removing BaseDendrite (#3016)
Preparations to actually remove/replace `BaseDendrite`.
Quite a few changes:
- SyncAPI accepts an `fulltext.Indexer` interface (fulltext is removed
from `BaseDendrite`)
- Caches are removed from `BaseDendrite`
- Introduces a `Router` struct (likely to change)
  - also fixes #2903
- Introduces a `sqlutil.ConnectionManager`, which should remove
`base.DatabaseConnection` later on
- probably more
2023-03-17 11:09:45 +00:00
Boris Rybalkin
d88f71ab71
simplify unix socket permission format (#3014)
### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [x] I have added Go unit tests or [Complement integration
tests](https://github.com/matrix-org/complement) for this PR _or_ I have
justified why this PR doesn't need tests
* [x] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)
_or_ I have already signed off privately

Signed-off-by: `Boris Rybalkin <ribalkin@gmail.com>`
2023-03-16 08:51:21 +01:00
Till Faelligen
2c58bab6a8
Fix UTs on x86 2023-03-15 08:21:00 +01:00
Till
74dc54684b
Version 0.12.0 (#3009) 2023-03-13 20:02:57 +01:00
Till
232aef016c
Add basic runtime tracing (#2996)
This allows us in almost all places to use regions to further trace down
long running tasks.
Also removes an unused function.
2023-03-13 16:45:14 +01:00
Till
689b5ee72f
Change default stats reporting endpoint (#3007)
It's the same instance we report to, only using the subdomain.
2023-03-10 12:27:08 +01:00
Till
c7303cbf76
Update dependencies (#3006)
In preparation for a new release, let's also update a few dependencies.
2023-03-10 10:32:50 +01:00
Till
70322699ab
Unset RoomServerEvent, since we can't be sure that Set actually updates the cached entry (#3002)
This should deflake UTs and be more correct in terms of getting
`Events`.
`Events` tries to fetch the event from the cache first and may get an
unredacted event from it, while it should already be redacted.
2023-03-09 09:52:13 +01:00
Till Faelligen
baef523cb0
Fix invalid roomNID returned 2023-03-07 15:35:08 +01:00
Till Faelligen
11a3fcc6cb
RoomServerEvents are mutable, given they can be redacted 2023-03-06 17:58:08 +01:00
Till Faelligen
a684b850b9
Actually ignore the error if we were able to backfill events 2023-03-06 17:45:21 +01:00
Till
7d83f8b633
Add tests for UpdateRelations (#2999)
This also fixes an issue regarding updates to relations for invalid
events, which could result in us retrying said event over and over
again, if we fail to unmarshal the event to
`gomatrixserverlib.RelationContent`, this was discovered by
`@sleroq:virto.community`
2023-03-06 12:43:59 +01:00
Tim McCormack
7fc839f751
Update admin-promotion instructions; clarify ID for evacuation (#2997)
Table name has changed since instructions were written.

There's probably a better way to describe how to get the internal room
ID than I've attempted here, so feel free to adjust as needed. (It may
even be good to show an example of what an internal room ID looks like,
e.g. `!nc93825:example.com`)

### Pull Request Checklist

* [x] I have added Go unit tests or [Complement integration
tests](https://github.com/matrix-org/complement) for this PR _or_ I have
justified why this PR doesn't need tests
  * Doc-only change
* [x] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)

Signed-off-by: `Tim McCormack <cortex@brainonfire.net>`

---------

Co-authored-by: Till <2353100+S7evinK@users.noreply.github.com>
2023-03-05 18:42:38 +01:00
Till Faelligen
56b28b01db
Update the cache with the redacted event 2023-03-03 14:49:41 +01:00
Till
9bcd0a2105
Make redaction check easier to read (#2995)
We need to check the redaction PL in Dendrite, if we do it in GMSL, we
end up not sending the event to the output stream because it will be
rejected.

---------

Co-authored-by: kegsay <kegan@matrix.org>
2023-03-03 14:03:17 +01:00
Robin Westerik
7cde99a7a7
Updated instructions and references to monolith to their new names (#2994)
Currently, the documentation makes use of the old names for the binary
and configuration files. This updates the documentation so that users
can follow the guide without issues again.
These changes don't require any go unit tests because it does not modify
any golang code.

Signed-off-by: `Robin Westerik <gh@westerik.me>`
2023-03-03 10:20:53 +01:00
Boris Rybalkin
6b1c9eafa9
unix socket support (#2974)
### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [x] I have added Go unit tests or [Complement integration
tests](https://github.com/matrix-org/complement) for this PR _or_ I have
justified why this PR doesn't need tests
* [x] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)
_or_ I have already signed off privately

Signed-off-by: `Boris Rybalkin <ribalkin@gmail.com>`

I need this for Syncloud project (https://github.com/syncloud/platform)
where I run multiple apps behind an nginx on the same RPi like device so
unix socket is very convenient to not have port conflicts between apps.
Also someone opened this Issue:
https://github.com/matrix-org/dendrite/issues/2924

---------

Co-authored-by: kegsay <kegan@matrix.org>
Co-authored-by: Till <2353100+S7evinK@users.noreply.github.com>
2023-03-01 22:57:30 +01:00
Till
6c20f8f742
Refactor StoreEvent, add MaybeRedactEvent, create an EventDatabase (#2989)
This PR changes the following:
- `StoreEvent` now only stores an event (and possibly prev event),
instead of also doing redactions
- Adds a `MaybeRedactEvent` (pulled out from `StoreEvent`), which should
be called after storing events
- a few other things
2023-03-01 17:06:47 +01:00
Till Faelligen
1aa70b0f56
Fix UTs 2023-03-01 15:09:10 +01:00
Till Faelligen
f1ccfcf150
Only run CI if there are changes to go files or the workflow [skip ci] 2023-02-28 15:35:53 +01:00
Till Faelligen
086e205eba
Deploy on gh-pages push 2023-02-28 15:15:19 +01:00
genofire
7fff7cd2ac
feat(helm): add prometheus resources to monitor (#2958)
### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [x] I have added Go unit tests or [Complement integration
tests](https://github.com/matrix-org/complement) for this PR _or_ I have
justified why this PR doesn't need tests
* [x] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)
_or_ I have already signed off privately

---

I do not know, how you run helm-docs ....


otherwise i would like to add somewhere:
````markdown
* Works well with [Prometheus Operator](https://prometheus-operator.dev/) ([Helmchart](https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack)) and there setup of [Grafana](https://grafana.com/grafana/), by enabling following values:
```yaml
prometheus:
  servicemonitor:
    enabled: true
    labels:
      release: "kube-prometheus-stack"
  rules:
    enabled: true # will deploy alert rules
    additionalLabels:
      release: "kube-prometheus-stack"
grafana:
  dashboards:
    enabled: true # will deploy default dashboards 
```
PS: The labels `release=kube-prometheus-stack` is setup with the helmchart of the Prometheus Operator. For Grafana Dashboards it maybe need scan enable to scan in correct namespaces (or ALL), enabled by `sidecar.dashboards.searchNamespace` in [Helmchart of grafana](https://artifacthub.io/packages/helm/grafana/grafana) (which is part of PrometheusOperator, so `grafana.sidecar.dashboards.searchNamespace`) 
````

Maybe also put somewhere the Screenshot of that Grafana Dashboard:
https://grafana.com/grafana/dashboards/13916-dendrite/

---


@S7evinK do you take a look?

Signed-off-by: genofire <geno+dev@fireorbit.de>
2023-02-28 14:18:26 +01:00
Devon Hudson
eddf31f915
Fix lint error 2023-02-24 15:49:51 -07:00
Devon Hudson
b28406c7d0
Tweaks to pinecone demo to shutdown more cleanly 2023-02-24 15:41:47 -07:00
Till Faelligen
3d31b131fc
Cache all the things 2023-02-24 11:45:01 +01:00
Till
ad07b169b8
Refactor StoreEvent and create a new RoomDatabase interface (#2985)
This PR changes a few things:
- It pulls out the creation of several NIDs from the `StoreEvent`
function to make the functions more reusable
- Uses more caching when using those NIDs to avoid DB round trips
2023-02-24 09:40:20 +01:00
David Schneider
e6aa0955ff
Unify logging by using logrus for jetstream logs (#2976)
I guess tests for the logging is rather unusual so I omitted tests for
this change.

* [x] I have added Go unit tests or [Complement integration
tests](https://github.com/matrix-org/complement) for this PR _or_ I have
justified why this PR doesn't need tests
* [x] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)
_or_ I have already signed off privately

Signed-off-by: `David Schneider <dsbrng25b@gmail.com>`

---------

Signed-off-by: David Schneider <dsbrng25b@gmail.com>
2023-02-24 08:56:53 +01:00
Yoann N
d34277a6c0
Update README sample config link (#2987)
### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [x] I have added Go unit tests or [Complement integration
tests](https://github.com/matrix-org/complement) for this PR _or_ I have
justified why this PR doesn't need tests
* [x] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)
_or_ I have already signed off privately
2023-02-23 09:07:20 +01:00
Till Faelligen
c8ca23acdb
Fix building Element web in CI 2023-02-20 15:28:45 +01:00
Till
7f114cc538
Fix issue where device keys are removed if a device ID is reused (#2982)
Fixes https://github.com/matrix-org/dendrite/issues/2980
2023-02-20 15:26:09 +01:00
Till
4594233f89
Merge keyserver & userapi (#2972)
As discussed yesterday, a first draft of merging the keyserver and the
userapi.
2023-02-20 14:58:03 +01:00
dependabot[bot]
bd6f0c14e5
Bump golang.org/x/net from 0.5.0 to 0.7.0 (#2979)
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.5.0 to
0.7.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="8e2b117aee"><code>8e2b117</code></a>
http2/hpack: avoid quadratic complexity in hpack decoding</li>
<li><a
href="547e7edf38"><code>547e7ed</code></a>
http2: avoid referencing ResponseWrite.Write parameter after
returning</li>
<li><a
href="39940adcaa"><code>39940ad</code></a>
html: parse comments per HTML spec</li>
<li><a
href="87ce33ecb4"><code>87ce33e</code></a>
go.mod: update golang.org/x dependencies</li>
<li><a
href="415cb6d518"><code>415cb6d</code></a>
all: fix some comments</li>
<li><a
href="7e3c19ca52"><code>7e3c19c</code></a>
all: correct typos in comments</li>
<li><a
href="296f09aa38"><code>296f09a</code></a>
http2: case insensitive handling for 100-continue</li>
<li><a
href="f8411da775"><code>f8411da</code></a>
nettest: fix tests on dragonfly and js/wasm</li>
<li>See full diff in <a
href="https://github.com/golang/net/compare/v0.5.0...v0.7.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=golang.org/x/net&package-manager=go_modules&previous-version=0.5.0&new-version=0.7.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the
default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as
the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as
the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the
default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/matrix-org/dendrite/network/alerts).

</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-17 19:33:40 +01:00
dependabot[bot]
22c4736495
Bump golang.org/x/image from 0.1.0 to 0.5.0 (#2978)
Bumps [golang.org/x/image](https://github.com/golang/image) from 0.1.0
to 0.5.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="e6c2a4cdd5"><code>e6c2a4c</code></a>
tiff: don't pre-allocate giant slices before reading</li>
<li><a
href="3db422c472"><code>3db422c</code></a>
go.mod: update golang.org/x dependencies</li>
<li><a
href="bb712eb67b"><code>bb712eb</code></a>
go.mod: update golang.org/x dependencies</li>
<li><a
href="9b8a3be425"><code>9b8a3be</code></a>
font/plan9font: limit the font image's dimensions</li>
<li><a
href="c5235ae222"><code>c5235ae</code></a>
font/plan9font: fix byteoffset for non-zero origin</li>
<li><a
href="0888fdd524"><code>0888fdd</code></a>
font/plan9font: fix bounds overflow</li>
<li><a
href="9fdfde75ec"><code>9fdfde7</code></a>
go.mod: update golang.org/x dependencies</li>
<li>See full diff in <a
href="https://github.com/golang/image/compare/v0.1.0...v0.5.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=golang.org/x/image&package-manager=go_modules&previous-version=0.1.0&new-version=0.5.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the
default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as
the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as
the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the
default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/matrix-org/dendrite/network/alerts).

</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-02-17 16:33:49 +01:00
Till
f0805071d5
Fix SQLite session_id (#2977)
This fixes an issue with device_id/session_ids.
If a `device_id` is reused, we would reuse the same `session_id`, since
we delete one device and insert a new one directly, resulting in the
query to get a new `session_id` to return the previous session_id.
(`SELECT count(access_token)`)
2023-02-17 11:39:46 +01:00
Till
11d9b9db0e
Remove polylith/API mode (#2967)
This removes most of the code used for polylith/API mode.

This removes the `/api` internal endpoints entirely. 

Binary size change roughly 5%: 
```
51437560 Feb 13 10:15 dendrite-monolith-server # old
48759008 Feb 13 10:15 dendrite-monolith-server # new
```
2023-02-14 12:47:47 +01:00
Till
cc59879faa
Version 0.11.1 (#2966) 2023-02-10 18:36:59 +01:00
ShalokShalom
e64ed0934d
Update coverage (#2964)
### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [x] I have added Go unit tests or [Complement integration
tests](https://github.com/matrix-org/complement) for this PR _or_ I have
justified why this PR doesn't need tests
* [x] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)
_or_ I have already signed off privately

Signed-off-by: `Matthias Gramberg <ShalokShalom@riseup.net>`
2023-02-08 08:56:16 +01:00
Till
eb29a31550
Optimize /sync and history visibility (#2961)
Should fix the following issues or make a lot less worse when using
Postgres:

The main issue behind #2911: The client gives up after a certain time,
causing a cascade of context errors, because the response couldn't be
built up fast enough. This mostly happens on accounts with many rooms,
due to the inefficient way we're getting recent events and current state

For #2777: The queries for getting the membership events for history
visibility were being executed for each room (I think 185?), resulting
in a whooping 2k queries for membership events. (Getting the
statesnapshot -> block nids -> actual wanted membership event)

Both should now be better by:
- Using a LATERAL join to get all recent events for all joined rooms in
one go (TODO: maybe do the same for room summary and current state etc)
- If we're lazy loading on initial syncs, we're now not getting the
whole current state, just to drop the majority of it because we're lazy
loading members - we add a filter to exclude membership events on the
first call to `CurrentState`.
- Using an optimized query to get the membership events needed to
calculate history visibility

---------

Co-authored-by: kegsay <kegan@matrix.org>
2023-02-07 14:31:23 +01:00
Devon Hudson
cf254ba044
Add max frame size to pinecone bindings 2023-02-04 06:05:39 -07:00
Devon Hudson
4ed61740ab
Disable fulltext search in pinecone builds 2023-02-04 05:56:33 -07:00