Commit Graph

2638 Commits

Author SHA1 Message Date
Till
0f777d421c
Remove empty fields from /sync response (#2755)
First attempt at removing empty fields from `/sync` responses. Needs
https://github.com/matrix-org/sytest/pull/1298 to keep Sytest happy.

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-10-05 13:47:13 +01:00
Neil Alexander
c85bc3434f
Optimise QuerySharedUsers so that we can only work on local users (#2766)
Otherwise the sync API key change consumer wastes a lot of time trying
to wake up the notifiers for non-local users.
2022-10-05 12:47:53 +01:00
Neil Alexander
6f602bb096
Demote Failed to query device keys for some users warning to level=debug
Many of these warnings are due to dead servers and are quite annoying when they fill up the logs.
2022-10-05 11:16:05 +01:00
Till
8c0c3441d8
Add RoomEventType nats.Header to avoid unneeded unmarshalling (#2765) 2022-10-05 12:12:42 +02:00
Neil Alexander
ebd137cf6b
Check PostgreSQL connection count (#2760)
This PR queries PostgreSQL for the `max_connections` and
`superuser_reserved_connections` settings and then ensures that
Dendrite's `max_open_conns` doesn't exceed the allowed value.

This is a really common source of configuration problems and can either
result in blocking queries or deadlocks, so it seems reasonable that we
complain as loudly as possible when it happens.
2022-10-05 11:07:17 +01:00
Till
e070352293
Side effect import bleve analyzer languages (#2763)
... to actually allow different languages. Fixes #2761 
Binary size increases by ~1MB.
2022-10-05 11:14:33 +02:00
Neil Alexander
21f8881985
Add indexes that optimise selectStateInRangeSQL (#2764)
This gets rid of some expensive scans on `add_state_ids` and
`remove_state_ids`, turning them into much cheaper and faster index
scans instead.
2022-10-04 16:43:10 +01:00
Neil Alexander
ae10aac456
Don't perform a federated join after invite if we are already joined to the room (#2762)
If we are already joined to the room then it shouldn't matter if you
were invited or not, so this looks like a bug.
2022-10-04 15:40:04 +01:00
Neil Alexander
3da182212e
Track reasons why the process is in a degraded state 2022-10-04 13:02:41 +01:00
Neil Alexander
a767102f8a
Reduce max_open_conns in monolith sample config 2022-10-04 11:34:52 +01:00
Neil Alexander
085bf5e28b
Revert Docker changes 2022-10-04 11:33:05 +01:00
Neil Alexander
98b73652e0
Try to populate -ldflags in Docker builds 2022-10-04 11:07:54 +01:00
Neil Alexander
ede4632835
Fix Docker GHA 2022-10-04 10:43:58 +01:00
Till
e6c992ba8b
Update Dockerfile (#2342)
Updates/adds a new multistage (build-kit) Dockerfile. (if accepted,
could make `Dockerfile.monolith` and `Dockerfile.polylith` in
`build/docker` obsolete)
There's no huge difference between the dockerfiles, except this uses a
non-root user when running the container, also doesn't copy the working
directory to the image when building.
Also adds vulnerabilities scans using
[Trivy](https://github.com/aquasecurity/trivy) for the created docker
images. (untested)

Building images is done using 
```
docker build . --target image-monolith -t dendrite-monolith
docker build . --target image-polylith -t dendrite-polylith
```

As noted in the comments, only adds `dendrite-polylith-multi` to the
polylith image and all required binaries to the monolith image.
Probably needs some docs updating, if this is accepted.

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-10-04 10:41:06 +01:00
Neil Alexander
34ed316584
Fix docs 2022-10-03 19:35:26 +01:00
Neil Alexander
ba66b5a3b9
Allow multiple static peers in Pinecone iOS/Android demos 2022-10-03 14:43:38 +01:00
Neil Alexander
34451d21b8
P2P demo tweaks 2022-10-03 14:35:10 +01:00
Neil Alexander
50fa50a343
Update P2P base directories 2022-10-03 13:05:58 +01:00
Neil Alexander
fec3ee384b
Stop CPU burn in PerformMarkAsStaleIfNeeded 2022-10-03 12:59:56 +01:00
Ashley Nelson
c1e16fd41e
Fix fragility of selectEventsWithEventIDsSQL queries (#2757)
This fixes a temporary workaround with the `selectEventsWithEventIDsSQL`
queries where fields need to be artificially added to the queries so the
row results match the format of the `syncapi_output_room_events` table.
I made similar functions that accept row results from the
`syncapi_current_room_state` table and convert them into StreamEvents
without the fields that are specific to output room events.

There is also a unit test in the first commit to ensure the resulting
behavior doesn't change from the modified queries and functions.

Fixes #601.

### Pull Request Checklist

<!-- Please read docs/CONTRIBUTING.md before submitting your pull
request -->

* [x] I have added tests for PR _or_ I have justified why this PR
doesn't need tests.
* [x] Pull request includes a [sign
off](https://github.com/matrix-org/dendrite/blob/main/docs/CONTRIBUTING.md#sign-off)

Signed-off-by: `Ashley Nelson <fant@shley.email>`

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-10-03 11:57:21 +01:00
Neil Alexander
b7f4bab358
Hopefully fix P2P --config error (re. #2756) 2022-10-03 11:38:31 +01:00
Neil Alexander
d32f60249d
Modify sync transaction behaviour (#2758)
This now uses a transaction per stream, so that errors in one stream
don't propagate to another, and we therefore no longer need to do hacks
to reopen a new transaction after aborting a failed one.
2022-10-03 11:38:20 +01:00
Till Faelligen
d4710217f8
Use non-HTTPS as default URL, as most people will be running behind a
reverse proxy
2022-10-02 11:31:40 +02:00
Neil Alexander
a050503d8d
Version 0.10.1 2022-09-30 17:15:43 +01:00
Neil Alexander
0116db79c6
Reset transaction after a failure 2022-09-30 17:07:37 +01:00
Neil Alexander
16048be236
Handle case when applying history visibility failed 2022-09-30 16:55:10 +01:00
Neil Alexander
7d9545ceea
Another /sync fix 2022-09-30 16:34:06 +01:00
Till Faelligen
3617d5a0ff
Remove SendAccountData, since InputAccountData is already doing that 2022-09-30 17:31:41 +02:00
Neil Alexander
60ec9180e6
Update NATS Server to v2.9.2 2022-09-30 16:10:44 +01:00
Neil Alexander
ee40a29e55
Fix broken /sync due to transaction error 2022-09-30 16:07:18 +01:00
Neil Alexander
aa8ec1acbf
Update 2_domainname.md 2022-09-30 15:39:05 +01:00
Neil Alexander
0e2fb63b4f
Version 0.10.0 (#2753)
Changelog and version bump.
2022-09-30 13:31:05 +01:00
Neil Alexander
6348486a13
Transactional isolation for /sync (#2745)
This should transactional snapshot isolation for `/sync` etc requests.

For now we don't use repeatable read due to some odd test failures with
invites.
2022-09-30 12:48:10 +01:00
Neil Alexander
8a82f10046
Allow more time for device list updates (#2749)
This updates the device list updater so that it has a context
per-request, rather than a global 30 seconds for the entire server. This
could mean that talking to a slow remote server or requesting a lot of
user IDs was pretty much guaranteed to fail.

It also uses the process context to allow correct cancellation when
Dendrite wants to shut down cleanly.
2022-09-30 09:41:16 +01:00
Till
9005e5b4a8
Add /_dendrite/admin/refreshDevices/{userID} (#2746)
Allows to immediately query `/devices/{userID}` over federation to
(hopefully) resolve E2EE issues.
2022-09-30 09:32:31 +01:00
Neil Alexander
e45ba35e97
Enable knocking on room versions 8 and 9 (update to matrix-org/gomatrixserverlib@91d455c) 2022-09-29 20:05:05 +01:00
devonh
90f1985bf3
Add network interface registration to pinecone demo (#2750)
### Pull Request Checklist

<!-- Please read docs/CONTRIBUTING.md before submitting your pull
request -->

No additional tests have been added as this simply extends the API for
the embedded pinecone demo.

Signed-off-by: `Devon Hudson <devonhudson@librem.one>`
2022-09-29 16:05:16 +00:00
Neil Alexander
b28bbadeb0
Update to matrix-org/gomatrixserverlib@377b320 2022-09-29 16:55:03 +01:00
Neil Alexander
1b65c97ad1
Remove Suppressing send-to-device log line
The behaviour is correct and I have no idea why we are logging it.
2022-09-29 16:18:42 +01:00
Neil Alexander
e1bf709eb3
Improve device list update parsing (update to matrix-org/gomatrixserverlib#342) 2022-09-29 16:08:41 +01:00
Neil Alexander
6f3dbee5f4
Fix Go 1.18 2022-09-29 16:08:00 +01:00
Neil Alexander
68d6eb0a6f
P2P demo fixes 2022-09-29 13:06:55 +01:00
Neil Alexander
3f9e38e80a
Consistent *sql.Tx usage across sync API (#2744)
This tidies up the `storage` package so that everything takes a
transaction parameter instead of something things that do and some that
don't.
2022-09-28 10:18:03 +01:00
texuf
a574ed5369
Fix for `sql: converting argument $1 type: unsupported type []interfa… (#2743)
…ce {}, a slice of interface` in new notifications select

The sqlite3 version was just not working, original pr here:
https://github.com/matrix-org/dendrite/pull/2688

signed off by: austin ellis <austin@hntlabs.com>

This doesn't fix the notification counts, they still only work about 1
out of every 5 times in my tests. I will stick with my other fix locally
for reliable notification delivery:
https://github.com/matrix-org/dendrite/pull/2701
2022-09-28 06:19:34 +02:00
Neil Alexander
083ae01520
Promote reindexing log level 2022-09-27 17:30:40 +01:00
Neil Alexander
34993717fd
Update search docs 2022-09-27 17:10:47 +01:00
Till
87be32ca26
Fulltext implementation using Bleve (#2675)
Based on #2480

This actually indexes events based on their event type. They are removed
from the index if we receive a `m.room.redaction` event on the
`OutputRoomEvent` stream.
An admin endpoint is added to reindex all existing events.


Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-09-27 18:06:49 +02:00
Neil Alexander
6c67552bf9
Return M_UNRECOGNIZED for unknown CS API endpoints/actions (#2740)
Fixes #2739.
2022-09-27 15:50:22 +01:00
Till
249b32c4f3
Refactor notifications (#2688)
This PR changes the handling of notifications
- removes the `StreamEvent` and `ReadUpdate` stream
- listens on the `OutputRoomEvent` stream in the UserAPI to inform the
SyncAPI about unread notifications
- listens on the `OutputReceiptEvent` stream in the UserAPI to set
receipts/update notifications
- sets the `read_markers` directly from within the internal UserAPI

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-09-27 15:01:34 +02:00
Neil Alexander
f18bce93cc
Pinecone hybrid routing (update to matrix-org/pinecone#67) 2022-09-27 11:15:49 +01:00