Commit Graph

236 Commits

Author SHA1 Message Date
Neil Alexander
a4601bb698
Update to matrix-org/gomatrixserverlib@f3e2ef8 (matrix-org/matrix-doc#3667) 2022-01-24 10:28:23 +00:00
kegsay
db7d9cba8a
BREAKING: Remove Partitioned Stream Positions (#2096)
* go mod tidy

* Break complement to check it fails CI

* Remove partitioned stream positions

This was used by the device list stream position. The device list position
now corresponds to the `Offset`, and the partition is always 0, in prep
for removing reliance on Kafka topics for device list changes.

* Linting

* Migrate old style tokens to new style because element-web doesn't soft-logoout on 4xx errors on /sync
2022-01-20 15:26:45 +00:00
S7evinK
161f145176
Add NATS JetStream support (#1866)
* Add NATS JetStream support
Update shopify/sarama

* Fix addresses

* Don't change Addresses in Defaults

* Update saramajetstream

* Add missing error check

Keep typing events for at least one minute

* Use all configured NATS addresses

* Update saramajetstream

* Try setting up with NATS

* Make sure NATS uses own persistent directory (TODO: make this configurable)

* Update go.mod/go.sum

* Jetstream package

* Various other refactoring

* Build fixes

* Config tweaks, make random jetstream storage path for CI

* Disable interest policies

* Try to sane default on jetstream base path

* Try to use in-memory for CI

* Restore storage/retention

* Update nats.go dependency

* Adapt changes to config

* Remove unneeded TopicFor

* Dep update

* Revert "Remove unneeded TopicFor"

This reverts commit f5a4e4a339b6f94ec215778dca22204adaa893d1.

* Revert changes made to streams

* Fix build problems

* Update nats-server

* Update go.mod/go.sum

* Roomserver input API queuing using NATS

* Fix topic naming

* Prometheus metrics

* More refactoring to remove saramajetstream

* Add missing topic

* Don't try to populate map that doesn't exist

* Roomserver output topic

* Update go.mod/go.sum

* Message acknowledgements

* Ack tweaks

* Try to resume transaction re-sends

* Try to resume transaction re-sends

* Update to matrix-org/gomatrixserverlib@91dadfb

* Remove internal.PartitionStorer from components that don't consume keychanges

* Try to reduce re-allocations a bit in resolveConflictsV2

* Tweak delivery options on RS input

* Publish send-to-device messages into correct JetStream subject

* Async and sync roomserver input

* Update dendrite-config.yaml

* Remove roomserver tests for now (they need rewriting)

* Remove roomserver test again (was merged back in)

* Update documentation

* Docker updates

* More Docker updates

* Update Docker readme again

* Fix lint issues

* Send final event in `processEvent` synchronously (since this might stop Sytest from being so upset)

* Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that

* Go 1.16 instead of Go 1.13 for upgrade tests and Complement

* Revert "Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that"

This reverts commit 368675283fc44501f227639811bdb16dd5deef8c.

* Don't report any errors on `/send` to see what fun that creates

* Fix panics on closed channel sends

* Enforce state key matches sender

* Do the same for leave

* Various tweaks to make tests happier

Squashed commit of the following:

commit 13f9028e7a63662759ce7c55504a9d2423058668
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 15:47:14 2022 +0000

    Do the same for leave

commit e6be7f05c349fafbdddfe818337a17a60c867be1
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 15:33:42 2022 +0000

    Enforce state key matches sender

commit 85ede6d64bf10ce9b91cdd6d80f87350ee55242f
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 14:07:04 2022 +0000

    Fix panics on closed channel sends

commit 9755494a98bed62450f8001d8128e40481d27e15
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 13:38:22 2022 +0000

    Don't report any errors on `/send` to see what fun that creates

commit 3bb4f87b5dd56882febb4db5621db484c8789b7c
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 13:00:26 2022 +0000

    Revert "Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that"

    This reverts commit 368675283fc44501f227639811bdb16dd5deef8c.

commit fe2673ed7be9559eaca134424e403a4faca100b0
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 12:09:34 2022 +0000

    Go 1.16 instead of Go 1.13 for upgrade tests and Complement

commit 368675283fc44501f227639811bdb16dd5deef8c
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 11:51:45 2022 +0000

    Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that

commit b028dfc08577bcf52e6cb498026e15fa5d46d07c
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 10:29:08 2022 +0000

    Send final event in `processEvent` synchronously (since this might stop Sytest from being so upset)

* Merge in NATS Server v2.6.6 and nats.go v1.13 into the in-process connection fork

* Add `jetstream.WithJetStreamMessage` to make ack/nak-ing less messy, use process context in consumers

* Fix consumer component name in  federation API

* Add comment explaining where streams are defined

* Tweaks to roomserver input with comments

* Finish that sentence that I apparently forgot to finish in INSTALL.md

* Bump version number of config to 2

* Add comments around asynchronous sends to roomserver in processEventWithMissingState

* More useful error message when the config version does not match

* Set version in generate-config

* Fix version in config.Defaults

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-01-05 17:44:49 +00:00
Martin Ashby
e5a5be9f54
Added std type to logging config. (#2060)
* Added std type to logging level.

Added demux-ing of logging to stdout/err working around
https://github.com/sirupsen/logrus/issues/403

Turned off the default logging in favour of using hooks always.

Added fallback, if no stdout/err logger was configured, add one
automatically. This prevents unexpected lack of logging after an
upgrade in case the user's configuration file is not updated.

Fixes: https://github.com/matrix-org/dendrite/issues/2054
Signed-off-by: Martin Ashby <martin@ashbysoft.com>

* Fix build on Windows - revert function rename SetupHookLogging

Fixes https://github.com/matrix-org/dendrite/pull/2060/files#r761692681

* Revert logging formatter changes to log.go, base.go.

The same formatting should be applied on both windows and unix, so it
makes sense to keep it in files which are shared between both platforms.

Fixes
https://github.com/matrix-org/dendrite/pull/2060#discussion_r762557679

* Remove unnecessary startup logging about info level logger.

Fixes
https://github.com/matrix-org/dendrite/pull/2060#discussion_r763959474

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-12-16 15:37:01 +00:00
Neil Alexander
6b574ac537
Update to matrix-org/pinecone@aa2808d 2021-12-13 13:29:01 +00:00
Neil Alexander
310edd737c
Fix P2P demo builds 2021-11-25 10:48:44 +00:00
Neil Alexander
8f2a838452
Update to matrix-org/pinecone@0cc483b 2021-11-25 09:49:00 +00:00
Neil Alexander
e81ef1649b
Version 0.5.1 (#2047)
* Version bump, changelog

* Update to matrix-org/pinecone@8c7a1ad

* Update to matrix-org/pinecone@febf350
2021-11-16 11:37:03 +00:00
Neil Alexander
58b831ad36
Mark room versions 8 and 9 as unstable (update to matrix-org/gomatrixserverlib@15a64d2) 2021-11-15 19:29:03 +00:00
Neil Alexander
5565e40395
Fix default power levels (update to matrix-org/gomatrixserverlib@af2616b) 2021-11-12 15:16:22 +00:00
Neil Alexander
59cf8e936e
Fix sytest 2021-11-08 10:32:21 +00:00
Neil Alexander
9cd4a1f97b
Update to matrix-org/gomatrixserverlib@ed70208 (hopefully will fix sytest) 2021-11-08 09:45:39 +00:00
Neil Alexander
77096898f0
Further state resolution v2 optimisations 2021-11-04 10:35:23 +00:00
Neil Alexander
3c049c068f
State resolution v2 performance improvements (reduce allocs in Kahn's algorithm, update to matrix-org/gomatrixserverlib@91dadfb) 2021-11-04 09:09:26 +00:00
Neil Alexander
b4a007ecce
Support joining version 8 and 9 rooms (update to matrix-org/gomatrixserverlib@13366e7) 2021-11-02 17:15:06 +00:00
hacktivista
1ec5b0709b
Support WebP based thumbnails (#2033)
Co-authored-by: Felix Freeman <libsys@hacktivista.org>
2021-11-02 10:15:13 +00:00
Neil Alexander
fbd1a0ab13
Update to matrix-org/gomatrixserverlib@5e02b64 2021-11-02 10:13:38 +00:00
Neil Alexander
4c129e3e3d
Update Pinecone demo 2021-10-22 11:38:04 +01:00
kegsay
09511c6086
Upgrade containerd dep (#2024) 2021-10-21 12:13:01 +01:00
Neil Alexander
2f68ab7c2f
Update to matrix-org/pinecone@d568c31638 2021-10-12 14:48:51 +01:00
Neil Alexander
a5cb8c00bd
Update Pinecone demo 2021-10-08 15:29:30 +01:00
Aaron Bieber
be9e1b5367
Bump github.com/matrix-org/dugong (#2011)
The latest version of dugong has a fix for removing log files after they are
compressed.

Signed-off-by: Aaron Bieber <aaron@bolddaemon.com>
2021-10-05 16:03:55 +01:00
Neil Alexander
d08399d1a4
Update to matrix-org/pinecone@4f07afa 2021-09-20 16:22:10 +01:00
Neil Alexander
57fb44ac4b
Update pinecone to matrix-org/pinecone#12 2021-09-10 14:47:59 +01:00
Neil Alexander
329c9e671c
Update to matrix-org/pinecone@e692df1 2021-08-19 16:07:28 +01:00
Neil Alexander
9a2dcd6078
Update to matrix-org/pinecone@21e3b39228 2021-08-19 15:02:50 +01:00
Neil Alexander
ff21675c5b
Cross-signing fixes, notifications via sync, federation (#1974)
* Initial work on signing key update EDUs

* Fix build

* Produce/consume EDUs

* Producer logging

* Only produce key change notifications for local users

* Better naming

* Try to notify sync

* Enable feature

* Use key change topic

* Don't bother verifying signatures, validate key lengths if we can, notifier fixes

* Copyright notices

* Remove tests from whitelist until matrix-org/sytest#1117

* Some review comment fixes

* Update to matrix-org/gomatrixserverlib@f9416ac

* Remove unneeded parameter
2021-08-17 13:44:30 +01:00
Neil Alexander
b1377d991a
Cross-signing signature handling (#1965)
* Handle other signatures

* Decorate key ID properly

* Match by key IDs

* Tweaks

* Fixes

* Fix /user/keys/query bug, review comments, update sytest-whitelist

* Various wtweaks

* Fix wiring for keyserver in API mode

* Additional fixes
2021-08-09 14:35:24 +01:00
Neil Alexander
8e5a0139b5
Update to matrix-org/gomatrixserverlib@03e40fa 2021-08-05 11:48:22 +01:00
Neil Alexander
7a9a2547b3
Cross-signing storage code (#1959) 2021-08-04 17:31:18 +01:00
Neil Alexander
da101469fa
Minimum version Go 1.15 (#1957) 2021-08-03 11:35:29 +01:00
Neil Alexander
d4aa311c74
Update containerd dependency 2021-08-03 11:17:27 +01:00
Neil Alexander
a81c499f65
Update dependencies (#1956) 2021-08-03 11:15:17 +01:00
Neil Alexander
3e01a88a0c
Update to neilalexander/utp@54ae7b1 2021-07-27 21:34:40 +01:00
Neil Alexander
39e8d1cc6f
Track knocking in membership updater (#1935)
* Topologically sort outliers in SendEventWithState

* Knock in membership updater

* Update gomatrixserverlib

* Update gomatrixserverlib

* Get the NID of the knock event properly for the membership updater
2021-07-22 12:26:58 +01:00
Neil Alexander
43ac66e0b4
Update to matrix-org/gomatrixserverlib@b9eb787 2021-07-22 09:46:31 +01:00
Neil Alexander
9008b1c111
Update to matrix-org/gomatrixserverlib@ae88543 2021-07-21 18:46:31 +01:00
Neil Alexander
7650998308
Update to matrix-org/gomatrixserverlib#271 2021-07-21 17:21:15 +01:00
Neil Alexander
c1447a58e5
Various alias fixes (#1934)
* Generate m.room.canonical_alias instead of legacy m.room.aliases

* Add omitempty tags

* Add aliases endpoint to client API

* Check power levels when setting aliases

* Don't return null on /aliases

* Don't return error if the state event fails

* Update sytest-whitelist

* Don't send updated m.room.canonical_alias events

* Don't check PLs after all because for local aliases they are apparently irrelevant

* Fix some bugs

* Allow deleting a local alias with enough PL

* Fix some more bugs

* Update sytest-whitelist

* Fix copyright notices

* Review comments
2021-07-21 16:53:50 +01:00
Neil Alexander
2f5e4f5991
Support initial_state properly in /createRoom (#1932)
* Refactor room creation to allow initial_state

* GMSL types

* Tweaks to alias

* Fix ordering

* Fix bugs

* Fix create content

* Only unmarshal create content if specified

* Review comments @Kegsay
2021-07-21 12:31:46 +01:00
Neil Alexander
c6acb94af4
Update to matrix-org/gomatrixserverlib#270 2021-07-20 15:14:24 +01:00
J. Ryan Stinnett
034f5ae3f1
Add startup testing for Wasm Pinecone build (#1910)
* Only include go-sqlite3 on the relevant binaries

* The driver name is always sqlite3 now

* Update to matrix-org/go-sqlite3-js@e537baa

* Add initial Wasm test harness

* Upgrade go-sqlite3-js

This fixes an error about semicolons in single statements.

* Add browser-like WebSocket API for testing

* Upgrade go-sqlite3-js

This upgrade includes printing panic messages next to stacks.

* Run for all PRs targeting any branch

* Use manual Node caching

* Temporarily run for all pushes

* Use npm ci instead of install

* Use HTTPS auth for repo packages

* Match path style from build.sh

* update utp

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-07-20 12:14:58 +01:00
Neil Alexander
f63068df3b
Only include go-sqlite3 on the relevant binaries (#1900)
* Only include go-sqlite3 on the relevant binaries

* The driver name is always sqlite3 now

* Update to matrix-org/go-sqlite3-js@e537baa
2021-07-20 11:18:14 +01:00
Neil Alexander
628e4cc709
Update to matrix-org/gomatrixserverlib#269 2021-07-19 15:09:01 +01:00
Neil Alexander
1827dd7c09
Update dendrite-demo-yggdrasil to Yggdrasil 0.4 (#1921)
* Update Yggdrasil P2P demo for Yggdrasil v0.4

* Build fixes
2021-07-15 12:16:44 +01:00
kegsay
e8de96b78c
sytests: fix failing PL tests by updating GMSL (#1920) 2021-07-14 15:48:14 +01:00
Kegan Dougal
93cf1e7590 Actually bump GMSL 2021-07-12 18:58:25 +01:00
kegsay
1ed732cc78
Implement /_synapse/admin/v1/register (#1911)
* Implement /_synapse/admin/v1/register

This is implemented identically to Synapse, so scripts which work
with Synapse should work with Dendrite.

```
    Test 27 POST /_synapse/admin/v1/register with shared secret... OK
    Test 28 POST /_synapse/admin/v1/register admin with shared secret... OK
    Test 29 POST /_synapse/admin/v1/register with shared secret downcases capitals... OK
    Test 30 POST /_synapse/admin/v1/register with shared secret disallows symbols... OK
```

Sytest however has `implementation_specific => "synapse"` which stops these
tests from running.

* Add missing muxes to gobind

* Linting
2021-07-09 16:52:31 +01:00
kegsay
d72d634391
Add dendrite-upgrade-test (#1901)
* Add WIP binary for testing dendrite version upgrades

* WIP dendrite upgrade work

* Finish dendrite upgrade checks

* go mod tidy

* Review comments; print container logs on failure

* Linting
2021-07-07 12:06:17 +01:00
Neil Alexander
355ce7ed30
Update to matrix-org/gomatrixserverlib#266 (+ missing file) 2021-07-02 16:30:37 +01:00