Commit Graph

138 Commits

Author SHA1 Message Date
S7evinK
161f145176
Add NATS JetStream support (#1866)
* Add NATS JetStream support
Update shopify/sarama

* Fix addresses

* Don't change Addresses in Defaults

* Update saramajetstream

* Add missing error check

Keep typing events for at least one minute

* Use all configured NATS addresses

* Update saramajetstream

* Try setting up with NATS

* Make sure NATS uses own persistent directory (TODO: make this configurable)

* Update go.mod/go.sum

* Jetstream package

* Various other refactoring

* Build fixes

* Config tweaks, make random jetstream storage path for CI

* Disable interest policies

* Try to sane default on jetstream base path

* Try to use in-memory for CI

* Restore storage/retention

* Update nats.go dependency

* Adapt changes to config

* Remove unneeded TopicFor

* Dep update

* Revert "Remove unneeded TopicFor"

This reverts commit f5a4e4a339b6f94ec215778dca22204adaa893d1.

* Revert changes made to streams

* Fix build problems

* Update nats-server

* Update go.mod/go.sum

* Roomserver input API queuing using NATS

* Fix topic naming

* Prometheus metrics

* More refactoring to remove saramajetstream

* Add missing topic

* Don't try to populate map that doesn't exist

* Roomserver output topic

* Update go.mod/go.sum

* Message acknowledgements

* Ack tweaks

* Try to resume transaction re-sends

* Try to resume transaction re-sends

* Update to matrix-org/gomatrixserverlib@91dadfb

* Remove internal.PartitionStorer from components that don't consume keychanges

* Try to reduce re-allocations a bit in resolveConflictsV2

* Tweak delivery options on RS input

* Publish send-to-device messages into correct JetStream subject

* Async and sync roomserver input

* Update dendrite-config.yaml

* Remove roomserver tests for now (they need rewriting)

* Remove roomserver test again (was merged back in)

* Update documentation

* Docker updates

* More Docker updates

* Update Docker readme again

* Fix lint issues

* Send final event in `processEvent` synchronously (since this might stop Sytest from being so upset)

* Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that

* Go 1.16 instead of Go 1.13 for upgrade tests and Complement

* Revert "Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that"

This reverts commit 368675283fc44501f227639811bdb16dd5deef8c.

* Don't report any errors on `/send` to see what fun that creates

* Fix panics on closed channel sends

* Enforce state key matches sender

* Do the same for leave

* Various tweaks to make tests happier

Squashed commit of the following:

commit 13f9028e7a63662759ce7c55504a9d2423058668
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 15:47:14 2022 +0000

    Do the same for leave

commit e6be7f05c349fafbdddfe818337a17a60c867be1
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 15:33:42 2022 +0000

    Enforce state key matches sender

commit 85ede6d64bf10ce9b91cdd6d80f87350ee55242f
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 14:07:04 2022 +0000

    Fix panics on closed channel sends

commit 9755494a98bed62450f8001d8128e40481d27e15
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 13:38:22 2022 +0000

    Don't report any errors on `/send` to see what fun that creates

commit 3bb4f87b5dd56882febb4db5621db484c8789b7c
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 13:00:26 2022 +0000

    Revert "Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that"

    This reverts commit 368675283fc44501f227639811bdb16dd5deef8c.

commit fe2673ed7be9559eaca134424e403a4faca100b0
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 12:09:34 2022 +0000

    Go 1.16 instead of Go 1.13 for upgrade tests and Complement

commit 368675283fc44501f227639811bdb16dd5deef8c
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 11:51:45 2022 +0000

    Don't report event rejection errors via `/send`, since apparently this is upsetting tests that don't expect that

commit b028dfc08577bcf52e6cb498026e15fa5d46d07c
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Jan 4 10:29:08 2022 +0000

    Send final event in `processEvent` synchronously (since this might stop Sytest from being so upset)

* Merge in NATS Server v2.6.6 and nats.go v1.13 into the in-process connection fork

* Add `jetstream.WithJetStreamMessage` to make ack/nak-ing less messy, use process context in consumers

* Fix consumer component name in  federation API

* Add comment explaining where streams are defined

* Tweaks to roomserver input with comments

* Finish that sentence that I apparently forgot to finish in INSTALL.md

* Bump version number of config to 2

* Add comments around asynchronous sends to roomserver in processEventWithMissingState

* More useful error message when the config version does not match

* Set version in generate-config

* Fix version in config.Defaults

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-01-05 17:44:49 +00:00
Martin Ashby
e5a5be9f54
Added std type to logging config. (#2060)
* Added std type to logging level.

Added demux-ing of logging to stdout/err working around
https://github.com/sirupsen/logrus/issues/403

Turned off the default logging in favour of using hooks always.

Added fallback, if no stdout/err logger was configured, add one
automatically. This prevents unexpected lack of logging after an
upgrade in case the user's configuration file is not updated.

Fixes: https://github.com/matrix-org/dendrite/issues/2054
Signed-off-by: Martin Ashby <martin@ashbysoft.com>

* Fix build on Windows - revert function rename SetupHookLogging

Fixes https://github.com/matrix-org/dendrite/pull/2060/files#r761692681

* Revert logging formatter changes to log.go, base.go.

The same formatting should be applied on both windows and unix, so it
makes sense to keep it in files which are shared between both platforms.

Fixes
https://github.com/matrix-org/dendrite/pull/2060#discussion_r762557679

* Remove unnecessary startup logging about info level logger.

Fixes
https://github.com/matrix-org/dendrite/pull/2060#discussion_r763959474

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-12-16 15:37:01 +00:00
S7evinK
08a0278760
Add missing HTTP mode for userapi (#1982)
* Add missing internal api endpoint

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Add missing performKeyBackup endpoint

* Add missing http mode for userapi

* Fix failing tests

* Add error checks

* Fix sytest

* Update startup logic for HTTP mode

* Use userImpl for AS (annoying)

* Don't send device list updates for appservice devices

* Fix build

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-12-03 17:18:35 +00:00
S7evinK
25dcf80180
Ratelimit requests to /media/r0/download|upload (#2020)
* Add /media/r0/config handler

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Add rate limiting to media api

* Rename variable

* Add passing tests

* Don't send multiple headers

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-11-24 12:55:44 +00:00
Neil Alexander
c9419e51af
Don't populate config defaults where it doesn't make sense (#2058)
* Don't populate config defaults where it doesn't make sense

* Fix dendritejs builds
2021-11-24 11:57:39 +00:00
Neil Alexander
ec716793eb
Merge federationapi, federationsender, signingkeyserver components (#2055)
* Initial federation sender -> federation API refactoring

* Move base into own package, avoids import cycle

* Fix build errors

* Fix tests

* Add signing key server tables

* Try to fold signing key server into federation API

* Fix dendritejs builds

* Update embedded interfaces

* Fix panic, fix lint error

* Update configs, docker

* Rename some things

* Reuse same keyring on the implementing side

* Fix federation tests, `NewBaseDendrite` can accept freeform options

* Fix build

* Update create_db, configs

* Name tables back

* Don't rename federationsender consumer for now
2021-11-24 10:45:23 +00:00
Neil Alexander
e81ef1649b
Version 0.5.1 (#2047)
* Version bump, changelog

* Update to matrix-org/pinecone@8c7a1ad

* Update to matrix-org/pinecone@febf350
2021-11-16 11:37:03 +00:00
S7evinK
0c2872828b
Fix building on Windows (#2008)
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
2021-11-02 16:50:59 +00:00
PiotrKozimor
dec05c3347
Run gofmt on dendrite - apply go 1.17 preferred build tags (#2021) 2021-11-02 16:48:48 +00:00
Ryan W
1cd4d50181
Added .well-known/matrix/server endpoint (#1988)
* Added .well-known/matrix/server endpoint

Signed-off-by: Ryan Whittington <twentybitdev@gmail.com>

* Replaced tabs with spaces

Signed-off-by: Ryan Whittington <twentybitdev@gmail.com>
2021-09-10 10:05:31 +01:00
Ryan W
a624eab309
- Removed double imports (#1989)
- Lower cased error messages

Signed-off-by: Ryan Whittington <twentybitdev@gmail.com>

Co-authored-by: kegsay <kegan@matrix.org>
2021-09-08 17:31:03 +01:00
Neil Alexander
037ff4fb23
Version 0.5.0 2021-08-24 15:03:28 +01:00
Neil Alexander
cd3823308c
Version 0.5.0rc1 2021-08-18 14:29:45 +01:00
Sambhav Saggi
1bee1ae204
Syslog integration (#1952)
* Syslog integration, part 1

* Add protocol, make sure syslog actually logs

* Make golangci-lint happy about shadow variables

* Add syslog tag, wrap syslog in logLevelHook
2021-08-17 16:18:51 +01:00
Kegan Dougal
ed4097825b Factor out StatementList to sqlutil and use it in userapi
It helps with the boilerplate.
2021-07-28 18:30:04 +01:00
Neil Alexander
e3679799ea
Version 0.4.1 2021-07-26 12:52:11 +01:00
Neil Alexander
c1447a58e5
Various alias fixes (#1934)
* Generate m.room.canonical_alias instead of legacy m.room.aliases

* Add omitempty tags

* Add aliases endpoint to client API

* Check power levels when setting aliases

* Don't return null on /aliases

* Don't return error if the state event fails

* Update sytest-whitelist

* Don't send updated m.room.canonical_alias events

* Don't check PLs after all because for local aliases they are apparently irrelevant

* Fix some bugs

* Allow deleting a local alias with enough PL

* Fix some more bugs

* Update sytest-whitelist

* Fix copyright notices

* Review comments
2021-07-21 16:53:50 +01:00
Neil Alexander
f63068df3b
Only include go-sqlite3 on the relevant binaries (#1900)
* Only include go-sqlite3 on the relevant binaries

* The driver name is always sqlite3 now

* Update to matrix-org/go-sqlite3-js@e537baa
2021-07-20 11:18:14 +01:00
Neil Alexander
0ff078ebd7
Fixed log printing bug (closes #1885)
Signed-off-by: Kilos kilosliu@workly.ai>
2021-07-20 10:49:12 +01:00
Neil Alexander
89a16bdcd9
Version 0.4.0 2021-07-12 11:48:29 +01:00
kegsay
70e4bbda3b
Only log filename and not entire path (#1906) 2021-07-08 13:13:27 +01:00
Kegsay
f8d3a762c4
Add a per-room mutex to federationapi when processing transactions (#1810)
* Add a per-room mutex to federationapi when processing transactions

This has numerous benefits:
 - Prevents us doing lots of state resolutions in busy rooms. Previously, room forks would always result
   in a state resolution being performed immediately, without checking if we were already doing this in
   a different transaction. Now they will queue up, resulting in fewer calls to `/state_ids`, `/g_m_e`, etc.
 - Prevents memory usage from growing too large as a result and potentially OOMing.

And costs:
 - High traffic rooms will be slightly slower due to head-of-line blocking from other servers,
   though this has always been an issue as roomserver has a per-room mutex already.

* Fix unit tests

* Correct mutex lock ordering
2021-03-30 10:01:32 +01:00
Kegsay
af41f6d454
Add Sentry support (#1803)
* Add Sentry support

* Use HTTP Sentry properly maybe

* Capture panics

* Log fed Sentry stuff correctly

* British english linter
2021-03-24 10:25:24 +00:00
Neil Alexander
6a35d9f1b1
Version 0.3.11 2021-03-02 11:46:35 +00:00
Neil Alexander
2fdc318f2c
Version 0.3.10 2021-02-17 16:05:04 +00:00
Neil Alexander
2386e0c7af
Gradually evict oldest cache entries (#1768)
* Gradually evict oldest cache entries

* Keep the remaining 10% of cached entries
2021-02-17 13:50:27 +00:00
Neil Alexander
397b158c05
Version 0.3.9 2021-02-04 12:39:43 +00:00
Neil Alexander
61090cbe35
Version 0.3.8 2021-01-28 13:37:50 +00:00
Neil Alexander
e14e9a1606
Version 0.3.7 2021-01-26 13:13:48 +00:00
Neil Alexander
9f443317bc
Graceful shutdowns (#1734)
* Initial graceful stop

* Fix dendritejs

* Use process context for outbound federation requests in destination queues

* Reduce logging

* Fix log level
2021-01-26 12:56:20 +00:00
Neil Alexander
67c89b3c5e
Version 0.3.6 2021-01-18 13:37:49 +00:00
Neil Alexander
ec1b017906
Version 0.3.5 2021-01-11 13:55:40 +00:00
Neil Alexander
4a0461378a
Version 0.3.4 2020-12-18 14:53:31 +00:00
Neil Alexander
b891c00b09
Add RoomInfo cache, remove RoomServerRoomNIDsCache (#1646)
* Add RoomInfo cache, remove RoomServerRoomNID cache, ensure caches are thread-safe

* Don't panic if the roomInfo isn't known yet

* LRU package is already threadsafe

* Use RoomInfo cache to find room version if possible in Events()

* Adding comments about RoomInfoCache safety
2020-12-16 12:15:12 +00:00
Neil Alexander
851c02659a
Version 0.3.3 2020-12-09 14:13:02 +00:00
Neil Alexander
5d65a879a5
Federation sender event cache (#1614)
* Cache federation sender events

* Store in the correct cache

* Update federation event cache

* Fix Unset

* Give EDUs same caching treatment as PDUs

* Make federationsender_cache_size configurable

* Default caches configuration

* Fix unit tests

* Revert "Fix unit tests"

This reverts commit 24eb5d22524f20e1024b1475debe61ae20538a5a.

* Revert "Default caches configuration"

This reverts commit 464ecd1e64b9d2983f6fd5430e9607519d543cb3.

* Revert "Make federationsender_cache_size configurable"

This reverts commit 4631f5324151e006a15d6f19008f06361b994607.
2020-12-04 14:52:10 +00:00
Neil Alexander
1f3a498601
Version 0.3.2 2020-12-02 17:46:28 +00:00
Neil Alexander
b5aa7ca3ab
Top-level setup package (#1605)
* Move config, setup, mscs into "setup" top-level folder

* oops, forgot the EDU server

* Add setup

* goimports
2020-12-02 17:41:00 +00:00
Neil Alexander
bdf6490375
Add ability to disable federation (#1604)
* Allow disabling federation

* Don't start federation queues if disabled

* Fix for Go 1.13
2020-12-02 15:10:03 +00:00
Neil Alexander
9c52f82736
Version 0.3.1 2020-11-20 11:38:58 +00:00
Neil Alexander
c636be5070
Update last seen on sync requests (#1593)
* Update last seen on sync requests

* Fix MSC2836 unit tests

* Only update once per minute

* Remove debug logging

* Configurable option

* Simplify updateLastSeen/cleanLastSeen
2020-11-20 11:29:02 +00:00
Kegsay
6353b0b7e4
MSC2836: Threading - part one (#1589)
* Add mscs/hooks package, begin work for msc2836

* Flesh out hooks and add SQL schema

* Begin implementing core msc2836 logic

* Add test harness

* Linting

* Implement visibility checks; stub out APIs for tests

* Flesh out testing

* Flesh out walkThread a bit

* Persist the origin_server_ts as well

* Edges table instead of relationships

* Add nodes table for event metadata

* LEFT JOIN to extract origin_server_ts for children

* Add graph walking structs

* Implement walking algorithm

* Add more graph walking tests

* Add auto_join for local rooms

* Fix create table syntax on postgres

* Add relationship_room_id|servers to the unsigned section of events

* Persist the parent room_id/servers in edge metadata

Other events cannot assert the true room_id/servers for the
parent event, only make claims to them, hence why this is
edge metadata.

* guts to pass through room_id/servers

* Refactor msc2836 to allow handling from federation

* Add JoinedVia to PerformJoin responses

* Fix tests; review comments
2020-11-19 11:34:59 +00:00
Neil Alexander
20a01bceb2
Pass pointers to events — reloaded (#1583)
* Pass events as pointers

* Fix lint errors

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update to matrix-org/gomatrixserverlib#240
2020-11-16 15:44:53 +00:00
Neil Alexander
3856e547de
Version 0.3.0 2020-11-16 11:13:05 +00:00
S7evinK
cb41c992f7
Add -version flag (#1581)
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
2020-11-16 09:54:58 +00:00
S7evinK
bcb89ada5e
Implement read receipts (#1528)
* fix conversion from int to string yields a string of one rune, not a string of digits

* Add receipts table to syncapi

* Use StreamingToken as the since value

* Add required method to testEDUProducer

* Make receipt json creation "easier" to read

* Add receipts api to the eduserver

* Add receipts endpoint

* Add eduserver kafka consumer

* Add missing kafka config

* Add passing tests to whitelist

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Fix copy & paste error

* Fix column count error

* Make outbound federation receipts pass

* Make "Inbound federation rejects receipts from wrong remote" pass

* Don't use errors package

* - Add TODO for batching requests
- Rename variable

* Return a better error message

* - Use OutputReceiptEvent instead of InputReceiptEvent as result
- Don't use the errors package for errors
- Defer CloseAndLogIfError to close rows
- Fix Copyright

* Better creation/usage of JoinResponse

* Query all joined rooms instead of just one

* Update gomatrixserverlib

* Add sqlite3 migration

* Add postgres migration

* Ensure required sequence exists before running migrations

* Clarification on comment

* - Fix a bug when creating client receipts
- Use concrete types instead of interface{}

* Remove dead code
Use key for timestamp

* Fix postgres query...

* Remove single purpose struct

* Use key/value directly

* Only apply receipts on initial sync or if edu positions differ,
otherwise we'll be sending the same receipts over and over again.

* Actually update the id, so it is correctly send in syncs

* Set receipt on request to /read_markers

* Fix issue with receipts getting overwritten

* Use fmt.Errorf instead of pkg/errors

* Revert "Add postgres migration"

This reverts commit 722fe5a04628882b787d096942459961db159b06.

* Revert "Add sqlite3 migration"

This reverts commit d113b03f6495a4b8f8bcf158a3d00b510b4240cc.

* Fix selectRoomReceipts query

* Make golangci-lint happy

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-11-09 18:46:11 +00:00
S7evinK
d5675feb96
Add possibilty to configure MaxMessageBytes for sarama (#1563)
* Add configuration for max_message_bytes for sarama

* Log all errors when sending multiple messages

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Add missing config

* - Better comments on what MaxMessageBytes is used for
- Also sets the size the consumer may use
2020-10-27 14:11:37 +00:00
Neil Alexander
59428cdde3
Version 0.2.1 (#1558)
* Version 0.2.1

* Credit where credit due

* Update CHANGES.md

* Include PRs
2020-10-22 12:38:12 +01:00
Neil Alexander
24e38c4135
Internal HTTP APIs over H2C (#1541)
* H2C on internal HTTP because SCIENCE

* Update comments
2020-10-20 17:13:12 +01:00
Neil Alexander
6a16d46fba
Version 0.2.0 (#1551)
* v0.2.0-rc2

* Update CHANGES.md (also with some markdown lint suggestions)

* Update version number

* Update CHANGES.md

* Update CHANGES.md

* Update CHANGES.md

* Add known issue

* Update CHANGES.md
2020-10-20 16:46:53 +01:00