Commit Graph

2506 Commits

Author SHA1 Message Date
Neil Alexander
99b696e775
Tweak cache counters (#2575)
* Tweak cache counters

This makes the number of counters relative to the
maximum cache size. Since the counters
effectively manage the size of the bloom filter,
larger caches need more counters and smaller
caches need less.

10 counters per 1KB data means that the default
cache size of 1GB should result in a bloom filter
and TinyLRU admission set of about 16MB
estimated.

* Remove line left by accident
2022-07-18 14:49:41 +01:00
Till
a7e92f8cb9
History visibility database changes (#2533)
* Add new history_visibility column

* Update SQL queries to include history_visibility

* Store the history visibilty calculated by the roomserver

* Update GMSL

* Update migrations

* Fix migration

* Update GMSL

* Fix `go.sum`

* Update GMSL to use sql.Scanner & sql.Valuer

* Re-order migration/table creation

* Update gomatrixserverlib

* Add history_visibility column to current_room_state

* Fix migrations

* Return error instead of Fatal log

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-07-18 14:46:15 +02:00
Maximilian Gaedig
c0c909d306
Fix connection_string format in dendrite-sample.polylith.yaml (#2574) 2022-07-17 23:01:52 +01:00
andreever
4ff57993ab
Update 1_createusers.md (#2571)
* Update 1_createusers.md

Added description on how to create user accounts when running in docker.

* Update 1_createusers.md

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-07-15 16:33:05 +01:00
Neil Alexander
90bf01d8b1
Use sync API database in filterSharedUsers (#2572)
* Add function to the sync API storage package for filtering shared users

* Use the database instead of asking the RS API

* Fix unit tests

* Fix map handling in `filterSharedUsers`
2022-07-15 16:25:26 +01:00
Neil Alexander
69c86295f7
Update sytest-whitelist 2022-07-14 13:51:00 +01:00
Neil Alexander
950659320e
Add a Troubleshooting page 2022-07-14 10:09:26 +01:00
Neil Alexander
a1f9b02edf
Pointerise types.RoomInfo in the cache so we can update it in-place in the latest events updater 2022-07-13 10:13:34 +01:00
Neil Alexander
9cd8e9d4b9
Use /v3 to request media from remote servers (update to matrix-org/gomatrixserverlib#324) 2022-07-13 09:32:03 +01:00
Till
09f0ff14c8
Minor SendToDevice fix (#2565)
* Avoid unnecessary marshalling if sending to the local server

* Fix ordering of ToDevice messages

* Revive SendToDevice test
2022-07-12 08:23:58 +02:00
Neil Alexander
3ea21273bc
Ristretto cache (#2563)
* Try Ristretto cache

* Tweak

* It's beautiful

* Update GMSL

* More strict keyable interface

* Fix that some more

* Make less panicky

* Don't enforce mutability checks for now

* Determine mutability using deep equality

* Tweaks

* Namespace keys

* Make federation caches mutable

* Update cost estimation, add metric

* Update GMSL

* Estimate cost for metrics better

* Reduce counters a bit

* Try caching events

* Some guards

* Try again

* Try this

* Use separate caches for hopefully better hash distribution

* Fix bug with admitting events into cache

* Try to fix bugs

* Check nil

* Try that again

* Preserve order jeezo this is messy

* thanks VS Code for doing exactly the wrong thing

* Try this again

* Be more specific

* aaaaargh

* One more time

* That might be better

* Stronger sorting

* Cache expiries, async publishing of EDUs

* Put it back

* Use a shared cache again

* Cost estimation fixes

* Update ristretto

* Reduce counters a bit

* Clean up a bit

* Update GMSL

* 1GB

* Configurable cache sizees

* Tweaks

* Add `config.DataUnit` for specifying friendly cache sizes

* Various tweaks

* Update GMSL

* Add back some lazy loading caching

* Include key in cost

* Include key in cost

* Tweak max age handling, config key name

* Only register prometheus metrics if requested

* Review comments @S7evinK

* Don't return errors when creating caches (it is better just to crash since otherwise we'll `nil`-pointer exception everywhere)

* Review comments

* Update sample configs

* Update GHA Workflow

* Update Complement images to Go 1.18

* Remove the cache test from the federation API as we no longer guarantee immediate cache admission

* Don't check the caches in the renewal test

* Possibly fix the upgrade tests

* Update to matrix-org/gomatrixserverlib#322

* Update documentation to refer to Go 1.18
2022-07-11 14:31:31 +01:00
Neil Alexander
eb8dc50a97
Update Pinecone to matrix-org/pinecone@1ce778f 2022-07-08 14:54:08 +01:00
Till
f76f28e6db
Fix issue uint64 values with high bit are not supported in presence (#2562)
* Fix issue #2528

* Use gomatrixserverlib.Timestamp

* Use ParseUint instead of ParseInt
2022-07-07 16:29:25 +02:00
Till
f3e8a9a4cb
Fix nil pointer access when redacting events (#2560) 2022-07-07 11:40:53 +02:00
Neil Alexander
460dccf93d
Hopefully fix read receipts timestamps (#2557)
This should avoid coercions between signed and unsigned ints which might fix problems like `sql: converting argument $5 type: uint64 values with high bit set are not supported`.
2022-07-05 17:13:26 +01:00
Neil Alexander
c0f824d437
Wrap error from SnapshotNIDFromEventID 2022-07-05 15:06:10 +01:00
Neil Alexander
d4341a2d97
Return clearer error when no state NID exists for an event (#2555) 2022-07-05 15:01:34 +01:00
Till
5087b36af0
Fix QuerySharedUsers for the SyncAPI keychange consumer (#2554)
* Make more use of base.BaseDendrite

* Fix QuerySharedUsers if no UserIDs are supplied
2022-07-05 14:50:56 +02:00
Till
f29cdb26f6
Use new testrig for key changes tests (#2552)
* Use new testrig for tests

* Log the error message
2022-07-05 14:50:24 +02:00
Kabir Kwatra
43147bd654
feat+fix: Ignore unknown keys and verify required fields are present in appservice registration files (#2550)
* fix: ignore unknown keys in appservice configs

fixes matrix-org/dendrite#1567

* feat: verify required fields in appservice configs
2022-07-05 12:53:51 +01:00
Neil Alexander
b5c55faf98
Version 0.8.9 (#2549)
* Version 0.8.9

* Update changelog
2022-07-01 12:00:32 +01:00
Neil Alexander
b50a24c666
Roomserver producers package (#2546)
* Give the roomserver a producers package

* Change init point

* Populate ACLs API

* Fix build issues

* `RoomEventProducer` naming
2022-07-01 10:54:07 +01:00
Till
89cd0e8fc1
Try to fix backfilling (#2548)
* Try to fix backfilling

* Return start/end to not confuse clients

* Update GMSL

* Update GMSL
2022-07-01 11:49:26 +02:00
Neil Alexander
086f182e24
Disable WebAssembly builds for now 2022-07-01 09:50:06 +01:00
Neil Alexander
54bed4c593
Blacklist Guest users can join guest_access rooms test until it can be investigated 2022-07-01 09:37:54 +01:00
Till
561c159ad7
Silence presence logs (#2547) 2022-06-30 12:34:37 +02:00
Neil Alexander
519bc1124b
Add evacuateUser endpoint, use it when deactivating accounts (#2545)
* Add `evacuateUser` endpoint, use it when deactivating accounts

* Populate the API

* Clean up user devices when deactivating

* Include invites, delete pushers
2022-06-29 15:29:39 +01:00
Neil Alexander
2dea466685
Return an error if trying to invite a malformed user ID (#2543) 2022-06-29 12:32:24 +01:00
Till
2086992caf
Don't return end if there are not more messages (#2542)
* Be more spec compliant

* Move lazyLoadMembers to own method
2022-06-29 10:49:12 +02:00
Jean Lucas
920a20821b
Fix nats.go commit (#2540)
Signed-off-by: Jean Lucas <jean@4ray.co>
2022-06-27 09:15:19 +01:00
Neil Alexander
7120eb6bc9
Add InputDeviceListUpdate to the keyserver, remove old input API (#2536)
* Add `InputDeviceListUpdate` to the keyserver, remove old input API

* Fix copyright

* Log more information when a device list update fails
2022-06-15 14:27:07 +01:00
Till
1b90cc9536
Fix rare panic when returning user devices over federation (#2534) 2022-06-15 12:50:02 +02:00
Neil Alexander
4c2a10f1a6
Handle state before, send history visibility in output (#2532)
* Check state before event

* Tweaks

* Refactor a bit, include in output events

* Don't waste time if soft failed either

* Tweak control flow, comments, use GMSL history visibility type
2022-06-13 15:11:10 +01:00
Emanuele Aliberti
c500958583
generic CaddyFile in front of Dendrite (monolith) (#2531)
for Caddy 2.5.x

Co-authored-by: emanuele.aliberti <emanuele.aliberti@mtka.eu>
2022-06-13 12:08:46 +01:00
Till Faelligen
e1136f4d3e Make the linter happy again 2022-06-13 11:46:59 +02:00
Neil Alexander
0a7f7dc716
Add --difference to resolve-state tool 2022-06-13 10:16:30 +01:00
Neil Alexander
89d2adadbd
Attempt to raise the file descriptor limit at startup (#2527) 2022-06-10 10:58:04 +01:00
Neil Alexander
1030072285
Rename the page to "Optimise your installation" 2022-06-10 10:18:32 +01:00
Neil Alexander
16ed1633b6
Highlighting in docs 2022-06-10 10:15:14 +01:00
Neil Alexander
e2a64773ce
Add new next steps page to the documentation 2022-06-10 10:14:15 +01:00
Till
660f7839f5
Correctly redact events over federation (#2526)
* Ensure we check powerlevel/origin before redacting an event

* Add passing test

* Use pl.UserLevel

* Make check more readable, also check for the sender
2022-06-09 18:38:07 +02:00
Neil Alexander
83797573be
Version 0.8.8 (#2525) 2022-06-09 12:18:50 +01:00
Till
289b3c5608
Allow "registration is idempotent, with username specified" to pass (#2488)
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-06-09 11:26:48 +01:00
Neil Alexander
3cdefcf765
Update sample configs for exempt_user_ids 2022-06-08 09:13:25 +01:00
Neil Alexander
c4df6d7723
Optimise state res v2 by only updating the allower when necessary (update to matrix-org/gomatrixserverlib@e55d796) 2022-06-07 15:34:58 +01:00
Neil Alexander
b21a2223ef
Optimise state res v2 by parsing power level content less often (update to matrix-org/gomatrixserverlib@e938c07) 2022-06-07 15:04:00 +01:00
Neil Alexander
6d4bd5d890
Rate limiting changes (#2519)
* Rate limiting changes

This makes the following changes:

* For logged in users, the rate limiting now applies to the device session rather than the remote IP address;
* For non-logged in users, the rate limiting continues to apply to remote address as it does today;
* It is now possible to add user IDs to the `exempt_user_ids` option under `rate_limiting` to exclude bots from rate limiting;
* Admin and appservice users are now exempt from rate limiting by default.

* Fix build with media API
2022-06-07 14:24:04 +01:00
Neil Alexander
27948fb304
Optimise loadAuthEvents, add roomserver tracing 2022-06-07 14:23:26 +01:00
Neil Alexander
aafb7bf120
Avoid panicking unnecessarily when shutting down the gobind P2P demos (#2520) 2022-06-07 10:46:21 +01:00
Neil Alexander
0d7020fbaf
Send tombstone to other servers when upgrading rooms 2022-06-06 17:27:50 +01:00