Commit Graph

2665 Commits

Author SHA1 Message Date
Neil Alexander
3afb161352
Reduce memory usage in federation /send endpoint (#1890)
* More aggressive event caching

* Deduplicate /state results

* Deduplicate more

* Ensure we use the correct list of events when excluding repeated state

* Fixes

* Ensure we track all events we already knew about properly
2021-06-30 10:01:56 +01:00
kegsay
c849e74dfc
db migration: fix #1844 and add additional assertions (#1889)
* db migration: fix #1844 and add additional assertions

- Migration scripts will now check to see if there are any unconverted
  snapshot IDs and fail the migration if there are any. This should
  prevent people from getting a corrupt database in the event the root
  cause is still unknown.
- Add an ORDER BY clause when doing batch queries in the postgres
  migration. LIMIT and OFFSET without ORDER BY are undefined and must
  not be relied upon to produce a deterministic ordering (e.g row order).
  See https://www.postgresql.org/docs/current/queries-limit.html

* Linting

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-06-29 11:25:17 +01:00
Neil Alexander
e2b6a90d90
Put gmectx back to 5 minutes 2021-06-29 10:22:26 +01:00
Neil Alexander
f645646ca9
Restore the getServers RS query (needs optimisation) 2021-06-29 09:37:28 +01:00
Neil Alexander
4417f24678
Protect processEventWithMissingState with per-room mutex, to prevent mass CPU burn/RAM usage
Squashed commit of the following:

commit 7fad77c10e3c1c78feddb37351812b209d9c0f25
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 15:06:52 2021 +0100

    Fix processEventWithMissingStateMutexes

commit 138cddcac7b8373a8e1816a232f84a7bda6adcdf
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 13:59:44 2021 +0100

    Use internal.MutexByRoom

commit 6e6f026cfad31da391ad261cfec16d41dff1b15b
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 13:50:18 2021 +0100

    Try to slow things down per room

commit b97d406dff2e11769a9202fbf58b138a541ca449
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 13:41:27 2021 +0100

    Try to slow things down

commit 8866120ebf880b4fd8a456937f69903e233c19a2
Merge: 9f2de8a2 4a37b19a
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 13:40:33 2021 +0100

    Merge branch 'neilalexander/rsinputfifo' into neilalexander/rsinputfifo2

commit 4a37b19a8f6fe8af02e979827253d22a0ccdedb8
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 13:34:54 2021 +0100

    Add comments

commit f9ab3f4b8157a42d657735101bc2c768c663e814
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 13:31:21 2021 +0100

    Tweaks

commit 9f2de8a29cadec4c785d9c2e4e74c1138305f759
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 13:15:59 2021 +0100

    Ask origin only for missing things for now

commit 8fd878c75a4066abb21597d524a4eb4670a392d4
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 11:18:11 2021 +0100

    Make sure someone wakes up

commit b63f699f1b74948d180885449398f999fafb18c8
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jun 28 11:12:58 2021 +0100

    Use a FIFO queue instead of a channel to reduce backpressure
2021-06-28 15:11:59 +01:00
Neil Alexander
7c3991ee2f
Use a custom FIFO queue for the RS input API (#1888)
* Use a FIFO queue instead of a channel to reduce backpressure

* Make sure someone wakes up

* Tweaks

* Add comments
2021-06-28 15:11:36 +01:00
Neil Alexander
a6f7e83596
Fix bugs in P2P demos 2021-06-25 15:59:59 +01:00
Neil Alexander
bd7b0b1787
Update go.mod/go.sum 2021-06-25 15:13:21 +01:00
Neil Alexander
097a5833c8
dendritejs-pinecone 2021-06-25 14:20:36 +01:00
Neil Alexander
6ae976a55a
dendrite-demo-pinecone: Accept any origin 2021-06-22 15:39:29 +01:00
database64128
79181f2348
✂️ Media API: Handle unlimited file size (#1881) 2021-06-22 10:01:44 +01:00
Neil Alexander
5357df36c9
Fix panic in roomserver 2021-06-21 09:41:12 +01:00
Neil Alexander
3a77bca28b
Revert "Ensure user IDs match the spec (matrix-org/gomatrixserverlib#261)"
This reverts commit 30e9353fdd.
2021-06-16 09:07:41 +01:00
Neil Alexander
30e9353fdd
Ensure user IDs match the spec (matrix-org/gomatrixserverlib#261) 2021-06-15 15:18:56 +01:00
Neil Alexander
5ec7d3a3ec
Update go.mod/go.sum 2021-06-14 15:04:05 +01:00
S7evinK
9ed0440421
Set MaxFileSizeBytes <= 0 to "unlimited" (#1875)
* Set MaxFileSizeBytes < 0 to "unlimited"

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* int64 overflows later in mediaapi/routing/upload.go[doUpload]

* Prevent int overflow when uploading
2021-06-14 14:12:03 +01:00
Arnaud Venturi
9633ed747d
Remove mention of enabling naffka in install instructions, as it is the default value (#1853)
Signed-off-by: Arnaud Venturi <git@toadjaune.eu>
2021-06-14 14:09:35 +01:00
Ben Yanke
ce574f5e29
Fix Key Generation Docs - resolves #1759 (#1865)
This fixes the issue found in #1759 which broke due to go changes. The new command allows you to generate keys with docker, and drop them in the current working directory.
2021-06-14 14:08:59 +01:00
S7evinK
183f43d38d
Add parameters to specify password (#1868)
* Add parameters to specify password

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Fix typo

* Add testdata

* Use go1.13 compatible way to read files
2021-06-14 14:08:29 +01:00
Adam Greig
eae1c46310
Use NotFound instead of Forbidden for missing account data (#1872)
Signed-off-by: Adam Greig <adam@adamgreig.com>
2021-06-14 14:06:14 +01:00
Neil Alexander
051cd63f1e
Update go.mod/go.sum 2021-06-14 13:26:02 +01:00
Neil Alexander
bd9dec8e06
Pinecone demo updates 2021-06-14 13:13:07 +01:00
Rasmus Thomsen
2c9a390fa6
Add CORS to nginx config (#1791)
Without this entry, setups where users have the homeserver on the URL
matrix.myurl.com but want the servername to be myurl.com don't work by default
since clients like element.io can't connect to the homeserver
2021-06-07 11:06:57 +01:00
Ben Langfeld
caa5c89ecb
Link to Synapse (#1863)
Not everyone is very familiar.
2021-06-07 10:10:29 +01:00
Arnaud Venturi
6fa5743ea0
Add missing IPv6 "ssl" keyword in nginx config examples (#1854)
Signed-off-by: Arnaud Venturi <git@toadjaune.eu>
2021-06-07 09:29:54 +01:00
S7evinK
89a6787fdb
Try to optimize SelectOneTimeKeys (#1851)
* Try to optimize SelectOneTimeKeys

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Use pg.Array when using ANY...

Co-authored-by: Kegsay <kegan@matrix.org>
2021-06-07 09:17:46 +01:00
S7evinK
8b22c4270d
Use LimitReader to prevent DoS risk (#1843)
* Use LimitReader to prevent DoS risk

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Check if bytesWritten is equal to the maxFileSize
Add tests

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Use oldschool defer to cleanup after the tests

* Let LimitReader read MaxFileSizeBytes + 1

Co-authored-by: Kegsay <kegan@matrix.org>
2021-06-07 09:17:20 +01:00
bodqhrohro
c488d3db75
Fix SIGSEGV in IsInterestedInRoomID (#1846)
* Avoid crash on non-compiled room regex

Roughly fixes #1845 (actual compiling still needed)

Signed-off-by: Bohdan Horbeshko <bodqhrohro@gmail.com>

* Compile regexes for all namespaces

Deadheres the regex compiling from building larger regexes for possibly
exclusive namespaces only. A complete fix for #1845, so regexes for
rooms namespaces and other non-whitelisted namespaces can be used
more safely.

Signed-off-by: Bohdan Horbeshko <bodqhrohro@gmail.com>

* Appservice config: handle regexp parsing errors

Signed-off-by: diamondburned <datutbrus@gmail.com>
Signed-off-by: Bohdan Horbeshko <bodqhrohro@gmail.com>

Co-authored-by: Kegsay <kegan@matrix.org>
2021-06-07 09:13:40 +01:00
bodqhrohro
f18001ecbb
Split the select+update query for txn_id counter (#1855)
The update part wasn't executed actually for SQLite, so it is moved to
a separate statement.

Fixes #1852.

Signed-off-by: Bohdan Horbeshko <bodqhrohro@gmail.com>
2021-06-07 08:51:19 +01:00
Neil Alexander
b0aa101dcd
Update go.mod/go.sum for matrix-org/pinecone (Build 79) 2021-06-02 12:23:01 +01:00
Neil Alexander
3797d818c0
Update gomatrixserverlib to matrix-org/gomatrixserverlib#259 2021-05-25 12:01:07 +01:00
Neil Alexander
3f84cd6cd0
Update go.mod/go.sum 2021-05-25 09:52:55 +01:00
Neil Alexander
30f021700a
Update go.mod/go.sum 2021-05-25 09:49:02 +01:00
Neil Alexander
a7f2845a6a
Demo tweaks 2021-05-24 13:12:05 +01:00
Neil Alexander
79c5485c8d
Allow clearing federation blacklist at startup for P2P demos 2021-05-24 11:43:24 +01:00
Neil Alexander
d2d2164025
Update pinecone demo 2021-05-24 11:17:32 +01:00
Neil Alexander
2948ffd782
Update go.mod/go.sum 2021-05-18 16:20:04 +01:00
Neil Alexander
08a4370c94
Update go.mod/go.sum 2021-05-18 15:58:50 +01:00
Neil Alexander
e9e0309089
Merge branch 'master' of github.com:matrix-org/dendrite 2021-05-18 15:47:23 +01:00
Neil Alexander
140cae81cc
Update dendrite-demo-pinecone 2021-05-18 15:47:15 +01:00
Michael Telatynski
4691adc8f8
Update MSC2946 implementation for stable spaces (#1859)
Now that MSC1772 passed FCP its identifiers have stabilised
This outright drops support for experimental spaces but that's what you get for being on the bleeding edge
2021-05-18 15:17:37 +01:00
Neil Alexander
1d89c4a3cd
Update go.mod/go.sum 2021-05-10 17:04:06 +01:00
Neil Alexander
c76f820c86
Update go.mod/go.sum 2021-05-10 17:03:02 +01:00
Neil Alexander
a49d06138e
Updates to dendrite-demo-pinecone 2021-05-10 16:59:03 +01:00
Neil Alexander
81d60d5448
Update room directory in Pinecone demo some more 2021-05-07 12:48:30 +01:00
Neil Alexander
603bf590f0
Fix public room directory in Pinecone demo 2021-05-07 12:17:14 +01:00
Neil Alexander
aa672068ab
Update pinecone in go.mod/go.sum 2021-05-07 11:49:44 +01:00
Neil Alexander
1002e87b60
Pinecone P2P demo (#1856)
* Pinecone demo

* Enable multicast, fix HTTP routing

* Fix multicast import

* Fix build

* Update Pinecone demo

* Fix the keys

* Tweaks

* Pinecone room directory support (early)

* Fix gobind-pinecone

* Add pinecone listener

* Fix public key value

* Use AuthenticatedConnect for dial

* Fix gobind-pinecone

* Stop panics

* Give fsAPI to keyserver

* Pinecone demo fixes

* Update gobind build scripts

* Account creation

* Tweaks

* Setup tweaks

* API tweaks

* API tweaks

* API tweaks

* Port mutex

* Re-enable multicast

* Add ReadCopy

* Update quic-go, fixes

* Shutdowns fixed for iOS

* Update build script

* Add WebSocket support

* Bug fixes

* Netconn context

* Fix WebSocket connectivity

* Fixes to gobind API

* Strip frameworks

* Configurability updates

* Update go.mod

* Update go.mod/go.sum

* Update go.mod/go.sum

* Update go.mod/go.sum

* Try to stay connected tto static peer

* Update gobind-pinecone

* Update go.mod/go.sum

* Test uTP+TLS

* Use HTTP/2

* Don't use HTTP/2

* Update go.mod/go.sum

* Attempt to reconnect to the static peer if it drops

* Stay connected to static peers more stickily

* Retry room directory lookups if they fail

* NewQUIC -> NewSessions

* Storage updates

* Don't return immediately when there's nothing to sync

* Updates

* Try to reconnect to static peer more

* Update go.mod/go.sum

* Require Go 1.14

* Update go.mod/go.sum

* Update go.mod/go.sum
2021-05-06 12:00:42 +01:00
Neil Alexander
464b908bd0
Don't return immediately when there's nothing to sync 2021-04-26 16:33:42 +01:00
Neil Alexander
c67d8da3eb
Fix bug in SQLite migration 2021-04-26 13:45:47 +01:00