Commit Graph

1634 Commits

Author SHA1 Message Date
Neil Alexander
25dc99f1b0
Upgrade dependencies (#1779) 2021-03-03 11:08:41 +00:00
Kegsay
586cc5be5e
Use new path for MSC2946 (#1778)
* Use new path for MSC2946

* Update GMSL
2021-03-02 16:41:28 +00:00
Neil Alexander
6a35d9f1b1
Version 0.3.11 2021-03-02 11:46:35 +00:00
Neil Alexander
81312b8a78
Return the current OTK count on an empty upload request (#1774)
* Always return OTK counts

* Fix parameter ordering

* Send IDs over to keyserver internal API

* Review comments

* Fix syntax error

* Fix panic, hopefully

* Require user ID to be set

* Fix user API call
2021-03-02 11:40:20 +00:00
Neil Alexander
f5cf241877
Fix user registration bug (#1777) 2021-03-02 10:43:25 +00:00
Neil Alexander
3069079e37
Revert quic-go dependency update 2021-02-18 10:18:14 +00:00
Neil Alexander
db637515a5
Update libp2p dependencies 2021-02-18 10:14:24 +00:00
Neil Alexander
11830fd4af
Fix transitive dependency for P2P demo 2021-02-18 09:45:55 +00:00
Neil Alexander
4c0103a2d5
Don't close channels when clearing queue (we might race and panic, when the GC will still clean it up for us anyway) 2021-02-18 09:26:40 +00:00
Neil Alexander
2fdc318f2c
Version 0.3.10 2021-02-17 16:05:04 +00:00
Kegan Dougal
bf9c530fdb Unbreak DendriteJS.Dockerfile; Riot is now Element 2021-02-17 15:45:50 +00:00
Marco Kundt
d1496793b9
fix database names to reflect renaming (#1636)
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-02-17 15:20:06 +00:00
Neil Alexander
c9f305f254
Don't exclude an event from sync if it was previously not excluded (#1767) 2021-02-17 15:18:41 +00:00
Neil Alexander
8b5cd256cb
Don't hold destination queues in memory forever (#1769)
* Don't hold destination queues in memory forever

* Close channels

* Fix ordering

* Clear more aggressively

* clearQueue only called by defer so should be safe to delete queue in any case

* Wake queue when created, otherwise cleanup doesn't get called in all cases

* Clean up periodically, we hit a race condition otherwise

* Tweaks

* Don't create queues for blacklisted hosts

* Check blacklist properly
2021-02-17 15:16:35 +00:00
darkgallium
da797c7998
Retrieve remote file size without the Content-Size header (#1537)
* Get downloaded file size regardless of the Content-Size header

Signed-off-by: Florian Le Minoux <darkgallium+git@deuxfleurs.fr>

* fixing lint issues

* Don't exhaust memory for large files, don't limit more than necessary

* Don't use errors.Wrap in download.go

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-02-17 13:54:53 +00:00
Neil Alexander
2386e0c7af
Gradually evict oldest cache entries (#1768)
* Gradually evict oldest cache entries

* Keep the remaining 10% of cached entries
2021-02-17 13:50:27 +00:00
Neil Alexander
5d74a1757f
Don't query for servers so often in /send (#1766)
* Look up servers less often, don't hit API for missing auth events unless there are actually missing auth events

* Remove ResolveConflictsAdhoc (since it is already in GMSL), other tweaks

* Update gomatrixserverlib to matrix-org/gomatrixserverlib#254

* Fix resolve-state

* Initialise t.servers on first use
2021-02-16 17:12:17 +00:00
Michael Telatynski
f448e8972a
Implement /createRoom power_level_content_override (#1761)
* Implement /createRoom power_level_content_override

* delint

* Skip having to re-marshal PowerLevelContentOverride by making use of `json.RawMessage`
2021-02-12 09:43:31 +00:00
David Florness
85aaaf9bb9
Fix a few indentation mistakes (#1752)
Signed-off-by: David Florness <david@florness.com>
2021-02-12 09:25:35 +00:00
Kegan Dougal
9a199ba179 Remove 100 default 2021-02-06 17:05:00 +00:00
Kegan Dougal
82df194858 Increase limit 2021-02-06 16:56:55 +00:00
Neil Alexander
02e6d89cc2
Fix crash in membership updater (#1753)
* Fix nil pointer exception in membership updater

* goimports
2021-02-06 11:49:18 +00:00
Neil Alexander
397b158c05
Version 0.3.9 2021-02-04 12:39:43 +00:00
Neil Alexander
bd72ed50d4
Reduce log level of 'Failed to send transaction' log line, since quite often it is flooding logs for dead servers 2021-02-04 12:25:31 +00:00
Neil Alexander
6e44450cc9
Don't re-request state events that are already in the timeline (#1739)
* Don't request state events if we already have the timeline events (Postgres only)

* Rename variable

* nocyclo

* Add SQLite

* Tweaks

* Revert query change

* Don't dedupe if asking for full state

* Update query
2021-02-04 12:20:37 +00:00
Neil Alexander
6099379ea4
Remove rooms table from federation sender (#1751)
* Remove last sent event ID column from federation sender

* Remove EventIDMismatchError

* Remove the federationsender rooms table altogether, it's useless

* Add migration

* Fix migrations

* Fix migrations
2021-02-04 11:52:49 +00:00
Neil Alexander
b7e3b81a22
Fix ON CONFLICT on sync API account data (#1745) (#1750) 2021-02-04 11:45:49 +00:00
Neil Alexander
de5f22a469
Remove redundant check (#1748) 2021-02-04 11:12:52 +00:00
Neil Alexander
369d3939fd
Drop state events we can't auth instead of failing altogether 2021-01-29 16:33:59 +00:00
Neil Alexander
62a325ded8
Complete sync performance (#1741)
* Parallelise PDU stream fetching for complete sync

* Fixes

* Fixes

* Worker queue

* Workers

* Don't populate device list changes on complete sync

* Don't fast-forward typing notifications either on complete sync

* Revert "Don't fast-forward typing notifications either on complete sync"
This reverts commit 01471f78431cdd840915111f71bd2b5176e584a8.

* Comments
2021-01-29 16:32:54 +00:00
Kegsay
6d1c6f29e0
Add m.room.create to invite stripped state (#1740)
MSC1772 needs this because the create event contains info on if
the room is a space or not. The create event itself isn't sensitive
so other people may find this useful too.
2021-01-29 11:36:26 +00:00
Neil Alexander
61090cbe35
Version 0.3.8 2021-01-28 13:37:50 +00:00
Neil Alexander
d56a5a6509
Fix well-known lookups 2021-01-28 13:25:45 +00:00
Neil Alexander
e14e9a1606
Version 0.3.7 2021-01-26 13:13:48 +00:00
Neil Alexander
9f443317bc
Graceful shutdowns (#1734)
* Initial graceful stop

* Fix dendritejs

* Use process context for outbound federation requests in destination queues

* Reduce logging

* Fix log level
2021-01-26 12:56:20 +00:00
Neil Alexander
64fb6de6d4
Don't retrieve same state events over and over again (#1737) 2021-01-26 09:12:17 +00:00
Kegsay
ef9d5ad4fe
Check peek state response and refactor checking send_join response (#1732) 2021-01-22 17:16:35 +00:00
Neil Alexander
6757b67a32
NewClient and NewFederationClient updates (#1730)
* Use matrix-org/gomatrixserverlib#252

* Add missing WithSkipVerify to test

* Functions instead

* Update gomatrixserverlib to matrix-org/gomatrixserverlib#252

* Fix disabling TLS validation
2021-01-22 16:09:05 +00:00
Kegsay
93942f8ab6
Gate peeking behind msc flags (#1731) 2021-01-22 16:08:47 +00:00
Matthew Hodgson
0571d395b5
Peeking over federation via MSC2444 (#1391)
* a very very WIP first cut of peeking via MSC2753.

doesn't yet compile or work.
needs to actually add the peeking block into the sync response.
checking in now before it gets any bigger, and to gather any initial feedback on the vague shape of it.

* make PeekingDeviceSet private

* add server_name param

* blind stab at adding a `peek` section to /sync

* make it build

* make it launch

* add peeking to getResponseWithPDUsForCompleteSync

* cancel any peeks when we join a room

* spell out how to runoutside of docker if you want speed

* fix SQL

* remove unnecessary txn for SelectPeeks

* fix s/join/peek/ cargocult fail

* HACK: Track goroutine IDs to determine when we write by the wrong thread

To use: set `DENDRITE_TRACE_SQL=1` then grep for `unsafe`

* Track partition offsets and only log unsafe for non-selects

* Put redactions in the writer goroutine

* Update filters on writer goroutine

* wrap peek storage in goid hack

* use exclusive writer, and MarkPeeksAsOld more efficiently

* don't log ascii in binary at sql trace...

* strip out empty roomd deltas

* re-add txn to SelectPeeks

* re-add accidentally deleted field

* reject peeks for non-worldreadable rooms

* move perform_peek

* fix package

* correctly refactor perform_peek

* WIP of implementing MSC2444

* typo

* Revert "Merge branch 'kegan/HACK-goid-sqlite-db-is-locked' into matthew/peeking"

This reverts commit 3cebd8dbfbccdf82b7930b7b6eda92095ca6ef41, reversing
changes made to ed4b3a58a7855acc43530693cc855b439edf9c7c.

* (almost) make it build

* clean up bad merge

* support SendEventWithState with optional event

* fix build & lint

* fix build & lint

* reinstate federated peeks in the roomserver (doh)

* fix sql thinko

* todo for authenticating state returned by /peek

* support returning current state from QueryStateAndAuthChain

* handle SS /peek

* reimplement SS /peek to prod the RS to tell the FS about the peek

* rename RemotePeeks as OutboundPeeks

* rename remote_peeks_table as outbound_peeks_table

* add perform_handle_remote_peek.go

* flesh out federation doc

* add inbound peeks table and hook it up

* rename ambiguous RemotePeek as InboundPeek

* rename FSAPI's PerformPeek as PerformOutboundPeek

* setup inbound peeks db correctly

* fix api.SendEventWithState with no event

* track latestevent on /peek

* go fmt

* document the peek send stream race better

* fix SendEventWithRewrite not to bail if handed a non-state event

* add fixme

* switch SS /peek to use SendEventWithRewrite

* fix comment

* use reverse topo ordering to find latest extrem

* support postgres for federated peeking

* go fmt

* back out bogus go.mod change

* Fix performOutboundPeekUsingServer

* Fix getAuthChain -> GetAuthChain

* Fix build issues

* Fix build again

* Fix getAuthChain -> GetAuthChain

* Don't repeat outbound peeks for the same room ID to the same servers

* Fix lint

* Don't omitempty to appease sytest

Co-authored-by: Kegan Dougal <kegan@matrix.org>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-01-22 14:55:08 +00:00
Neil Alexander
5d8ec0ff1a
Fix build error 2021-01-22 14:18:36 +00:00
Neil Alexander
805a74892e
DNS caching (#1728)
* Allow configuring DNS cache

* Update sample configs

* Fix build errors

* Fix time resolution

* Default 5m

* In seconds

* Use WithDNScache

* Correct field name

* Update go.mod/go.sum to matrix-org/gomatrixserverlib#251
2021-01-22 14:16:59 +00:00
Neil Alexander
745ee20b90
Update sytest-whitelist after matrix-org/sytest#991 2021-01-22 12:56:20 +00:00
Neil Alexander
58f78e08af
Limit well-known response sizes (matrix-org/gomatrixserverlib#250) 2021-01-21 13:57:41 +00:00
Neil Alexander
8fe51019ad
Add sync API memberships table (#1726) 2021-01-20 20:43:20 +00:00
Kegsay
c08e38df2c
MSC2946: Treat federation responses the same way as local responses (#1724)
* Start treating fed rooms/events the same as local rooms/events

* Share more code
2021-01-20 17:03:35 +00:00
Neil Alexander
b70238f2d5
Basic sync filtering (#1721)
* Add some filtering (postgres only for now)

* Fix build error

* Try to use request filter

* Use default filter as a template when retrieving from the database

* Remove unused strut

* Update sytest-whitelist

* Add filtering to SelectEarlyEvents

* Fix Postgres selectEarlyEvents query

* Attempt filtering on SQLite

* Test limit, set field for limit/order in prepareWithFilters

* Remove debug logging, add comments

* Tweaks, debug logging

* Separate SQLite stream IDs

* Fix filtering in current state table

* Fix lock issues

* More tweaks

* Current state requires room ID

* Review comments
2021-01-19 18:00:42 +00:00
Kegsay
80aa9aa8b0
Implement MSC2946 over federation (#1722)
* Add fedsender dep on msc2946

* Add MSC2946Spaces to fsAPI

* Add exclude_rooms impl

* Implement fed spaces handler

* Use stripped state not room version

* Call federated spaces at the right time
2021-01-19 17:14:25 +00:00
Neil Alexander
ccfcb2d280
Create bin dir 2021-01-18 15:09:54 +00:00
Neil Alexander
f6151b6290
Use dendritegithub for Docker Hub images 2021-01-18 14:00:47 +00:00