* a very very WIP first cut of peeking via MSC2753.
doesn't yet compile or work.
needs to actually add the peeking block into the sync response.
checking in now before it gets any bigger, and to gather any initial feedback on the vague shape of it.
* make PeekingDeviceSet private
* add server_name param
* blind stab at adding a `peek` section to /sync
* make it build
* make it launch
* add peeking to getResponseWithPDUsForCompleteSync
* cancel any peeks when we join a room
* spell out how to runoutside of docker if you want speed
* fix SQL
* remove unnecessary txn for SelectPeeks
* fix s/join/peek/ cargocult fail
* HACK: Track goroutine IDs to determine when we write by the wrong thread
To use: set `DENDRITE_TRACE_SQL=1` then grep for `unsafe`
* Track partition offsets and only log unsafe for non-selects
* Put redactions in the writer goroutine
* Update filters on writer goroutine
* wrap peek storage in goid hack
* use exclusive writer, and MarkPeeksAsOld more efficiently
* don't log ascii in binary at sql trace...
* strip out empty roomd deltas
* re-add txn to SelectPeeks
* re-add accidentally deleted field
* reject peeks for non-worldreadable rooms
* move perform_peek
* fix package
* correctly refactor perform_peek
* WIP of implementing MSC2444
* typo
* Revert "Merge branch 'kegan/HACK-goid-sqlite-db-is-locked' into matthew/peeking"
This reverts commit 3cebd8dbfbccdf82b7930b7b6eda92095ca6ef41, reversing
changes made to ed4b3a58a7855acc43530693cc855b439edf9c7c.
* (almost) make it build
* clean up bad merge
* support SendEventWithState with optional event
* fix build & lint
* fix build & lint
* reinstate federated peeks in the roomserver (doh)
* fix sql thinko
* todo for authenticating state returned by /peek
* support returning current state from QueryStateAndAuthChain
* handle SS /peek
* reimplement SS /peek to prod the RS to tell the FS about the peek
* rename RemotePeeks as OutboundPeeks
* rename remote_peeks_table as outbound_peeks_table
* add perform_handle_remote_peek.go
* flesh out federation doc
* add inbound peeks table and hook it up
* rename ambiguous RemotePeek as InboundPeek
* rename FSAPI's PerformPeek as PerformOutboundPeek
* setup inbound peeks db correctly
* fix api.SendEventWithState with no event
* track latestevent on /peek
* go fmt
* document the peek send stream race better
* fix SendEventWithRewrite not to bail if handed a non-state event
* add fixme
* switch SS /peek to use SendEventWithRewrite
* fix comment
* use reverse topo ordering to find latest extrem
* support postgres for federated peeking
* go fmt
* back out bogus go.mod change
* Fix performOutboundPeekUsingServer
* Fix getAuthChain -> GetAuthChain
* Fix build issues
* Fix build again
* Fix getAuthChain -> GetAuthChain
* Don't repeat outbound peeks for the same room ID to the same servers
* Fix lint
* Don't omitempty to appease sytest
Co-authored-by: Kegan Dougal <kegan@matrix.org>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Allow configuring DNS cache
* Update sample configs
* Fix build errors
* Fix time resolution
* Default 5m
* In seconds
* Use WithDNScache
* Correct field name
* Update go.mod/go.sum to matrix-org/gomatrixserverlib#251
* Add some filtering (postgres only for now)
* Fix build error
* Try to use request filter
* Use default filter as a template when retrieving from the database
* Remove unused strut
* Update sytest-whitelist
* Add filtering to SelectEarlyEvents
* Fix Postgres selectEarlyEvents query
* Attempt filtering on SQLite
* Test limit, set field for limit/order in prepareWithFilters
* Remove debug logging, add comments
* Tweaks, debug logging
* Separate SQLite stream IDs
* Fix filtering in current state table
* Fix lock issues
* More tweaks
* Current state requires room ID
* Review comments
* Add fedsender dep on msc2946
* Add MSC2946Spaces to fsAPI
* Add exclude_rooms impl
* Implement fed spaces handler
* Use stripped state not room version
* Call federated spaces at the right time
* Light-weight checking of state changes when updating forward extremities
* Only do this for non-state events, since state events will always result in state change at extremities
* Mention unstable features in /_matrix/client/versions
* list enabled mscs
* Don't update go.mod/go.sum
* goimports
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Use go build instead of go install
go install doesn't like to cross-compile things. (Try running build.sh with GOARCH set to something other than what it "should" be.)
With go build, it appears that cross-compilation is really, really straightforward. Simply install a compiler for your target platform and set `GOARCH` and `CC` accordingly.
* Use shell expansion instead of loop
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Fix#1511
On 32-bits systems, int(hash.Sum32()) can be negative.
This makes the computation of array indices using modulo invalid, crashing dendrite.
Signed-off-by: Loïck Bonniot <git@lesterpig.com>
Squashed commit of the following:
commit 4a61aa711473deece2adf415cfd65501dbca63b2
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Mon Jan 18 12:19:24 2021 +0000
Set back to matrixdotorg on published releases
commit 6d1ac53f2c0c9b30e1e70c0bb1559e1b8ec874a2
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Mon Jan 18 11:55:28 2021 +0000
Rename
commit 258999f7fb7b655b3a02a06a7ea05e66fb7740fb
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Mon Jan 18 11:52:26 2021 +0000
Refactor multi-stage builds
commit c7ab8e476939899571e7b5668860dec372b9b60f
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Mon Jan 18 11:13:19 2021 +0000
Let's try this again
* Create docker-build-and-push.yml
* Switched to using official Docker buildx action
* Added comment to docker-build-and-push.yml
In case something needs to be tweaked in the future, the link
contains some examples and explanations which would be useful
* Run only on release (and produce release tags)
As this workflow takes quite a lot of time, and [pushing to master happens frequently](https://github.com/matrix-org/dendrite/pull/1613#issuecomment-746086980), the container will now only be built when a release is created, and the builds will also be correctly tagged.
* Add latest tag, test at neilalexander/dendrite*
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Add madvdontneed=1 to decrease RAM consumption for systemd service
* Update monolith-example.service
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Add stub functions for MSC2946
* Implement core space walking algorithm
* Flesh out stub functions; add test stubs
* Implement storage bits and add sanity check test
* Implement world_readable auth with test
* Linting
Squashed commit of the following:
commit e5e2d793119733ecbcf9b85f966e018ab0318741
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Wed Jan 13 17:28:12 2021 +0000
Add dendrite_roomserver_processroomevent_duration_millis to prometheus
* Simplify send-to-device messaging
* Don't return error if there's no work to do
* Remove SQLite migrations for now
* Tweak Postgres migrations
* Tweaks
* Fixes
* Cleanup separately
* Fix SQLite migration
* Tweak pagination tokens
* start should be the specified from
* Don't reverse start and end
* Tweak getStartEnd again
* Update sytest-whitelist
* NOTSPEC: Re-add iOS end of topology
* It's half-alive
* Wakeups largely working
* Other tweaks, typing works
* Fix bugs, add receipt stream
* Delete notifier, other tweaks
* Dedupe a bit, add a template for the invite stream
* Clean up, add templates for other streams
* Don't leak channels
* Bring forward some more PDU logic, clean up other places
* Add some more wakeups
* Use addRoomDeltaToResponse
* Log tweaks, typing fixed?
* Fix timed out syncs
* Don't reset next batch position on timeout
* Add account data stream/position
* End of day
* Fix complete sync for receipt, typing
* Streams package
* Clean up a bit
* Complete sync send-to-device
* Don't drop errors
* More lightweight notifications
* Fix typing positions
* Don't advance position on remove again unless needed
* Device list updates
* Advance account data position
* Use limit for incremental sync
* Limit fixes, amongst other things
* Remove some fmt.Println
* Tweaks
* Re-add notifier
* Fix invite position
* Fixes
* Notify account data without advancing PDU position in notifier
* Apply account data position
* Get initial position for account data
* Fix position update
* Fix complete sync positions
* Review comments @Kegsay
* Room consumer parameters
The config section for MSC's was missing from the example config. Even though these are experimental features, it might be good to add the section into the example config for visibility.
Signed-off-by: Jason Robinson <mail@jasonrobinson.me>
* Ensure appservices can auth as users in their namespaces
Currently in Dendrite appservices can only auth as a user if the user was created by said appservice. This does not align with the appservices spec which specifically says:
> The application service may specify the virtual user to act as through use of a user_id query string parameter on the request. The user specified in the query string must be covered by one of the application service’s user namespaces.
https://matrix.org/docs/spec/application_service/r0.1.2#identity-assertion
In the case that a user has been created for example via manual registration but belongs to an appservice namespace, the current functionality does not allow appservices to auth as them. This PR fixes that by replacing the appservice ID check with a check against the appservice namespace.
This also matches Synapse functionality, which I confirmed to allow appservices to auth as a user in their namespace, irregardless of how the user was registered.
* Also allow appservice itself to auth with user_id
Appservice user_id + access token check needs to work both when user_id is the appservice and when appservice has the user in their user namespace.
Signed-off-by: Jason Robinson <mail@jasonrobinson.me>
* Do not check if a username is exclusive if the request is for an appservice in /register
* remove useless comment
* Move statements
* fix broken test
* Also fix the senderLocalpart problem
* Check domain name is ours
* Handle accessTokenErr
* Return unauthorised instead of forbidden
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
