* Add Pushserver component with Pushers API
Co-authored-by: Tommie Gannert <tommie@gannert.se>
Co-authored-by: Dan Peleg <dan@globekeeper.com>
* Wire Pushserver component
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Add PushGatewayClient.
The full event format is required for Sytest.
* Add a pushrules module.
* Change user API account creation to use the new pushrules module's defaults.
Introduces "scope" as required by client API, and some small field
tweaks to make some 61push Sytests pass.
* Add push rules query/put API in Pushserver.
This manipulates account data over User API, and fires sync messages
for changes. Those sync messages should, according to an existing TODO
in clientapi, be moved to userapi.
Forks clientapi/producers/syncapi.go to pushserver/ for later extension.
* Add clientapi routes for push rules to Pushserver.
A cleanup would be to move more of the name-splitting logic into
pushrules.go, to depollute routing.go.
* Output rooms.join.unread_notifications in /sync.
This is the read-side. Pushserver will be the write-side.
* Implement pushserver/storage for notifications.
* Use PushGatewayClient and the pushrules module in Pushserver's room consumer.
* Use one goroutine per user to avoid locking up the entire server for
one bad push gateway.
* Split pushing by format.
* Send one device per push. Sytest does not support coalescing
multiple devices into one push. Matches Synapse. Either we change
Sytest, or remove the group-by-url-and-format logic.
* Write OutputNotificationData from push server. Sync API is already
the consumer.
* Implement read receipt consumers in Pushserver.
Supports m.read and m.fully_read receipts.
* Add clientapi route for /unstable/notifications.
* Rename to UpsertPusher for clarity and handle pusher update
* Fix linter errors
* Ignore body.Close() error check
* Fix push server internal http wiring
* Add 40 newly passing 61push tests to whitelist
* Add next 12 newly passing 61push tests to whitelist
* Send notification data before notifying users in EDU server consumer
* NATS JetStream
* Goodbye sarama
* Fix `NewStreamTokenFromString`
* Consume on the correct topic for the roomserver
* Don't panic, NAK instead
* Move push notifications into the User API
* Don't set null values since that apparently causes Element upsetti
* Also set omitempty on conditions
* Fix bug so that we don't override the push rules unnecessarily
* Tweak defaults
* Update defaults
* More tweaks
* Move `/notifications` onto `r0`/`v3` mux
* User API will consume events and read/fully read markers from the sync API with stream positions, instead of consuming directly
Co-authored-by: Piotr Kozimor <p1996k@gmail.com>
Co-authored-by: Tommie Gannert <tommie@gannert.se>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Add server_notices config
* Disallow rejecting "server notice" invites
* Update config
* Slightly refactor sendEvent and CreateRoom so it can be reused
* Implement unspecced server notices
* Validate the request
* Set the user api when starting
* Rename function/variables
* Update comments
* Update config
* Set the avatar on account creation
* Update test
* Only create the account when starting
Only add routes if sever notices are enabled
* Use reserver username
Check that we actually got roomData
* Add check for admin account
Enable server notices for CI
Return same values as Synapse
* Add custom error for rejecting server notice invite
* Move building an invite to it's own function, for reusability
* Don't create new rooms, use the existing one (follow Synapse behavior)
Co-authored-by: kegsay <kegan@matrix.org>
* Merge user API databases into one
* Remove DeviceDatabase from config
* Fix tests
* Try that again
* Clean up keyserver device keys when the devices no longer exist in the user API
* Tweak ordering
* Fix UserExists flag, device check
* Allow including empty entries so we can clean them up
* Remove logging
* Listen for /v3 on CSAPI
* Docs
* More docs
* Rename path variable to fix key backup tests
* Update routing.go
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Add GOPATH to PATH in find-lint.sh.
The user doesn't necessarily have it in PATH.
* Refactor LoginTypePassword and Type to support m.login.token and m.login.sso.
For login token:
* m.login.token will require deleting the token after completeAuth has
generated an access token, so a cleanup function is returned by
Type.Login.
* Allowing different login types will require parsing the /login body
twice: first to extract the "type" and then the type-specific parsing.
Thus, we will have to buffer the request JSON in /login, like
UserInteractive already does.
For SSO:
* NewUserInteractive will have to also use GetAccountByLocalpart. It
makes more sense to just pass a (narrowed-down) accountDB interface
to it than adding more function pointers.
Code quality:
* Passing around (and down-casting) interface{} for login request types
has drawbacks in terms of type-safety, and no inherent benefits. We
always decode JSON anyway. Hence renaming to Type.LoginFromJSON. Code
that directly uses LoginTypePassword with parsed data can still use
Login.
* Removed a TODO for SSO. This is already tracked in #1297.
* httputil.UnmarshalJSON is useful because it returns a JSONResponse.
This change is intended to have no functional changes.
* Support login tokens in User API.
This adds full lifecycle functions for login tokens: create, query, delete.
* Support m.login.token in /login.
* Fixes for PR review.
* Set @matrix-org/dendrite-core as repository code owner
* Return event NID from `StoreEvent`, match PSQL vs SQLite behaviour, tweak backfill persistence (#2071)
Co-authored-by: kegsay <kegan@matrix.org>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Initial federation sender -> federation API refactoring
* Move base into own package, avoids import cycle
* Fix build errors
* Fix tests
* Add signing key server tables
* Try to fold signing key server into federation API
* Fix dendritejs builds
* Update embedded interfaces
* Fix panic, fix lint error
* Update configs, docker
* Rename some things
* Reuse same keyring on the implementing side
* Fix federation tests, `NewBaseDendrite` can accept freeform options
* Fix build
* Update create_db, configs
* Name tables back
* Don't rename federationsender consumer for now
* Cross-signing groundwork
* Update to matrix-org/gomatrixserverlib#274
* Fix gobind builds, which stops unit tests in CI from yelling
* Some changes from review comments
* Fix build by passing in UIA
* Update to matrix-org/gomatrixserverlib@bec8d22
* Process master/self-signing keys from devices call
* nolint
* Enum-ify the key type in the database
* Process self-signing key too
* Fix sanity check in device list updater
* Fix check
* Fix sytest, hopefully
* Fix build
* Default /unstable requests to stable endpoints if not overridden specifically with a custom route
* Rewrite URL
* Try something different
* Fix routing manually
* Fix selectLatestVersionSQL
* Don't return 0 if no backup version exists
* Log more useful error
* fix up replace keys check
* Don't enforce uniqueness on e2e_room_keys_versions_idx
Co-authored-by: kegsay <kegan@matrix.org>
* Add PUT key backup endpoints and glue them to PerformKeyBackup
* Add tables for storing backup keys and glue them into the user API
* Don't create tables whilst still WIPing
* writer on sqlite please
* Linting
* Generate m.room.canonical_alias instead of legacy m.room.aliases
* Add omitempty tags
* Add aliases endpoint to client API
* Check power levels when setting aliases
* Don't return null on /aliases
* Don't return error if the state event fails
* Update sytest-whitelist
* Don't send updated m.room.canonical_alias events
* Don't check PLs after all because for local aliases they are apparently irrelevant
* Fix some bugs
* Allow deleting a local alias with enough PL
* Fix some more bugs
* Update sytest-whitelist
* Fix copyright notices
* Review comments
* Implement /_synapse/admin/v1/register
This is implemented identically to Synapse, so scripts which work
with Synapse should work with Dendrite.
```
Test 27 POST /_synapse/admin/v1/register with shared secret... OK
Test 28 POST /_synapse/admin/v1/register admin with shared secret... OK
Test 29 POST /_synapse/admin/v1/register with shared secret downcases capitals... OK
Test 30 POST /_synapse/admin/v1/register with shared secret disallows symbols... OK
```
Sytest however has `implementation_specific => "synapse"` which stops these
tests from running.
* Add missing muxes to gobind
* Linting
* Implement OpenID module (#599)
- Unrelated: change Riot references to Element in client API routing
Signed-off-by: Bruce MacDonald <contact@bruce-macdonald.com>
* OpenID module tweaks (#599)
- specify expiry is ms rather than vague ts
- add OpenID token lifetime to configuration
- use Go naming conventions for the path params
- store plaintext token rather than hash
- remove openid table sqllite mutex
* Add default OpenID token lifetime (#599)
* Update dendrite-config.yaml
Co-authored-by: Kegsay <kegsay@gmail.com>
Co-authored-by: Kegsay <kegan@matrix.org>
* Mention unstable features in /_matrix/client/versions
* list enabled mscs
* Don't update go.mod/go.sum
* goimports
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* fix conversion from int to string yields a string of one rune, not a string of digits
* Add receipts table to syncapi
* Use StreamingToken as the since value
* Add required method to testEDUProducer
* Make receipt json creation "easier" to read
* Add receipts api to the eduserver
* Add receipts endpoint
* Add eduserver kafka consumer
* Add missing kafka config
* Add passing tests to whitelist
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
* Fix copy & paste error
* Fix column count error
* Make outbound federation receipts pass
* Make "Inbound federation rejects receipts from wrong remote" pass
* Don't use errors package
* - Add TODO for batching requests
- Rename variable
* Return a better error message
* - Use OutputReceiptEvent instead of InputReceiptEvent as result
- Don't use the errors package for errors
- Defer CloseAndLogIfError to close rows
- Fix Copyright
* Better creation/usage of JoinResponse
* Query all joined rooms instead of just one
* Update gomatrixserverlib
* Add sqlite3 migration
* Add postgres migration
* Ensure required sequence exists before running migrations
* Clarification on comment
* - Fix a bug when creating client receipts
- Use concrete types instead of interface{}
* Remove dead code
Use key for timestamp
* Fix postgres query...
* Remove single purpose struct
* Use key/value directly
* Only apply receipts on initial sync or if edu positions differ,
otherwise we'll be sending the same receipts over and over again.
* Actually update the id, so it is correctly send in syncs
* Set receipt on request to /read_markers
* Fix issue with receipts getting overwritten
* Use fmt.Errorf instead of pkg/errors
* Revert "Add postgres migration"
This reverts commit 722fe5a04628882b787d096942459961db159b06.
* Revert "Add sqlite3 migration"
This reverts commit d113b03f6495a4b8f8bcf158a3d00b510b4240cc.
* Fix selectRoomReceipts query
* Make golangci-lint happy
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Add basic storage methods
* Add internal api handler
* Add check for forgotten room
* Add /rooms/{roomID}/forget endpoint
* Add missing rsAPI method
* Remove unused parameters
* Add passing tests
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
* Add missing file
* Add postgres migration
* Add sqlite migration
* Use Forgetter to forget room
* Remove empty line
* Update HTTP status codes
It looks like the spec calls for these to be 400, rather than 403: https://matrix.org/docs/spec/client_server/r0.6.1#post-matrix-client-r0-rooms-roomid-forget
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Remove QueryBulkStateContent from current state server
Expected fail due to db impl not existing
* Implement query bulk state content
* Fix up rejecting invites over federation
* Fix bulk content marshalling
* User API support for password changes
* Password changes in client API
* Update sytest-whitelist
* Remove debug logging
* Default logout_devices to true
* Fix deleting devices by local part
* Enforce history visibility etc for /rooms/{roomID}/state
* Deduplicate OnIncomingStateRequest and OnIncomingStateTypeRequest
* Revert "Deduplicate OnIncomingStateRequest and OnIncomingStateTypeRequest"
This reverts commit 335035d66e629022232abc682d6631e3cf669e23.
* Initial pass at refactoring config (not finished)
* Don't forget current state and EDU servers
* More shifting around
* Update server key API tests
* Fix roomserver test
* Fix more tests
* Further tweaks
* Fix current state server test (sort of)
* Maybe fix appservices
* Fix client API test
* Include database connection string in database options
* Fix sync API build
* Update config test
* Fix unit tests
* Fix federation sender build
* Fix gobind build
* Set Listen address for all services in HTTP monolith mode
* Validate config, reinstate appservice derived in directory, tweaks
* Tweak federation API test
* Set MaxOpenConnections/MaxIdleConnections to previous values
* Update generate-config
* Fix New users appear in /keys/changes
* Create blank device keys when logging in on a new device
* Add PerformDeviceUpdate and fix a few bugs
- Correct device deletion query on sqlite
- Return no keys on /keys/query rather than an empty key
* Unbreak sqlite properly
* Use a real DB for currentstateserver integration tests
* Race fix
* Recheck device lists when join/leave events come in
* Add PerformDeviceDeletion
* Notify clients when devices are deleted
* Unbreak things
* Remove debug logging
* User directory
* Fix syncapi unit test
* Make user directory only show remote users you know about from your joined rooms
* Update sytest-whitelist
* Review comments
* Modify /state/{eventType}/{stateKey} to return the event at the time the user left
Or live, depending on their current state. Hopefully fixes some sytests!
* Linting
* Set HasBeenInRoom
* Fix cases for world-readable history visibility
* Fix bug in finding the requested state event
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Add storage layer for postgres/sqlite
* Return OTK counts when inserting new keys
* Hook up the key DB and make a test pass
* Convert postgres queries to be sqlite queries
* Blacklist test due to requiring rejected events
* Unbreak tests
* Update blacklist
* Add User-Interactive Authentication
And use it when deleting a device. With tests.
* Make remaining sytest pass
* Linting
* 403 not 401 on wrong user/pass
* Use content_value instead of membership
* Fix build
* Replace publicroomsapi with a combination of clientapi/roomserver/currentstateserver
- All public rooms paths are now handled by clientapi
- Requests to (un)publish rooms are sent to the roomserver via `PerformPublish`
which are stored in a new `published_table.go`
- Requests for public rooms are handled in clientapi by:
* Fetch all room IDs which are published using `QueryPublishedRooms` on the roomserver.
* Apply pagination parameters to the slice.
* Do a `QueryBulkStateContent` request to the currentstateserver to pull out
required state event *content* (not entire events).
* Aggregate and return the chunk.
Mostly but not fully implemented (DB queries on currentstateserver are missing)
* Fix pq query
* Make postgres work
* Make sqlite work
* Fix tests
* Unbreak pagination tests
* Linting
* Remove membership table from account DB
And make code which needs that data use the currentstate server
* Unbreak tests; use a membership enum for space
* Move filter table to syncapi where it is used
* Implement /sync `limited` and read timeline limit from stored filters
We now fully handle `room.timeline.limit` filters (in-line + stored) and
return the right value for `limited` syncs.
* Update whitelist
* Default to the default timeline limit if it's unset, also strip the extra event correctly
* Update whitelist
* Pass join errors through internal API boundaries
Required for certain invite sytests. We will need to think of a
better way of handling this going forwards.
* Include m.room.avatar in stripped state; handle trailing slashes when GETing state events
* Update whitelist
* Update whitelist
* Make userapi responsible for checking access tokens
There's still plenty of dependencies on account/device DBs, but this
is a start. This is a breaking change as it adds a required config
value `listen.user_api`.
* Cleanup
* Review comments and test fix
* Remove clientapi producers which aren't actually producers
They are actually just convenience wrappers around the internal APIs
for roomserver/eduserver. Move their logic to their respective `api`
packages and call them directly.
* Remove TODO
* unbreak ygg
* Groundwork for send-to-device messaging
* Update sample config
* Add unstable routing for now
* Send to device consumer in sync API
* Start the send-to-device consumer
* fix indentation in dendrite-config.yaml
* Create send-to-device database tables, other tweaks
* Add some logic for send-to-device messages, add them into sync stream
* Handle incoming send-to-device messages, count them with EDU stream pos
* Undo changes to test
* pq.Array
* Fix sync
* Logging
* Fix a couple of transaction things, fix client API
* Add send-to-device test, hopefully fix bugs
* Comments
* Refactor a bit
* Fix schema
* Fix queries
* Debug logging
* Fix storing and retrieving of send-to-device messages
* Try to avoid database locks
* Update sync position
* Use latest sync position
* Jiggle about sync a bit
* Fix tests
* Break out the retrieval from the update/delete behaviour
* Comments
* nolint on getResponseWithPDUsForCompleteSync
* Try to line up sync tokens again
* Implement wildcard
* Add all send-to-device tests to whitelist, what could possibly go wrong?
* Only care about wildcard when targeted locally
* Deduplicate transactions
* Handle tokens properly, return immediately if waiting send-to-device messages
* Fix sync
* Update sytest-whitelist
* Fix copyright notice (need to do more of this)
* Comments, copyrights
* Return errors from Do, fix dendritejs
* Review comments
* Comments
* Constructor for TransactionWriter
* defletions
* Update gomatrixserverlib, sytest-blacklist
* Separate muxes for public and internal APIs
* Update client-api-proxy and federation-api-proxy so they don't add /api to the path
* Tidy up
* Consistent HTTP setup
* Set up prefixes properly
* Add PerformJoin template
* Try roomserver perform join
* Send correct server name to FS API
* Pass through content, try to handle multiple server names
* Fix local server checks
* Don't refer to non-existent error
* Add directory lookups of aliases
* Remove unneeded parameters
* Don't repeat join events into the roomserver
* Unmarshal the content, that would help
* Check if the user is already in the room in the fedeationapi too
* Return incompatible room version error
* Use Membership, don't try more servers than needed
* Review comments, make FS API take list of servernames, dedupe them, break out of loop properly on success
* Tweaks
* Consolidation of roomserver APIs
* Comment out alias tests for now, they are broken
* Wire AS API into roomserver again
* Roomserver didn't take asAPI param before so return to that
* Prevent roomserver asking AS API for alias info
* Rename some files
* Remove alias_test, incoherent tests and unwanted appservice integration
* Remove FS API inject on syncapi component
* Define an input API for the federationsender
* Wiring for rooomserver input API and federation sender input API
* Whoops, commit common too
* Merge input API into query API
* Rename FederationSenderQueryAPI to FederationSenderInternalAPI
* Fix dendritejs
* Rename Input to Perform
* Fix a couple of inputs -> performs
* Remove needless storage interface, add comments
* Create and glue ExternalPublicRoomsProvider into the public rooms component
This is how we will link p2p stuff to dendrite proper.
* Use gmsl structs rather than our own
* Implement federated public rooms
- Make thirdparty endpoint r0 so riot-web loads the public room list
* Typo
* Missing callsites
* Move current work into single branch
* Initial massaging of clientapi etc (not working yet)
* Interfaces for accounts/devices databases
* Duplicate postgres package for sqlite3 (no changes made to it yet)
* Some keydb, accountdb, devicedb, common partition fixes, some more syncapi tweaking
* Fix accounts DB, device DB
* Update naffka dependency for SQLite
* Naffka SQLite
* Update naffka to latest master
* SQLite support for federationsender
* Mostly not-bad support for SQLite in syncapi (although there are problems where lots of events get classed incorrectly as backward extremities, probably because of IN/ANY clauses that are badly supported)
* Update Dockerfile -> Go 1.13.7, add build-base (as gcc and friends are needed for SQLite)
* Implement GET endpoints for account_data in clientapi
* Nuke filtering for now...
* Revert "Implement GET endpoints for account_data in clientapi"
This reverts commit 4d80dff4583d278620d9b3ed437e9fcd8d4674ee.
* Implement GET endpoints for account_data in clientapi (#861)
* Implement GET endpoints for account_data in clientapi
* Fix accountDB parameter
* Remove fmt.Println
* Fix insertAccountData SQLite query
* Fix accountDB storage interfaces
* Add empty push rules into account data on account creation (#862)
* Put SaveAccountData into the right function this time
* Not sure if roomserver is better or worse now
* sqlite work
* Allow empty last sent ID for the first event
* sqlite: room creation works
* Support sending messages
* Nuke fmt.println
* Move QueryVariadic etc into common, other device fixes
* Fix some linter issues
* Fix bugs
* Fix some linting errors
* Fix errcheck lint errors
* Make naffka use postgres as fallback, fix couple of compile errors
* What on earth happened to the /rooms/{roomID}/send/{eventType} routing
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Implement missing device management features
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
* Add a little more documentation
* Undo changes
* Use non-anonymous struct to decode devices list
* Update sytest-whitelist
* Update sytest-whitelist
* Update sytest-blacklist
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Pass cfg by reference around the codebase
* Merge branch 'master' into pass-cfg-by-ref
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Add wiring for querying the roomserver for the default room version
* Try to implement /capabilities for room versions
* Update copyright notices
* Update sytests, add /capabilities endpoint into CS API
* Update sytest-whitelist
* Add GetDefaultRoomVersion
* Fix cases where state package was shadowed
* Fix version formatting
* Update Dockerfile to Go 1.13.6
* oh yes types I remember
* And fix the default too
We were escaping the URL before performing any pattern matching on it.
This meant that if you sent data that URLdecoded to a "/", it would count as
a "/" in the URL, potentially causing a 404. This was causing some flaky tests
with some randomly-generated query parameters.
Now, we keep URLs encoded while doing the pattern matching, and only afterwards
do we URL decode each query parameter individually before passing them to their
respective handler functions.
github.com/gorilla/mux was also updated to v1.7.3 to fix a bug with URL encoding and subrouters.
