Commit Graph

77 Commits

Author SHA1 Message Date
Till
d357615452
Don't send device list updates upon registration (#3307)
Fixes https://github.com/matrix-org/dendrite/issues/3273

As we otherwise send down device list updates which are merely useful
for the user and causes tests to be flakey:

```
 TestPushSync/Adding_a_push_rule_wakes_up_an_incremental_/sync (10ms)
      push_test.go:57: no pushrules found in sync response: {"next_batch":"s0_0_0_0_0_1_1_0_1","device_lists":{"changed":["@user-1:hs1"]}}
```

What this does: If a `PerformDeviceCreation` request is coming from
registering an account, it does **not** send device list updates, as
they are merely useful (no joined rooms, no one to inform) . In all
other cases, the behavior is unchanged and device list updates are sent
as usual.
2024-01-20 21:20:37 +01:00
Till
7863a405a5
Use IsBlacklistedOrBackingOff to determine if we should try to fetch devices (#3254)
Use `IsBlacklistedOrBackingOff` from the federation API to check if we
should fetch devices.

To reduce back pressure, we now only queue retrying servers if there's
space in the channel.
2023-11-09 08:43:27 +01:00
Till
da7bca0224
Some tweaks for the device list updater (#3251)
This makes the following changes:
- Adds two new metrics observing the usage of the `DeviceListUpdater`
workers
- Makes the number of workers configurable
- Adds a 30s timeout for DB requests when receiving a device list update
over federation
2023-10-31 16:39:45 +01:00
Till
8c23c1150c
Tweaks around the device list updater (#3227)
I hope the comments explain the changes.

`notifyWorkers` notifies a worker which then calls `processServer`,
which in turn gets all users and calls `processServerUser`. There is no
need to call `processServer` for the same domain on startup.
2023-10-23 11:09:05 +02:00
Tulir Asokan
bea73c765a
Fix user_id query param breaking auth for non-appservices (#3196)
The `user_id` query param only has defined behavior when authenticating
with an `as_token`. For any other tokens, the presence of the parameter
should simply be ignored.

Fixes #1738

Signed-off-by: Tulir Asokan <tulir@maunium.net>
Co-authored-by: devonh <devon.dmytro@gmail.com>
2023-09-12 14:44:51 +00:00
santhoshivan23
45082d4dce
feat: admin APIs for token authenticated registration (#3101)
### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [x] I have added Go unit tests or [Complement integration
tests](https://github.com/matrix-org/complement) for this PR _or_ I have
justified why this PR doesn't need tests
* [x] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)
_or_ I have already signed off privately

Signed-off-by: `Santhoshivan Amudhan santhoshivan23@gmail.com`
2023-06-22 16:37:21 +00:00
devonh
0489d16f95
Move json errors over to gmsl (#3080) 2023-05-09 22:46:49 +00:00
Till
9e9617ff84
Add key backup tests (#3071)
Also slightly refactors the functions and methods to rely less on the
req/res pattern we had for polylith.

Returns `M_WRONG_ROOM_KEYS_VERSION` for some endpoints as per the spec
2023-04-28 17:49:38 +02:00
Till
6b47cf0f6a
Remove PerformError (#3066)
This removes `PerformError`, which was needed when we still had
polylith.

This removes quite a bunch of
```go
if err != nil {
	return err
}
if err := res.Error; err != nil {
	return err.JSONResponse()
}
```

Hopefully can be read commit by commit. [skip ci]
2023-04-28 17:46:01 +02:00
devonh
ed19efc5d7
Move fedclient interface over to gmsl (#3061)
Companion PR: https://github.com/matrix-org/gomatrixserverlib/pull/366
2023-04-24 16:23:25 +00:00
kegsay
72285b2659
refactor: update GMSL (#3058)
Sister PR to https://github.com/matrix-org/gomatrixserverlib/pull/364

Read this commit by commit to avoid going insane.
2023-04-19 15:50:33 +01:00
Till
c45d8cd688
Add pushrules tests (#3044)
partly takes care of https://github.com/matrix-org/dendrite/issues/2870
by making sure that rule IDs don't start with a dot.

Co-authored-by: kegsay <kegan@matrix.org>
2023-04-14 12:35:27 +01:00
kegsay
0db43f13a6
refactor: use latest GMSL which splits fed client from matrix room logic (#3051)
Part of a series of refactors on GMSL.
2023-04-06 09:55:01 +01:00
Till
c2db38d295
Add user profile tests, refactor user API methods (#3030)
This adds tests for `/profile`.
Also, as a first change in this regard, refactors the methods defined on
the `UserInternalAPI` to not use structs as the request/response
parameters.
2023-04-03 20:19:26 +02:00
Till
2854ffeb7d
Add CS API device tests (#3029)
Adds tests for
- `/devices`
- `/delete_devices` (also adds UIA)
2023-03-31 10:15:01 +02:00
Till
5e85a00cb3
Remove BaseDendrite (#3023)
Removes `BaseDendrite` to, hopefully, make testing and composing of
components easier in the future.
2023-03-22 09:21:32 +01:00
Till
5579121c6f
Preparations for removing BaseDendrite (#3016)
Preparations to actually remove/replace `BaseDendrite`.
Quite a few changes:
- SyncAPI accepts an `fulltext.Indexer` interface (fulltext is removed
from `BaseDendrite`)
- Caches are removed from `BaseDendrite`
- Introduces a `Router` struct (likely to change)
  - also fixes #2903
- Introduces a `sqlutil.ConnectionManager`, which should remove
`base.DatabaseConnection` later on
- probably more
2023-03-17 11:09:45 +00:00
Till
7f114cc538
Fix issue where device keys are removed if a device ID is reused (#2982)
Fixes https://github.com/matrix-org/dendrite/issues/2980
2023-02-20 15:26:09 +01:00
Till
4594233f89
Merge keyserver & userapi (#2972)
As discussed yesterday, a first draft of merging the keyserver and the
userapi.
2023-02-20 14:58:03 +01:00
Till
5eed31fea3
Handle guest access [1/2?] (#2872)
Needs https://github.com/matrix-org/sytest/pull/1315, as otherwise the
membership events aren't persisted yet when hitting `/state` after
kicking guest users.

Makes the following tests pass:
```
Guest users denied access over federation if guest access prohibited
Guest users are kicked from guest_access rooms on revocation of guest_access
Guest users are kicked from guest_access rooms on revocation of guest_access over federation
```

Todo (in a follow up PR):
- Restrict access to CS API Endpoints as per
https://spec.matrix.org/v1.4/client-server-api/#client-behaviour-14

Co-authored-by: kegsay <kegan@matrix.org>
2022-12-22 13:05:59 +01:00
Neil Alexander
529df30b56
Virtual hosting schema and logic changes (#2876)
Note that virtual users cannot federate correctly yet.
2022-11-11 16:41:37 +00:00
Neil Alexander
f6dea712d2
Initial support for multiple server names (#2829)
This PR is the first step towards virtual hosting by laying the
groundwork for multiple server names being configured.
2022-10-26 12:59:19 +01:00
Neboer
2a4c7f45b3
Add support for config "auto_join_rooms" (#2823)
Add support for config "auto_join_rooms". Now new accounts can join the
rooms in config file automatically.

### Pull Request Checklist

<!-- Please read
https://matrix-org.github.io/dendrite/development/contributing before
submitting your pull request -->

* [x] I have justified why this PR doesn't need tests.
* [x] Pull request includes a [sign off below using a legally
identifiable
name](https://matrix-org.github.io/dendrite/development/contributing#sign-off)

Signed-off-by: `Rubin Poster <rubinposter@gmail.com>`
2022-10-26 11:04:53 +02:00
Till
e57b301722
Set display_name and/or avatar_url for server notices (#2820)
This should fix #2815 by making sure we actually set the `display_name`
and/or `avatar_url` and create the needed membership event.
To avoid creating a new membership event when starting Dendrite,
`SetAvatarURL` and `SetDisplayName` now return a `Changed` value, which
also makes the regular endpoints idempotent.
2022-10-21 10:48:25 +02:00
Neil Alexander
980fa55846
Stronger passwordless account checks (fixes #2780) 2022-10-10 10:39:29 +01:00
Till Faelligen
3617d5a0ff
Remove SendAccountData, since InputAccountData is already doing that 2022-09-30 17:31:41 +02:00
Till
249b32c4f3
Refactor notifications (#2688)
This PR changes the handling of notifications
- removes the `StreamEvent` and `ReadUpdate` stream
- listens on the `OutputRoomEvent` stream in the UserAPI to inform the
SyncAPI about unread notifications
- listens on the `OutputReceiptEvent` stream in the UserAPI to set
receipts/update notifications
- sets the `read_markers` directly from within the internal UserAPI

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-09-27 15:01:34 +02:00
Till
64472d9aab
Update getting pushrules, add tests, tweak pushrules (#2705)
This PR
- adds tests for `evaluatePushrules`
- removes the need for the UserAPI on the `OutputStreamEventConsumer`
(for easier testing)
- adds a method to get the pushrules from the database
- adds a new default pushrule for `m.reaction` events (and some other
tweaks)
2022-09-09 13:56:33 +02:00
Neil Alexander
ad6b902b84
Refactor appservices component (#2687)
This PR refactors the app services component. It makes the following changes:

* Each appservice now gets its own NATS JetStream consumer
* The appservice database is now removed entirely, since we just use JetStream as a data source instead
* The entire component is now much simpler and we deleted lots of lines of code 💅

The result is that it should be much lighter and hopefully much more performant.
2022-09-01 09:20:40 +01:00
Till
48600d5540
Use /admin/v1/register in create-account (#2484)
* Get all account data on CompleteSync

* Revert "Get all account data on CompleteSync"

This reverts commit 44a3e566d8fb940b0b757aea9b8408fa19ea9f54.

* Use /_synapse/admin/v1/register to create account

* Linting

* Linter again :)

* Update docs

* Use HTTP API to reset password, add option to User API `PerformPasswordUpdate` to invalidate sessions

* Fix routing name

* Tell me more about what went wrong

* Deprecate the `-reset-password` flag, document the new API

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-08-12 12:00:07 +01:00
Neil Alexander
c45d0936b5
Generic-based internal HTTP API (#2626)
* Generic-based internal HTTP API (tested out on a few endpoints in the federation API)

* Add `PerformInvite`

* More tweaks

* Fix metric name

* Fix LookupStateIDs

* Lots of changes to clients

* Some serverside stuff

* Some error handling

* Use paths as metric names

* Revert "Use paths as metric names"

This reverts commit a9323a6a343f5ce6461a2e5bd570fe06465f1b15.

* Namespace metric names

* Remove duplicate entry

* Remove another duplicate entry

* Tweak error handling

* Some more tweaks

* Update error behaviour

* Some more error tweaking

* Fix API path for `PerformDeleteKeys`

* Fix another path

* Tweak federation client proxying

* Fix another path

* Don't return typed nils

* Some more tweaks, not that it makes any difference

* Tweak federation client proxying

* Maybe fix the key backup test
2022-08-11 15:29:33 +01:00
Neil Alexander
962b76da44
Make the User API responsible for sending account data output events (#2592)
* Make the User API responsible for sending account data output events

* Clean up producer

* Review comments
2022-07-25 17:30:07 +01:00
Neil Alexander
519bc1124b
Add evacuateUser endpoint, use it when deactivating accounts (#2545)
* Add `evacuateUser` endpoint, use it when deactivating accounts

* Populate the API

* Clean up user devices when deactivating

* Include invites, delete pushers
2022-06-29 15:29:39 +01:00
kegsay
9957752a9d
Define component interfaces based on consumers (2/2) (#2425)
* convert remaining interfaces

* Tidy up the userapi interfaces
2022-05-05 19:30:38 +01:00
Till
3c940c428d
Add opt-in anonymous stats reporting (#2249)
* Initial phone home stats queries

* Add userAgent to UpdateDeviceLastSeen
Add new Table for tracking daily user vists

* Add user_daily_visits table

* Fix queries

* userapi stats tables & queries

* userapi interface and internal api

* sycnapi stats queries

* testing phone home stats

* Add complete config to syncapi

* add missing files

* Fix queries

* Send empty request

* Add version & monolith stats

* Add configuration for phone home stats

* Move WASM to its own file, add config and comments

* Add tracing methods

* Add total rooms

* Add more fields, actually send data somewhere

* Move stats to the userapi

* Move phone home stats to util package

* Cleanup

* Linter & parts of GH comments

* More GH comments changes
- Move comments to SQL statements
- Shrink interface, add struct for stats
- No fatal errors, use defaults

* Be more explicit when querying

* Fix wrong calculation & wrong query params
Add tests

* Add Windows stats

* ADd build constraint

* Use new testing structure
Fix issues with getting values when using SQLite
Fix wrong AddDate value
Export UpdateUserDailyVisits

* Fix query params

* Fix test

* Add comment about countR30UsersSQL and countR30UsersV2SQL; fix test

* Update config

* Also update example config file

* Use OS level proxy, update logging

Co-authored-by: kegsay <kegan@matrix.org>
2022-05-04 19:04:28 +02:00
Till
e95fc5c5e3
Use provided filter for account_data (#2372)
* Reuse IncrementalSync, use provided filter

* Inform SyncAPI about newly created push_rules
2022-04-25 19:04:46 +02:00
Kegan Dougal
073972646f Use unix not second 2022-04-19 13:57:02 +01:00
Neil Alexander
7e745665a4
Change pushkey_ts to be seconds (fix #2354) (#2358) 2022-04-19 09:51:02 +01:00
Neil Alexander
0692be44d9
Fix account availability on register 2022-03-28 13:31:17 +01:00
S7evinK
f2e550efd8
Refactor appservice & client API to use userapi internal (#2290)
* Refactor user api internal

* Refactor clientapi to use internal userapi

* Use internal userapi instead of user DB directly

* Remove AccountDB dependency

* Fix linter issues

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-03-24 22:45:44 +01:00
Dan
f05ce478f0
Implement Push Notifications (#1842)
* Add Pushserver component with Pushers API

Co-authored-by: Tommie Gannert <tommie@gannert.se>
Co-authored-by: Dan Peleg <dan@globekeeper.com>

* Wire Pushserver component

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>

* Add PushGatewayClient.

The full event format is required for Sytest.

* Add a pushrules module.

* Change user API account creation to use the new pushrules module's defaults.

Introduces "scope" as required by client API, and some small field
tweaks to make some 61push Sytests pass.

* Add push rules query/put API in Pushserver.

This manipulates account data over User API, and fires sync messages
for changes. Those sync messages should, according to an existing TODO
in clientapi, be moved to userapi.

Forks clientapi/producers/syncapi.go to pushserver/ for later extension.

* Add clientapi routes for push rules to Pushserver.

A cleanup would be to move more of the name-splitting logic into
pushrules.go, to depollute routing.go.

* Output rooms.join.unread_notifications in /sync.

This is the read-side. Pushserver will be the write-side.

* Implement pushserver/storage for notifications.

* Use PushGatewayClient and the pushrules module in Pushserver's room consumer.

* Use one goroutine per user to avoid locking up the entire server for
  one bad push gateway.
* Split pushing by format.
* Send one device per push. Sytest does not support coalescing
  multiple devices into one push. Matches Synapse. Either we change
  Sytest, or remove the group-by-url-and-format logic.
* Write OutputNotificationData from push server. Sync API is already
  the consumer.

* Implement read receipt consumers in Pushserver.

Supports m.read and m.fully_read receipts.

* Add clientapi route for /unstable/notifications.

* Rename to UpsertPusher for clarity and handle pusher update

* Fix linter errors

* Ignore body.Close() error check

* Fix push server internal http wiring

* Add 40 newly passing 61push tests to whitelist

* Add next 12 newly passing 61push tests to whitelist

* Send notification data before notifying users in EDU server consumer

* NATS JetStream

* Goodbye sarama

* Fix `NewStreamTokenFromString`

* Consume on the correct topic for the roomserver

* Don't panic, NAK instead

* Move push notifications into the User API

* Don't set null values since that apparently causes Element upsetti

* Also set omitempty on conditions

* Fix bug so that we don't override the push rules unnecessarily

* Tweak defaults

* Update defaults

* More tweaks

* Move `/notifications` onto `r0`/`v3` mux

* User API will consume events and read/fully read markers from the sync API with stream positions, instead of consuming directly

Co-authored-by: Piotr Kozimor <p1996k@gmail.com>
Co-authored-by: Tommie Gannert <tommie@gannert.se>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-03-03 11:40:53 +00:00
Neil Alexander
153bfbbea5
Merge both user API databases into one (#2186)
* Merge user API databases into one

* Remove DeviceDatabase from config

* Fix tests

* Try that again

* Clean up keyserver device keys when the devices no longer exist in the user API

* Tweak ordering

* Fix UserExists flag, device check

* Allow including empty entries so we can clean them up

* Remove logging
2022-02-18 11:31:05 +00:00
S7evinK
5a39512f5f
Add account type (#2171)
* Add account_type for sqlite3

* Add account_type for postgres

* Remove CreateGuestAccount from interface

* Add new AccountTypes & update test

* Use newly added AccountType for account creation

* Add migrations

* Reuse type

* Add AccounnType to Device, so it can be verified on requests

* Rename migration, add missing update for appservices

* Rename sqlite3 migration

* Add missing AccountType to return value

* Update sqlite migration
Change allowance check on /admin/whois

* Fix migration, add IS NULL

* Move accountType to completeRegistration

* Fix migrations

* Add passing test
2022-02-16 18:55:38 +01:00
tommie
c36e4546c3
Support for m.login.token (#2014)
* Add GOPATH to PATH in find-lint.sh.

The user doesn't necessarily have it in PATH.

* Refactor LoginTypePassword and Type to support m.login.token and m.login.sso.

For login token:

* m.login.token will require deleting the token after completeAuth has
  generated an access token, so a cleanup function is returned by
  Type.Login.
* Allowing different login types will require parsing the /login body
  twice: first to extract the "type" and then the type-specific parsing.
  Thus, we will have to buffer the request JSON in /login, like
  UserInteractive already does.

For SSO:

* NewUserInteractive will have to also use GetAccountByLocalpart. It
  makes more sense to just pass a (narrowed-down) accountDB interface
  to it than adding more function pointers.

Code quality:

* Passing around (and down-casting) interface{} for login request types
  has drawbacks in terms of type-safety, and no inherent benefits. We
  always decode JSON anyway. Hence renaming to Type.LoginFromJSON. Code
  that directly uses LoginTypePassword with parsed data can still use
  Login.
* Removed a TODO for SSO. This is already tracked in #1297.
* httputil.UnmarshalJSON is useful because it returns a JSONResponse.

This change is intended to have no functional changes.

* Support login tokens in User API.

This adds full lifecycle functions for login tokens: create, query, delete.

* Support m.login.token in /login.

* Fixes for PR review.

* Set @matrix-org/dendrite-core as repository code owner

* Return event NID from `StoreEvent`, match PSQL vs SQLite behaviour, tweak backfill persistence (#2071)

Co-authored-by: kegsay <kegan@matrix.org>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-02-10 10:27:26 +00:00
S7evinK
08a0278760
Add missing HTTP mode for userapi (#1982)
* Add missing internal api endpoint

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Add missing performKeyBackup endpoint

* Add missing http mode for userapi

* Fix failing tests

* Add error checks

* Fix sytest

* Update startup logic for HTTP mode

* Use userImpl for AS (annoying)

* Don't send device list updates for appservice devices

* Fix build

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-12-03 17:18:35 +00:00
Ryan W
a624eab309
- Removed double imports (#1989)
- Lower cased error messages

Signed-off-by: Ryan Whittington <twentybitdev@gmail.com>

Co-authored-by: kegsay <kegan@matrix.org>
2021-09-08 17:31:03 +01:00
Neil Alexander
614e67280d
Delete device keys/signatures from key server when deleting devices (#1979)
* Delete device keys/signatures from key server when deleting device from user API

* Move loop to within database transaction

* Don't fall over deleting no rows
2021-08-18 12:07:09 +01:00
kegsay
32bf14a37c
Key Backups (3/3) : Implement querying keys and various bugfixes (#1946)
* Add querying device keys

Makes a bunch of sytests pass

* Apparently only the current version supports uploading keys

* Linting
2021-07-27 19:29:32 +01:00
kegsay
b3754d68fc
Key Backups (2/3) : Add E2E backup key tables (#1945)
* Add PUT key backup endpoints and glue them to PerformKeyBackup

* Add tables for storing backup keys and glue them into the user API

* Don't create tables whilst still WIPing

* writer on sqlite please

* Linting
2021-07-27 17:08:53 +01:00
kegsay
32538640db
Key backups (1/2) : Add E2E session backup metadata tables (#1943)
* Initial key backup paths and userapi API

* Fix unit tests

* Add key backup table

* Glue REST API to database

* Linting

* use writer on sqlite
2021-07-27 12:47:32 +01:00