Commit Graph

111 Commits

Author SHA1 Message Date
kegsay
0bf5104bbb
Fix #2027 by gracefully handling stub rooms (#2100)
The server ACL code on startup will grab all known rooms from
the rooms_table and then call `GetStateEvent` with each found
room ID to find the server ACL event. This can fail for stub
rooms, which will be present in the rooms table. Previously
this would result in an error being returned and the server
failing to start (!). Now we just return no event for stub
rooms.
2022-01-21 14:23:37 +00:00
Neil Alexander
c3dda0779d
Return event NID from StoreEvent, match PSQL vs SQLite behaviour, tweak backfill persistence (#2071) 2021-12-09 15:03:26 +00:00
Neil Alexander
6e93531e94
Don't persist transaction IDs in the roomserver (#2048) 2021-11-22 09:13:12 +00:00
Neil Alexander
403498a85b
Only return non-stub rooms from GetKnownRooms (#2049)
* Only return non-stub rooms from `GetKnownRooms`

This should stop a bunch of errors at startup with invalid server ACLs.

* Fix query
2021-11-18 11:34:19 +00:00
PiotrKozimor
dec05c3347
Run gofmt on dendrite - apply go 1.17 preferred build tags (#2021) 2021-11-02 16:48:48 +00:00
Neil Alexander
b99f594a93
Fix #2028 (#2036) 2021-11-02 16:47:39 +00:00
Ryan W
a624eab309
- Removed double imports (#1989)
- Lower cased error messages

Signed-off-by: Ryan Whittington <twentybitdev@gmail.com>

Co-authored-by: kegsay <kegan@matrix.org>
2021-09-08 17:31:03 +01:00
kegsay
7dc8fb1fe7
Add more logs (#2005)
* Add more logs

To help debug the migration issue in #1924 along with manual data-loss-inducing fixes.
Also log the origin server on processed txns to help debug buggy server origins.

* Fix query
2021-09-07 15:07:14 +01:00
kegsay
4cc8b28b7f
Ensure all create events have a snapshot NID of 0 (#1961)
Fixes #1924 for postgres users, though the underlying cause of why
they aren't 0 in the first place is unresolved.
2021-08-04 17:48:23 +01:00
kegsay
ed04eed441
Fix sqlite migration issues (#1960)
* Do not store 'null' in the database for empty JSON arrays

This can cause issues, though it should be noted that the majority
of the time this will marshal/unmarshal just fine, see
https://play.golang.org/p/Doe2NZUgv7Q

* bugfix: sqlite migration should handle create events as having no 'before' snapshot

The state snapshot for any given event in the roomserver represents the state _before_
the event. For the create event, this is nothing, so the state snapshot nid should be 0.

In some cases this wasn't happening, resulting in a nice mix of possible options including:
 - A state snapshot without any state blocks `[]` or `null`.
 - A state snapshot with a single state block with a single event, the create event, causing
   a circular loop. This is incorrect as it represents the state before the event, not after.

* Add state key check
2021-08-04 17:08:17 +01:00
Kegan Dougal
ed4097825b Factor out StatementList to sqlutil and use it in userapi
It helps with the boilerplate.
2021-07-28 18:30:04 +01:00
kegsay
16bf94f239
Not finding the snapshot is not fatal (#1940) 2021-07-26 12:30:44 +01:00
Neil Alexander
39e8d1cc6f
Track knocking in membership updater (#1935)
* Topologically sort outliers in SendEventWithState

* Knock in membership updater

* Update gomatrixserverlib

* Update gomatrixserverlib

* Get the NID of the knock event properly for the membership updater
2021-07-22 12:26:58 +01:00
Neil Alexander
c1447a58e5
Various alias fixes (#1934)
* Generate m.room.canonical_alias instead of legacy m.room.aliases

* Add omitempty tags

* Add aliases endpoint to client API

* Check power levels when setting aliases

* Don't return null on /aliases

* Don't return error if the state event fails

* Update sytest-whitelist

* Don't send updated m.room.canonical_alias events

* Don't check PLs after all because for local aliases they are apparently irrelevant

* Fix some bugs

* Allow deleting a local alias with enough PL

* Fix some more bugs

* Update sytest-whitelist

* Fix copyright notices

* Review comments
2021-07-21 16:53:50 +01:00
Neil Alexander
f0f8c7f055
Optimise QueryServerJoinedToRoom (#1933)
* Optimise checking if a server is in a room

* Fix queries

* Fix queries
2021-07-21 13:06:32 +01:00
Neil Alexander
f63068df3b
Only include go-sqlite3 on the relevant binaries (#1900)
* Only include go-sqlite3 on the relevant binaries

* The driver name is always sqlite3 now

* Update to matrix-org/go-sqlite3-js@e537baa
2021-07-20 11:18:14 +01:00
David Spenler
8d8fe485b4
Fix failing ban tests (#1884)
* Add room membership and powerlevel checks for func SendBan

* Added non-error return to func GetStateEvent when no state events with the specified state key are found

* Add passing tests to whitelist

* Fixed formatting

* Update roomserver/storage/shared/storage.go

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: kegsay <kegan@matrix.org>
Co-authored-by: kegsay <kegsay@gmail.com>
2021-07-19 18:33:05 +01:00
Neil Alexander
c8408a6387
Add more optimised code path for checking if we're in a room (#1909)
* Add more optimised code path for checking if we're in a room

* Fix database queries

* Fix federation API test

* Fix logging

* Review comments

* Make separate API call for room membership
2021-07-09 16:36:45 +01:00
kegsay
3e50bac944
bugfix: order the state blocks so recreating state snapshots works correctly (#1908)
* Logging

* Revert "Logging"

This reverts commit 23ce334182d8a70f8e0381786f9dea77a9b91ed8.

* bugfix: order the state blocks so recreating state snapshots works correctly
2021-07-09 10:49:49 +01:00
kegsay
5a09290c32
db migration: handle create events with no state blocks from v0.1.0 (#1904) 2021-07-07 17:07:33 +01:00
kegsay
c849e74dfc
db migration: fix #1844 and add additional assertions (#1889)
* db migration: fix #1844 and add additional assertions

- Migration scripts will now check to see if there are any unconverted
  snapshot IDs and fail the migration if there are any. This should
  prevent people from getting a corrupt database in the event the root
  cause is still unknown.
- Add an ORDER BY clause when doing batch queries in the postgres
  migration. LIMIT and OFFSET without ORDER BY are undefined and must
  not be relied upon to produce a deterministic ordering (e.g row order).
  See https://www.postgresql.org/docs/current/queries-limit.html

* Linting

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-06-29 11:25:17 +01:00
Neil Alexander
5357df36c9
Fix panic in roomserver 2021-06-21 09:41:12 +01:00
Neil Alexander
c67d8da3eb
Fix bug in SQLite migration 2021-04-26 13:45:47 +01:00
Neil Alexander
5ce1fe80de
State storage refactor (#1839)
* Hash-deduplicated state storage (and migrations) for PostgreSQL and SQLite

* Refactor droomserver database setup for migrations

* Fix conflict statements

* Update migration names

* Set a boundary for old to new block/snapshot IDs so we don't rewrite them more than once accidentally

* Create sequence if not exists

* Fix boundary queries

* Fix boundary queries

* Use Query

* Break out queries a bit

* More sequence tweaks

* Query parameters are not playing the game

* Injection escaping may not work for CREATE SEQUENCE after all

* Fix snapshot sequence name

* Use boundaried IDs in SQLite too

* Use IFNULL for SQLite

* Use COALESCE in PostgreSQL

* Review comments @Kegsay
2021-04-26 13:25:57 +01:00
Neil Alexander
d15836e260
Increase gocyclo complexity to 25 (and remove all but 2 golint directives related to it) (#1783) 2021-03-03 14:35:57 +00:00
Neil Alexander
b891c00b09
Add RoomInfo cache, remove RoomServerRoomNIDsCache (#1646)
* Add RoomInfo cache, remove RoomServerRoomNID cache, ensure caches are thread-safe

* Don't panic if the roomInfo isn't known yet

* LRU package is already threadsafe

* Use RoomInfo cache to find room version if possible in Events()

* Adding comments about RoomInfoCache safety
2020-12-16 12:15:12 +00:00
Neil Alexander
9057143033
Hit the database far less in Events to find room NIDs and room versions (#1643)
* Hit the database far less to find room NIDs for event NIDs

* Close the rows

* Fix SQLite selectRoomNIDsForEventNIDsSQL

* Give same treatment to room version lookups
2020-12-16 10:33:28 +00:00
Neil Alexander
1ce9c52442
Don't recalculate event IDs so often (#1610)
* Don't recalculate event IDs so often

* Revert invite change

* Make sure we're using the right NIDs

* Update gomatrixserverlib

* Update to NewEventFromTrustedJSONWithEventID

* Fix go.mod

* Update gomatrixserverlib to matrix-org/gomatrixserverlib#243

* Use BulkSelectEventID
2020-12-04 10:41:07 +00:00
Neil Alexander
b5aa7ca3ab
Top-level setup package (#1605)
* Move config, setup, mscs into "setup" top-level folder

* oops, forgot the EDU server

* Add setup

* goimports
2020-12-02 17:41:00 +00:00
Neil Alexander
20a01bceb2
Pass pointers to events — reloaded (#1583)
* Pass events as pointers

* Fix lint errors

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update to matrix-org/gomatrixserverlib#240
2020-11-16 15:44:53 +00:00
S7evinK
eccd0d2c1b
Implement forgetting about rooms (#1572)
* Add basic storage methods

* Add internal api handler

* Add check for forgotten room

* Add /rooms/{roomID}/forget endpoint

* Add missing rsAPI method

* Remove unused parameters

* Add passing tests

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Add missing file

* Add postgres migration

* Add sqlite migration

* Use Forgetter to forget room

* Remove empty line

* Update HTTP status codes

It looks like the spec calls for these to be 400, rather than 403: https://matrix.org/docs/spec/client_server/r0.6.1#post-matrix-client-r0-rooms-roomid-forget

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-11-05 10:19:23 +00:00
Kegsay
6b8791b868
Always call overridden form of GetLatestEventsForUpdate (#1554)
This ensures we don't make txns on sqlite still, which can cause
'database is locked' errors.
2020-10-20 19:32:33 +01:00
Kegsay
eb86e2b336
Fix sqlite locking bugs present on sytest (#1543)
* Fix sqite locking bugs present on sytest

Comments do the explaining.

* Fix deadlock in sqlite mode

Caused by starting a writer whilst within a writer

* Only complain about invalid state deltas for non-overwrite events

* Do not re-process outlier unnecessarily
2020-10-20 11:42:54 +01:00
Neil Alexander
e154c45b51
Better logging around db.StoreEvent 2020-10-15 14:14:17 +01:00
Kegsay
e3c2b081c7
txn nil guard when rolling back LatestEventsUpdater (#1524)
* txn nil guard when rolling back LatestEventsUpdater

* Spell lint correctly
2020-10-14 17:05:09 +01:00
Neil Alexander
8d9ecb3996
Ignore duplicate redaction entries (#1522) 2020-10-14 15:24:43 +01:00
Kegsay
d7ea814fa8
Wrap NewMembershipUpdater in a db writer (#1515) 2020-10-13 10:20:27 +01:00
Neil Alexander
d821f9d3c9
Deep checking of forward extremities (#1491)
* Deep forward extremity calculation

* Use updater txn

* Update error

* Update error

* Create previous event references in StoreEvent

* Use latest events updater to row-lock prev events

* Fix unexpected fallthrough

* Fix deadlock

* Don't roll back

* Update comments in calculateLatest

* Don't include events that we can't find references for in the forward extremities

* Add another passing test
2020-10-07 14:05:33 +01:00
Neil Alexander
91fc1f1c92
Fix bug in error handling in SQLite InsertPreviousEvent (#1456) 2020-10-01 10:55:00 +01:00
Sam
a6700331ce
Update all usages of tx.Stmt to sqlutil.TxStmt (#1423)
* Replace all usages of txn.Stmt with sqlutil.TxStmt

Signed-off-by: Sam Day <me@samcday.com>

* Fix sign off link in PR template.

Signed-off-by: Sam Day <me@samcday.com>

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-09-24 11:10:14 +01:00
Neil Alexander
45de9dc1c0
Use room version cache in Events() 2020-09-21 16:49:37 +01:00
Neil Alexander
a06c18bb56
Soft-fail (#1364)
* Initial work on soft-fail

* Fix state block retrieval

* Copy-pasta QueryLatestEventsAndState code

* Fix state lookup

* Clean up

* Fix up failing sytest

* Linting

* Update previous events SQLite insert query

* Update SQLite InsertPreviousEvent properly

* Hopefully fix the event references updates

Co-authored-by: Kegan Dougal <kegan@matrix.org>
2020-09-21 14:55:46 +01:00
Kegsay
18231f25b4
Implement rejected events (#1426)
* WIP Event rejection

* Still send back errors for rejected events

Instead, discard them at the federationapi /send layer rather than
re-implementing checks at the clientapi/PerformJoin layer.

* Implement rejected events

Critically, rejected events CAN cause state resolution to happen
as it can merge forks in the DAG. This is fine, _provided_ we
do not add the rejected event when performing state resolution,
which is what this PR does. It also fixes the error handling
when NotAllowed happens, as we were checking too early and needlessly
handling NotAllowed in more than one place.

* Update test to match reality

* Modify InputRoomEvents to no longer return an error

Errors do not serialise across HTTP boundaries in polylith mode,
so instead set fields on the InputRoomEventsResponse. Add `Err()`
function to make the API shape basically the same.

* Remove redundant returns; linting

* Update blacklist
2020-09-16 13:00:52 +01:00
Matthew Hodgson
39507bacc3
Peeking via MSC2753 (#1370)
Initial implementation of MSC2753, as tested by https://github.com/matrix-org/sytest/pull/944.
Doesn't yet handle unpeeks, peeked EDUs, or history viz changing during a peek - these will follow.
https://github.com/matrix-org/dendrite/pull/1370 has full details.
2020-09-10 14:39:18 +01:00
Neil Alexander
35564dd73c
Process membership updates in writers (#1414) 2020-09-08 17:48:07 +01:00
Kegsay
7913759921
Remove QueryBulkStateContent from current state server (#1404)
* Remove QueryBulkStateContent from current state server

Expected fail due to db impl not existing

* Implement query bulk state content

* Fix up rejecting invites over federation

* Fix bulk content marshalling
2020-09-07 12:38:09 +01:00
Neil Alexander
b9caccbce8
Update GetStateEvent behaviour (#1399) 2020-09-04 19:40:21 +01:00
Kegsay
ca8dcf46b7
Remove QuerySharedUsers from current state server (#1396)
* Remove QuerySharedUsers from current state server

* Bugfixes
2020-09-04 14:25:01 +01:00
Neil Alexander
f1a98e1193
Fix nil txn bug 2020-09-04 10:22:32 +01:00
Kegsay
33b8143a95
Implement more CSS storage functions in roomserver (#1388) 2020-09-03 18:27:02 +01:00