Commit Graph

164 Commits

Author SHA1 Message Date
Neil Alexander
6e63df1d9a
KindOld (#1531)
* Add KindOld

* Don't process latest events/memberships for old events

* Allow federationsender to ignore duplicate key entries when LatestEventIDs is duplicated by RS output events

* Signal to downstream components if an event has become a forward extremity

* Don't exclude from sync

* Soft-fail checks on KindNew

* Don't run the latest events updater at all for KindOld

* Don't make federation sender change after all

* Kind in federation sender join

* Don't send isForwardExtremity

* Fix syncapi

* Update comments

* Fix SendEventWithState

* Update sytest-whitelist

* Generate old output events

* Sync API consumes old room events

* Update comments
2020-10-19 14:59:13 +01:00
Neil Alexander
e154c45b51
Better logging around db.StoreEvent 2020-10-15 14:14:17 +01:00
Neil Alexander
49abe359e6
Start Kafka connections for each component that needs them (#1527)
* Start Kafka connection for each component that needs one

* Fix roomserver unit tests

* Rename to naffkaInstance (@Kegsay review comment)

* Fix import cycle
2020-10-15 13:27:13 +01:00
Kegsay
e3c2b081c7
txn nil guard when rolling back LatestEventsUpdater (#1524)
* txn nil guard when rolling back LatestEventsUpdater

* Spell lint correctly
2020-10-14 17:05:09 +01:00
Neil Alexander
8d9ecb3996
Ignore duplicate redaction entries (#1522) 2020-10-14 15:24:43 +01:00
Neil Alexander
7a1fd123de
Improved state handling in /send (#1521)
* Capture errors

* Don't request only state key tuples needed for auth (we end up discarding room state this way)

* QueryStateAfterEvent returns all state when no tuples supplied

* Resolve state

* Comments
2020-10-14 12:39:37 +01:00
Kegsay
d7ea814fa8
Wrap NewMembershipUpdater in a db writer (#1515) 2020-10-13 10:20:27 +01:00
Neil Alexander
8001627cfc
Get missing event tweaks (#1514)
* Adjust backfill to send backward extremity with state before other backfilled events, include prev_events with no state amongst missing events

* Not finished refactor

* Fix test

* Remove isInboundTxn

* Remove debug logging
2020-10-12 15:56:15 +01:00
Neil Alexander
8035c50c06
Don't get into situations where we have no forward extremities 2020-10-08 14:46:25 +01:00
Neil Alexander
78f6e1a31e
Don't set new state NID if state regression 2020-10-08 13:30:30 +01:00
Kegsay
a846dad0e1
Return what we have when we encounter missing events when servicing backfill/gme (#1499)
We expect to have missing events as we walk back in the DAG over federation
as we didn't always create the room. When checking if the server is allowed
to see those events, just give up and stop rather than fail the request.
2020-10-08 12:22:00 +01:00
Neil Alexander
429bd48129
Return a non-fatal error to the federation API on a state regression (#1498)
* Return a non-fatal error to the federation API on a state regression

* Return no error but don't morph state
2020-10-08 12:13:50 +01:00
Kegsay
bf7e85848b
Rename serverkeyapi to signingkeyserver (#1492)
* Rename serverkeyapi to signingkeyserver

We use "api" for public facing stuff and "server" for internal stuff.
As the server key API is internal only, we call it 'signing key server',
which also clarifies the type of key (as opposed to TLS keys, E2E keys, etc)

* Convert docker/scripts to use signing-key-server

* Rename missed bits
2020-10-07 16:23:18 +01:00
Neil Alexander
d821f9d3c9
Deep checking of forward extremities (#1491)
* Deep forward extremity calculation

* Use updater txn

* Update error

* Update error

* Create previous event references in StoreEvent

* Use latest events updater to row-lock prev events

* Fix unexpected fallthrough

* Fix deadlock

* Don't roll back

* Update comments in calculateLatest

* Don't include events that we can't find references for in the forward extremities

* Add another passing test
2020-10-07 14:05:33 +01:00
Neil Alexander
f7c15071de
Don't return 500s on checking to see if a remote server is allowed to see an event we don't know about (#1490) 2020-10-07 10:30:27 +01:00
Kegsay
0f7e707f39
Optimise servers to backfill from (#1485)
- Prefer perspective servers if they are in the room.
- Limit the number of backfill servers to 5 to avoid taking too long.
2020-10-06 18:09:02 +01:00
Neil Alexander
4feff8e8d9
Don't give up if we fail to fetch a key (#1483)
* Don't give up if we fail to fetch a key

* Fix logging line

* furl nolint
2020-10-06 17:59:08 +01:00
Neil Alexander
bf90db5b60
Remove KindRewrite (#1481)
* Don't send rewrite events

* Remove final traces of rewrite events

* Remove test that is no longer needed

* Revert "Remove test that is no longer needed"

This reverts commit 9a45babff690480acd656a52f2c2950a5f7e9ada.

* Update test to use KindOutlier
2020-10-06 11:05:00 +01:00
Neil Alexander
2e71d2708f
Resolve state after event against current room state when determining latest state changes (#1479)
* Resolve state after event against current room state when determining latest state changes

* Update sytest-whitelist

* Update sytest-whitelist, blacklist
2020-10-05 17:47:08 +01:00
S7evinK
3e01db0049
Fix golangci-lint issues (#1464)
* Fix S1039: unnecessary use of fmt.Sprintf

* Fix S1036: unnecessary guard around map access

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
2020-10-01 20:00:56 +01:00
Neil Alexander
91fc1f1c92
Fix bug in error handling in SQLite InsertPreviousEvent (#1456) 2020-10-01 10:55:00 +01:00
Neil Alexander
738b829a23
Fetch missing auth events, implement QueryMissingAuthPrevEvents, try other servers in room for /event and /get_missing_events (#1450)
* Try to ask other servers in the room for missing events if the origin won't provide them

* Logging

* More logging

* Implement QueryMissingAuthPrevEvents

* Try to get missing auth events badly

* Use processEvent

* Logging

* Update QueryMissingAuthPrevEvents

* Try to find missing auth events

* Patchy fix for test

* Logging tweaks

* Send auth events as outliers

* Update check in QueryMissingAuthPrevEvents

* Error responses

* More return codes

* Don't return error on reject/soft-fail since it was ultimately handled

* More tweaks

* More error tweaks
2020-09-29 13:40:29 +01:00
Neil Alexander
6275669e65
Set default room version to v6 (#1438) 2020-09-25 12:59:57 +01:00
Neil Alexander
3013ade84f
Reject make_join for empty rooms (#1439)
* Sanity-check room version on RS event input

* Update gomatrixserverlib

* Reject make_join when no room members are left

* Revert some changes from wrong branch

* Distinguish between room not existing and room being abandoned on this server

* nolint
2020-09-24 16:18:13 +01:00
Sam
a6700331ce
Update all usages of tx.Stmt to sqlutil.TxStmt (#1423)
* Replace all usages of txn.Stmt with sqlutil.TxStmt

Signed-off-by: Sam Day <me@samcday.com>

* Fix sign off link in PR template.

Signed-off-by: Sam Day <me@samcday.com>

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-09-24 11:10:14 +01:00
Neil Alexander
45de9dc1c0
Use room version cache in Events() 2020-09-21 16:49:37 +01:00
Neil Alexander
a06c18bb56
Soft-fail (#1364)
* Initial work on soft-fail

* Fix state block retrieval

* Copy-pasta QueryLatestEventsAndState code

* Fix state lookup

* Clean up

* Fix up failing sytest

* Linting

* Update previous events SQLite insert query

* Update SQLite InsertPreviousEvent properly

* Hopefully fix the event references updates

Co-authored-by: Kegan Dougal <kegan@matrix.org>
2020-09-21 14:55:46 +01:00
Kegsay
18231f25b4
Implement rejected events (#1426)
* WIP Event rejection

* Still send back errors for rejected events

Instead, discard them at the federationapi /send layer rather than
re-implementing checks at the clientapi/PerformJoin layer.

* Implement rejected events

Critically, rejected events CAN cause state resolution to happen
as it can merge forks in the DAG. This is fine, _provided_ we
do not add the rejected event when performing state resolution,
which is what this PR does. It also fixes the error handling
when NotAllowed happens, as we were checking too early and needlessly
handling NotAllowed in more than one place.

* Update test to match reality

* Modify InputRoomEvents to no longer return an error

Errors do not serialise across HTTP boundaries in polylith mode,
so instead set fields on the InputRoomEventsResponse. Add `Err()`
function to make the API shape basically the same.

* Remove redundant returns; linting

* Update blacklist
2020-09-16 13:00:52 +01:00
Kegan Dougal
ba6c7c4a5c Disable prometheus to unbreak tests 2020-09-15 16:15:34 +01:00
Neil Alexander
965f068d1a
Handle state with input event as new events (#1415)
* SendEventWithState events as new

* Use cumulative state IDs for final event

* Error wrapping in calculateAndSetState

* Handle overwriting same event type and state key

* Hacky way to spot historical events

* Don't exclude from sync

* Don't generate output events when rewriting forward extremities

* Update output event check

* Historical output events

* Define output room event type

* Notify key changes on state

* Don't send our membership event twice

* Deduplicate state entries

* Tweaks

* Remove unnecessary nolint

* Fix current state upsert in sync API

* Send auth events as outliers, state events as rewrite

* Sync API don't consume state events

* Process events actually

* Improve outlier check

* Fix local room check

* Remove extra room check, it seems to break the whole damn world

* Fix federated join check

* Fix nil pointer exception

* Better comments on DeduplicateStateEntries

* Reflow forced federated joins

* Don't force federated join for possibly even local invites

* Comment SendEventWithState better

* Rewrite room state in sync API storage

* Add TODO

* Clean up all room data when receiving create event

* Don't generate output events for rewrites, but instead notify that state is rewritten on the final new event

* Rename to PurgeRoom

* Exclude backfilled messages from /sync

* Split out rewriting state from updating state from state res

Co-authored-by: Kegan Dougal <kegan@matrix.org>
2020-09-15 11:17:46 +01:00
Matthew Hodgson
39507bacc3
Peeking via MSC2753 (#1370)
Initial implementation of MSC2753, as tested by https://github.com/matrix-org/sytest/pull/944.
Doesn't yet handle unpeeks, peeked EDUs, or history viz changing during a peek - these will follow.
https://github.com/matrix-org/dendrite/pull/1370 has full details.
2020-09-10 14:39:18 +01:00
Neil Alexander
35564dd73c
Process membership updates in writers (#1414) 2020-09-08 17:48:07 +01:00
Kegsay
7913759921
Remove QueryBulkStateContent from current state server (#1404)
* Remove QueryBulkStateContent from current state server

Expected fail due to db impl not existing

* Implement query bulk state content

* Fix up rejecting invites over federation

* Fix bulk content marshalling
2020-09-07 12:38:09 +01:00
Neil Alexander
b9caccbce8
Update GetStateEvent behaviour (#1399) 2020-09-04 19:40:21 +01:00
Kegsay
ca8dcf46b7
Remove QuerySharedUsers from current state server (#1396)
* Remove QuerySharedUsers from current state server

* Bugfixes
2020-09-04 14:25:01 +01:00
Kegsay
67cc8619d3
Remove QueryKnownUsers from current state server (#1393)
* Remove QueryKnownUsers from current state server

* Fix HTTP mode
2020-09-04 11:46:01 +01:00
Kegsay
2570418f42
Remove ServerACLs from the current state server (#1390)
* Remove ServerACLs from the current state server

Functionality moved to roomserver

* Nothing to see here, move along
2020-09-04 10:40:58 +01:00
Neil Alexander
f1a98e1193
Fix nil txn bug 2020-09-04 10:22:32 +01:00
Kegsay
33b8143a95
Implement more CSS storage functions in roomserver (#1388) 2020-09-03 18:27:02 +01:00
Kegsay
b20386123e
Move currentstateserver API to roomserver (#1387)
* Move currentstateserver API to roomserver

Stub out DB functions for now, nothing uses the roomserver version yet.

* Allow it to startup

* Implement some current-state-server storage interface functions

* Add missing package
2020-09-03 17:20:54 +01:00
Neil Alexander
6150de6cb3
FIFO ordering of input events (#1386)
* Initial FIFOing of roomserver inputs

* Remove EventID response from api.InputRoomEventsResponse

* Don't send back event ID unnecessarily

* Fix ordering hopefully

* Reduce copies, use buffered task channel to reduce contention on other rooms

* Fix error handling
2020-09-03 15:22:16 +01:00
Kegsay
9d9e854fe0
Add Queryer and Inputer and factor out more RSAPI stuff (#1382)
* Add Queryer and use embedded structs

* Add Inputer and factor out more RS API stuff

This neatly splits up the RS API based on the functionality it provides,
whilst providing a useful place for code sharing via the `helpers` package.
2020-09-02 17:13:15 +01:00
Neil Alexander
096191ca24
Use federation sender for backfill/getting missing events (#1379)
* Use federation sender for backfill and getting missing events

* Fix internal URL paths

* Update go.mod/go.sum for matrix-org/gomatrixserverlib#218

* Add missing server implementations in HTTP interface
2020-09-02 15:26:30 +01:00
Kegsay
e473320e73
Refactor roomserver/internal - split perform stuff out (#1380)
- New package `perform` which contains all `Perform` functions
- New package `helpers` which contains helper functions used by both
  perform and query/input functions.
- Perform invite/leave have no idea how to `WriteOutputEvents` and this
  is now returned from `PerformInvite` or `PerformLeave` respectively.

Still to do:
 - RSAPI is fed into the inviter/joiner/leaver - this introduces circular
   logic so will need to be removed.
 - Put query operations in a `query` package.
 - Put input operations (and output) in an `input` package.
 - Factor out helper functions as much as possible, possibly rejigging the
   storage layer in the process.
2020-09-02 13:47:31 +01:00
Kegsay
02a73f29f8
Expand RoomInfo to cover more DB storage functions (#1377)
* Factor more things to RoomInfo

* Factor out remaining bits for RoomInfo

* Linting for now
2020-09-02 10:02:48 +01:00
Neil Alexander
a9f4d83d30
Fix duplicate writers (#1376)
* Fix writers

* Don't use writers in both shared and sqlite3
2020-09-01 16:58:21 +01:00
Kegsay
6d79f04354
Add RoomInfo metadata struct (#1367)
* Add RoomInfo struct

* Remove RoomNID and replace with RoomInfo

* Bugfix and remove another needless query

* nil guard
2020-09-01 12:40:49 +01:00
Neil Alexander
034e5d5a55
Speed up accepting invites from remote users if we're already in the room (#1349) 2020-08-27 11:47:13 +01:00
oliverpool
3205b9212d
[readability] use event.StateKeyEquals where relevant and minor for-loop refactoring (#1339)
* use event.StateKeyEquals where relevant

Signed-off-by: Olivier Charvin <git@olivier.pfad.fr>

* use userID
2020-08-25 21:04:35 +01:00
Neil Alexander
819b926820
Make redactions permanent (#1337)
* Make redactions permanent

* Update storage.go
2020-08-25 15:44:19 +01:00