* Improve server selection somewhat
* Remove things from the map when we're done
* Be less panicky about auth event signatures in case they are not fatal after all
* Accept HasState in all cases
* Send join asynchronously
* Revert "Send join asynchronously"
This reverts commit 5b685bfcd0b1150a66c7b1e70fb3a3eda509efd1.
* Joins and leaves use background context
* Put federation client functions into their own file
* Look for missing auth events in RS input
* Remove retrieveMissingAuthEvents from federation API
* Logging
* Sorta transplanted the code over
* Use event origin failing all else
* Don't get stuck on mutexes:
* Add verifier
* Don't mark state events with zero snapshot NID as not existing
* Check missing state if not an outlier before storing the event
* Reject instead of soft-fail, don't copy roominfo so much
* Use synchronous contexts, limit time to fetch missing events
* Clean up some commented out bits
* Simplify `/send` endpoint significantly
* Submit async
* Report errors on sending to RS input
* Set max payload in NATS to 16MB
* Tweak metrics
* Add `workerForRoom` for tidiness
* Try skipping unmarshalling errors for RespMissingEvents
* Track missing prev events separately to avoid calculating state when not possible
* Tweak logic around checking missing state
* Care about state when checking missing prev events
* Don't check missing state for create events
* Try that again
* Handle create events better
* Send create room events as new
* Use given event kind when sending auth/state events
* Revert "Use given event kind when sending auth/state events"
This reverts commit 089d64d271b5fca8c104e1554711187420dbebca.
* Only search for missing prev events or state for new events
* Tweaks
* We only have missing prev if we don't supply state
* Room version tweaks
* Allow async inputs again
* Apply backpressure to consumers/synchronous requests to hopefully stop things being overwhelmed
* Set timeouts on roomserver input tasks (need to decide what timeout makes sense)
* Use work queue policy, deliver all on restart
* Reduce chance of duplicates being sent by NATS
* Limit the number of servers we attempt to reduce backpressure
* Some review comment fixes
* Tidy up a couple things
* Don't limit servers, randomise order using map
* Some context refactoring
* Update gmsl
* Don't resend create events
* Set stateIDs length correctly or else the roomserver thinks there are missing events when there aren't
* Exclude our own servername
* Try backing off servers
* Make excluding self behaviour optional
* Exclude self from g_m_e
* Update sytest-whitelist
* Update consumers for the roomserver output stream
* Remember to send outliers for state returned from /gme
* Make full HTTP tests less upsetti
* Remove 'If a device list update goes missing, the server resyncs on the next one' from the sytest blacklist
* Remove debugging test
* Fix blacklist again, remove unnecessary duplicate context
* Clearer contexts, don't use background in case there's something happening there
* Don't queue up events more than once in memory
* Correctly identify create events when checking for state
* Fill in gaps again in /gme code
* Remove `AuthEventIDs` from `InputRoomEvent`
* Remove stray field
Co-authored-by: Kegan Dougal <kegan@matrix.org>
* Initial federation sender -> federation API refactoring
* Move base into own package, avoids import cycle
* Fix build errors
* Fix tests
* Add signing key server tables
* Try to fold signing key server into federation API
* Fix dendritejs builds
* Update embedded interfaces
* Fix panic, fix lint error
* Update configs, docker
* Rename some things
* Reuse same keyring on the implementing side
* Fix federation tests, `NewBaseDendrite` can accept freeform options
* Fix build
* Update create_db, configs
* Name tables back
* Don't rename federationsender consumer for now
* Check membership of room
* Use QueryStateAfterEventsResponse
* Fix complexity
* Add field ShouldHitAppservice to GetRoomIDForAlias
* Hit appservice when trying to join a non-existent alias
* remove unused
* Changes that I made a long time ago
* Rename to appserviceJoinedAtEvent
* Check membership in GetMemberships
* Update QueryMembershipsForRoom
* Tweaks in client API
* Update appserviceJoinedAtEvent
* Comments
* Try QueryMembershipForUser instead
* Undo some changes to client API that shouldn't be needed
* More /event tweaks
* Refactor /event bit
* Go back to QueryMembershipsForRoom because appservices are hard
* Fix bugs in onMessage
* Add comments
* More logical naming, clean up a bit
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Add mscs/hooks package, begin work for msc2836
* Flesh out hooks and add SQL schema
* Begin implementing core msc2836 logic
* Add test harness
* Linting
* Implement visibility checks; stub out APIs for tests
* Flesh out testing
* Flesh out walkThread a bit
* Persist the origin_server_ts as well
* Edges table instead of relationships
* Add nodes table for event metadata
* LEFT JOIN to extract origin_server_ts for children
* Add graph walking structs
* Implement walking algorithm
* Add more graph walking tests
* Add auto_join for local rooms
* Fix create table syntax on postgres
* Add relationship_room_id|servers to the unsigned section of events
* Persist the parent room_id/servers in edge metadata
Other events cannot assert the true room_id/servers for the
parent event, only make claims to them, hence why this is
edge metadata.
* guts to pass through room_id/servers
* Refactor msc2836 to allow handling from federation
* Add JoinedVia to PerformJoin responses
* Fix tests; review comments
* Try to ask other servers in the room for missing events if the origin won't provide them
* Logging
* More logging
* Implement QueryMissingAuthPrevEvents
* Try to get missing auth events badly
* Use processEvent
* Logging
* Update QueryMissingAuthPrevEvents
* Try to find missing auth events
* Patchy fix for test
* Logging tweaks
* Send auth events as outliers
* Update check in QueryMissingAuthPrevEvents
* Error responses
* More return codes
* Don't return error on reject/soft-fail since it was ultimately handled
* More tweaks
* More error tweaks
* WIP Event rejection
* Still send back errors for rejected events
Instead, discard them at the federationapi /send layer rather than
re-implementing checks at the clientapi/PerformJoin layer.
* Implement rejected events
Critically, rejected events CAN cause state resolution to happen
as it can merge forks in the DAG. This is fine, _provided_ we
do not add the rejected event when performing state resolution,
which is what this PR does. It also fixes the error handling
when NotAllowed happens, as we were checking too early and needlessly
handling NotAllowed in more than one place.
* Update test to match reality
* Modify InputRoomEvents to no longer return an error
Errors do not serialise across HTTP boundaries in polylith mode,
so instead set fields on the InputRoomEventsResponse. Add `Err()`
function to make the API shape basically the same.
* Remove redundant returns; linting
* Update blacklist
* SendEventWithState events as new
* Use cumulative state IDs for final event
* Error wrapping in calculateAndSetState
* Handle overwriting same event type and state key
* Hacky way to spot historical events
* Don't exclude from sync
* Don't generate output events when rewriting forward extremities
* Update output event check
* Historical output events
* Define output room event type
* Notify key changes on state
* Don't send our membership event twice
* Deduplicate state entries
* Tweaks
* Remove unnecessary nolint
* Fix current state upsert in sync API
* Send auth events as outliers, state events as rewrite
* Sync API don't consume state events
* Process events actually
* Improve outlier check
* Fix local room check
* Remove extra room check, it seems to break the whole damn world
* Fix federated join check
* Fix nil pointer exception
* Better comments on DeduplicateStateEntries
* Reflow forced federated joins
* Don't force federated join for possibly even local invites
* Comment SendEventWithState better
* Rewrite room state in sync API storage
* Add TODO
* Clean up all room data when receiving create event
* Don't generate output events for rewrites, but instead notify that state is rewritten on the final new event
* Rename to PurgeRoom
* Exclude backfilled messages from /sync
* Split out rewriting state from updating state from state res
Co-authored-by: Kegan Dougal <kegan@matrix.org>
* Add Queryer and use embedded structs
* Add Inputer and factor out more RS API stuff
This neatly splits up the RS API based on the functionality it provides,
whilst providing a useful place for code sharing via the `helpers` package.
- New package `perform` which contains all `Perform` functions
- New package `helpers` which contains helper functions used by both
perform and query/input functions.
- Perform invite/leave have no idea how to `WriteOutputEvents` and this
is now returned from `PerformInvite` or `PerformLeave` respectively.
Still to do:
- RSAPI is fed into the inviter/joiner/leaver - this introduces circular
logic so will need to be removed.
- Put query operations in a `query` package.
- Put input operations (and output) in an `input` package.
- Factor out helper functions as much as possible, possibly rejigging the
storage layer in the process.
