6942ee1de0
* Upgrade gomatrixserverlib dependency Signed-off-by: Thibaut CHARLES cromfr@gmail.com * Added missing passing sytest Signed-off-by: Thibaut CHARLES cromfr@gmail.com * Fix login using identifier key Not a full fix, it only really supports logging in with the localpart of an mxid. Signed-off-by: Serra Allgood <serra@allgood.dev> * Replace deprecated prometheus.InstrumentHandler and unsafe time.Ticker * goimports * re-add temporarily missing deps? * Refactor InstrumentHandlerCounter definition * URL decode args * Return server names (#833) * Remove unnecessary map->array processing * Return server names in room federation directory query * Knock off a TODO * Fix /send_join and /send_leave (#821) Fix the /send_join and /send_leave endpoints, so that they use the v2 endpoints as mandated by MSC1802. Also comment out the SyTest tests that are failing because of lack of support for the v1 endpoints. * Refuse /send_join without m.room.create (#824) Signed-off-by: Abhishek Kumar <abhishekkumar2718@gmail.com> * AS should use the v1 endpoint, rather than r0 (#827) * docker: Passthrough parameters to dendrite-monolith-server * Fix copy & paste error (#812) * Use gomatrixserverlib.Transaction instead of local type (#590) (#811) * Move files back if linting fails (#810) * replaced gometalinter description with golangci-lint (#837) * Amend syncapi SQL queries to return missing columns (#840) * This commit updates a couple of the syncapi SQL queries to return additional columns that are required/expected by rowsToStreamEvents in output_room_events_table.go. It's not exactly clear to me yet what transaction_id and session_id do, but these being added n #367 results in state events breaking the /sync endpoint. This is a temporary fix. We need to come up with a better solution. * gomatrix to gomatrixserverlib on some weird line change * Tweaks from @babolivier review comments * Implement storage interfaces (#841) * Implement interfaces for federationsender storage * Implement interfaces for mediaapi storage * Implement interfaces for publicroomsapi storage * Implement interfaces for roomserver storage * Implement interfaces for syncapi storage * Implement interfaces for keydb storage * common.PartitionStorer in publicroomsapi interface * Update copyright notices * make cmd directory path absolute in build.sh (#830) * Resync testfile with current sytest pass/fail (#832) * Resync testfile with current sytest pass/fail * Add displayname test * Fall back to postgres when database connection string parsing fails (#842) * Fall back to postgres when parsing the database connection string for a URI schema fails * Fix behaviour so that it really tries postgres when URL parsing fails and it complains about unknown schema if it succeeds * Fix #842 * Fix #842 - again... * Federation fixes (#845) * Update gomatrixserverlib to p2p commit 92c0338, other tweaks * Update gomatrixserverlib to p2p commit e5dcc65 * Rewrite getAuthChain * Update gomatrixserverlib in go.mod/go.sum * Correct a couple of package refs for updated gmsl/gomatrix * Update gomatrixserverlib ref in go.mod/go.sum * Update getAuthChain comments following @babolivier review * Add a Sytest blacklist file (#849) * Add more passing tests to the testfile, add test blacklist file (#848) * CS API: Support for /messages, fixes for /sync (#847) * Merge forward * Tidy up a bit * TODO: What to do with NextBatch here? * Replace SyncPosition with PaginationToken throughout syncapi * Fix PaginationTokens * Fix lint errors * Add a couple of missing functions into the syncapi external storage interface * Some updates based on review comments from @babolivier * Some updates based on review comments from @babolivier * argh whitespacing * Fix opentracing span * Remove dead code * Don't overshadow err (fix lint issue) * Handle extremities after inserting event into topology * Try insert event topology as ON CONFLICT DO NOTHING * Prevent OOB error in addRoomDeltaToResponse * Thwarted by gocyclo again * Fix NewPaginationTokenFromString, define unit test for it * Update pagination token test * Update sytest-whitelist * Hopefully fix some of the sync batch tokens * Remove extraneous sync position func * Revert to topology tokens in addRoomDeltaToResponse etc * Fix typo * Remove prevPDUPos as dead now that backwardTopologyPos is used instead * Fix selectEventsWithEventIDsSQL * Update sytest-blacklist * Update sytest-whitelist * Some fixes for #847 (#850) * Fix a couple of cases where backfilling events we already had causes panics, hopefully fix ordering of events, update GMSL dependency for backfill URL fixes * Remove commented out lines from output_room_events_table schema * Wire up publicroomsapi for roomserver events (#851) * Wire up publicroomsapi to roomserver events * Remove parameter that was incorrectly brought over from p2p work * nolint containsBackwardExtremity for now * Store our own keys in the keydb (#853) * Store our own keys in the keydb The DirectKeyFetcher makes the assumption that you can always reach the key/v2/server endpoint of any server, including our own. We previously haven't bothered to store our own keys in the keydb so this would mean we end up making key requests to ourselves. In the libp2p world as an example, self-dialling is not possible, therefore this would render it impossible to get our own keys. This commit adds our own keys into the keydb so that we don't create unnecessarily (and maybe impossible) requests. * Use golang.org/x/crypto/ed25519 instead of crypto/ed25519 for pre-Go 1.13 * More sync fixes (#854) * Further sync tweaks * Remove unnecessary blank line * getBackwardTopologyPos always returns a usable value * Revert order fixing * Implement GET endpoints for account_data in clientapi (#861) * Implement GET endpoints for account_data in clientapi * Fix accountDB parameter * Remove fmt.Println * Add empty push rules into account data on account creation (#862) * Handle kind=guest query parameter on /register (#860) * Handle kind=guest query parameter on /register * Reorganized imports * Pass device_id as nil * Added tests to systest-whitelist * Update sytest-whitelist * Blacklist 'displayname updates affect room member events' (#859) * Room version abstractions (#865) * Rough first pass at adding room version abstractions * Define newer room versions * Update room version metadata * Fix roomserver/versions * Try to fix whitespace in roomsSchema * Implement room version capabilities in CS API (#866) * Add wiring for querying the roomserver for the default room version * Try to implement /capabilities for room versions * Update copyright notices * Update sytests, add /capabilities endpoint into CS API * Update sytest-whitelist * Add GetDefaultRoomVersion * Fix cases where state package was shadowed * Fix version formatting * Update Dockerfile to Go 1.13.6 * oh yes types I remember * And fix the default too * Update documentation for Go 1.13 (#867) * Pass cfg by reference around the codebase (#819) * Pass cfg by reference around the codebase * Merge branch 'master' into pass-cfg-by-ref Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com> * Implement missing device management features (#835) * Implement missing device management features Signed-off-by: Till Faelligen <tfaelligen@gmail.com> * Add a little more documentation * Undo changes * Use non-anonymous struct to decode devices list * Update sytest-whitelist * Update sytest-whitelist * Update sytest-blacklist Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com> * Adding sslmode: disable to sytest server config (#813) Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com> * Fix AppService bind addrs in test (#805) Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com> * Always defer *sql.Rows.Close and consult with Err (#844) * Always defer *sql.Rows.Close and consult with Err database/sql.Rows.Next() makes sure to call Close only after exhausting result rows which would NOT happen when returning early from a bad Scan. Close being idempotent makes it a great candidate to get always deferred regardless of what happens later on the result set. This change also makes sure call Err() after exhausting Next() and propagate non-nil results from it as the documentation advises. Closes #764 Signed-off-by: Kiril Vladimiroff <kiril@vladimiroff.org> * Override named result parameters in last returns Signed-off-by: Kiril Vladimiroff <kiril@vladimiroff.org> * Do the same over new changes that got merged Signed-off-by: Kiril Vladimiroff <kiril@vladimiroff.org> Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com> * Clean up Co-authored-by: Serra Allgood <serra@allgood.dev> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com> Co-authored-by: Brendan Abolivier <github@brendanabolivier.com> Co-authored-by: Abhishek Kumar <31231064+abhishekkumar2718@users.noreply.github.com> Co-authored-by: Will Hunt <will@half-shot.uk> Co-authored-by: S7evinK <tfaelligen@gmail.com> Co-authored-by: Arshpreet <30545756+arsh-7@users.noreply.github.com> Co-authored-by: Prateek Sachan <42961174+prateek2211@users.noreply.github.com> Co-authored-by: Behouba Manassé <behouba@gmail.com> Co-authored-by: aditsachde <23707194+aditsachde@users.noreply.github.com> Co-authored-by: Kiril Vladimiroff <kiril@vladimiroff.org> |
||
---|---|---|
.. | ||
api | ||
consumers | ||
routing | ||
storage | ||
sync | ||
types | ||
README.md | ||
syncapi.go |
Sync API Server
This server is responsible for servicing /sync
requests. It gets its data from the room server output log. Currently, the sync server will:
- Return a valid
/sync
response for the user represented by the providedaccess_token
. - Return a "complete sync" if no
since
value is provided, and return a validnext_batch
token. This contains all rooms the user has been invited to or has joined. For joined rooms, this includes the complete current room state and the most recent 20 (hard-coded) events in the timeline. - For "incremental syncs" (a
since
value is provided), as you get invited to, join, or leave rooms they will be reflected correctly in the/sync
response. - For very large state deltas, the
state
section of a room is correctly populated with the state of the room at the start of the timeline. - When you join a room, the
/sync
which transitions your client to be "joined" will include the complete current room state as per the specification. - Only wake up user streams it needs to wake up.
- Honours the
timeout
query parameter value.
Internals
When the server gets a /sync
request, it needs to:
- Work out which rooms to return to the client.
- For each room, work out which events to return to the client.
The logic for working out which rooms is based on Synapse:
- Get the CURRENT joined room list for this user.
- Get membership list changes for this user between the provided stream position and now.
- For each room which has membership list changes:
- Check if the room is 'newly joined' (insufficient to just check for a join event because we allow dupe joins). If it is, then we need to send the full room state down (and 'limited' is always true).
- Check if user is still CURRENTLY invited to the room. If so, add room to 'invited' block.
- Check if the user is CURRENTLY left/banned. If so, add room to 'archived' block.
- Add joined rooms (joined room list)
For each room, the /sync response returns the most recent timeline events and the state of the room at the start of the timeline.
The logic for working out which events is not based entirely on Synapse code, as it is known broken with respect to working out
room state. In order to know which events to return, the server needs to calculate room state at various points in the history of
the room. For example, imagine a room with the following 15 events (letters are state events (updated via '
), numbers are timeline events):
index 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 (1-based indexing as StreamPosition(0) represents no event)
timeline [A, B, C, D, 1, 2, 3, D', 4, D'', 5, B', D''', D'''', 6]
The current state of this room is: [A, B', C, D'''']
.
If this room was requested with ?since=14&limit=5
then 1 timeline event would be returned, the most recent one:
15
[ 6 ]
If this room was requested with ?since=9&limit=5
then 5 timeline events would be returned, the most recent ones:
11 12 13 14 15
[5, B', D''', D'''', 6]
The state of the room at the START of the timeline can be represented in 2 ways:
- The
full_state
from index 0 :[A, B, C, D'']
(aka the state between 0-11 exclusive) - A partial state from index 9 :
[D'']
(aka the state between 9-11 exclusive)
Servers advance state events (e.g from D'
to D''
) based on the state conflict resolution algorithm.
You might think that you could advance the current state by just updating the entry for the (event type, state_key)
tuple
for each state event, but this state can diverge from the state calculated using the state conflict resolution algorithm.
For example, if there are two "simultaneous" updates to the same state key, that is two updates at the same depth in the
event graph, then the final result of the state conflict resolution algorithm might not match the order the events appear
in the timeline.
The correct advancement for state events is represented by the AddsStateEventIDs
and RemovesStateEventIDs
that
are in OutputRoomEvents
from the room server.
This version of the sync server uses very simple indexing to calculate room state at various points.
This is inefficient when a very old since
value is provided, or the full_state
is requested, as the state delta becomes
very large. This is mitigated slightly with indexes, but better data structures could be used in the future.
Known Issues
m.room.history_visibility
is not honoured: it is always treated as "shared".- All ephemeral events are not implemented (presence, typing, receipts).
- Account data (both user and room) is not implemented.
to_device
messages are not implemented.- Back-pagination via
prev_batch
is not implemented. - The
limited
flag can lie. - Filters are not honoured or implemented. The
limit
for each room is hard-coded to 20. - The
full_state
query parameter is not implemented. - The
set_presence
query parameter is not implemented. - "Ignored" users are not ignored.
- Redacted events are still sent to clients.
- Invites over federation (if it existed) won't work as they aren't "real" events and so won't be in the right tables.
invite_state
is not implemented (for similar reasons to the above point).- The current implementation scales badly when a very old
since
token is provided. - The entire current room state can be re-sent to the client if they send a duplicate "join" event which should be a no-op.