From 855f8628e1c615bd84f162607d7f329d47a89f3b Mon Sep 17 00:00:00 2001 From: Mark Haines Date: Fri, 20 Jan 2017 18:25:51 +0000 Subject: [PATCH] Add README --- README.md | 102 +++++++++++++++++++++++++++++ WIRING.md | 192 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 294 insertions(+) create mode 100644 README.md create mode 100644 WIRING.md diff --git a/README.md b/README.md new file mode 100644 index 00000000..11c1858a --- /dev/null +++ b/README.md @@ -0,0 +1,102 @@ +# Dendrite + +Dendrite will be a matrix homeserver written in go. + +# Design + + * + + +## Log Based Architecture + +### Decomposition and Decoupling + +A matrix homeserver can be built around append-only event logs built from the +messages, receipts, presence, typing notifications, device messages and other +events sent by users on the homeservers or by other homeservers. + +The server would then decompose into two categories: writers that add new +entries to the logs and readers that read those entries. + +The event logs then serve to decouple the two components, the writers and +readers need only agree on the format of the entries in the event log. +This format could be largely dervived from the wire format of the events used +in the client and federation protocols: + + + C-S API +---------+ Event Log +---------+ C-S API + ---------> | |+ (e.g. kafka) | |+ ---------> + | Writers || =============> | Readers || + ---------> | || | || ---------> + S-S API +---------+| +---------+| S-S API + +---------+ +---------+ + +However the way matrix handles state events in a room creates a few +complications for this model. + + 1) Writers require the room state at an event to check if it is allowed. + 2) Readers require the room state at an event to determine the users and + servers that are allowed to see the event. + 3) A client can query the current state of the room from a reader. + +The writers and readers cannot extract the necessary information directly from +the event logs because it would take to long to extract the information as the +state is built up by collecting individual state events from the event history. + +The writers and readers therefore need access to something that stores copies +of the event state in a form that can be efficiently queried. One possibility +would be for the readers and writers to maintain copies of the current state +in local databases. A second possibility would be to add a dedicated component +that maintained the state of the room and exposed an API that the readers and +writers could query to get the state. The second has the advantage that the +state is calculated and stored in a single location. + + + C-S API +---------+ Log +--------+ Log +---------+ C-S API + ---------> | |+ ======> | | ======> | |+ ---------> + | Writers || | Room | | Readers || + ---------> | || <------ | Server | ------> | || ---------> + S-S API +---------+| Query | | Query +---------+| S-S API + +---------+ +--------+ +---------+ + + +The room server can annotate the events it logs to the readers with room state +so that the readers can avoid querying the room server unnecessarily. + +[This architecture can be extended to cover most of the APIs.](WIRING.md) + +# TODO + + - [ ] gomatrixlib + - [x] Canonical JSON. + - [x] Signed JSON. + - [ ] Event hashing. + - [ ] Event signing. + - [x] Federation server discovery. + - [x] Federation key lookup. + - [ ] Federation request signing. + - [x] Event authentication. + - [ ] Event visibility. + - [ ] State resolution. + - [ ] Third party invites authentication. + - [ ] Room Server + - [ ] Inputing new events from logs. + - [ ] Inputing backfilled events from logs. + - [ ] Outputing events and current state to logs. + - [ ] Querying state at an event. + - [ ] Querying current forward extremities and state. + - [ ] Querying message history. + - [ ] Exporting/importing messages from other servers. + - [ ] Other Room Server stuff. + - [ ] Client Room Send + - [ ] Handling /client/r0/room/... HTTP PUTs + - [ ] Talk to remote servers for joins to remote servers. + - [ ] Outputting new events to logs. + - [ ] Updating the last active time in presence. + - [ ] Other Client Room Send stuff. + - [ ] Client Sync + - [ ] Inputing new room events and state from the logs. + - [ ] Handling /client/r0/sync HTTP GETs + - [ ] Outputing whether the client is syncing to the logs. + - [ ] Other Client Sync Stuff. + - [ ] Other Components. diff --git a/WIRING.md b/WIRING.md new file mode 100644 index 00000000..5bd1485c --- /dev/null +++ b/WIRING.md @@ -0,0 +1,192 @@ +# Wiring + +The diagram is incomplete. The following things aren't shown on the diagram: + + - [ ] Device Messages + - [ ] User Profiles + - [ ] Notification Counts + - [ ] Sending federation. + - [ ] Querying federation. + - [ ] Other things that aren't shown on the diagram. + +Diagram: + + + W -> Writer + S -> Server/Store/Service/Something/Stuff + R -> Reader + + +---+ +---+ +---+ + +----------| W | +----------| S | +--------| R | + | +---+ | Receipts +---+ | Client +---+ + | Federation |>=========================================>| Server |>=====================>| Sync | + | Send | | | | | + | | +---+ | | | | + | | +--------| W | | | | | + | | | Client +---+ | | | | + | | | Receipt |>=====>| | | | + | | | Updater | | | | | + | | +----------+ | | | | + | | | | | | + | | +---+ +---+ | | +---+ | | + | | +------------| W | +------| S | | | +--------| R | | | + | | | Federation +---+ | Room +---+ | | | Client +---+ | | + | | | Backfill |>=====>| Server |>=====>| |>=====>| Push | | | + | | +--------------+ | | +------------+ | | | | + | | | | | | | | + | | | |>==========================>| | | | + | | | | +----------+ | | + | | | | | | + | | | | +---+ | | + | | | | +--------| R | | | + | | | | | Client +---+ | | + | |>========================>| |>==========================>| Search | | | + | | | | | | | | + | | | | +----------+ | | + | | | | | | + | | | |>==========================================>| | + | | | | | | + | | +---+ | | +---+ | | + | | +--------| W | | | +----------| S | | | + | | | Client +---+ | | | Presence +---+ | | + | | | Room |>=====>| |>=====>| Server |>=====================>| | + | | | Send | +--------+ | | | | + | | | | | | | | + | | | |>======================>| |<=====================<| | + | | +----------+ | | | | + | | | | | | + | | +---+ | | | | + | | +--------| W | | | | | + | | | Client +---+ | | | | + | | | Presence |>=====>| | | | + | | | Setter | | | | | + | | +----------+ | | | | + | | | | | | + | | | | | | + | |>=========================================>| | | | + | | +------------+ | | + | | | | + | | +---+ | | + | | +----------| S | | | + | | | Typing +---+ | | + | |>=========================================>| Server |>=====================>| | + +------------+ | | +----------+ + +---+ | | + +--------| W | | | + | Client +---+ | | + | Typing |>=====>| | + | Setter | | | + +----------+ +------------+ + + +# Component Descriptions + +Many of the components are logical rather than physical. For example it is +possible that all of the client API writers will end up being glued together +and always deployed as a single unit. + +Outbound federation requests will probably need to be funnelled through a +choke-point to implement ratelimiting and backoff correctly. + +## Federation Send + + * Handles `/federation/v1/send/` requests. + * Fetches missing ``prev_events`` from the remote server if needed. + * Fetches missing room state from the remote server if needed. + * Checks signatures on remote events, downloading keys if needed. + * Queries information needed to process events from the Room Server. + * Writes room events to logs. + * Writes presence updates to logs. + * Writes receipt updates to logs. + * Writes typing updates to logs. + * Writes other updates to logs. + +## Client Room Send + + * Handles puts to `/client/v1/rooms/` that create room events. + * Queries information needed to process events from the Room Server. + * Talks to remote servers if needed for joins and invites. + * Writes room event pdus. + * Writes presence updates to logs. + +## Client Presence Setter + + * Handles puts to whatever the client API path for presence is? + * Writes presence updates to logs. + +## Client Typing Setter + + * Handles puts to whatever the client API path for typing is? + * Writes typing updates to logs. + +## Client Receipt Updater + + * Handles puts to whatever the client API path for receipts is? + * Writes typing updates to logs. + +## Federation Backfill + + * Backfills events from other servers + * Writes the resulting room events to logs. + * Is a different component from the room server itself cause it'll + be easier if the room server component isn't making outbound HTTP requests + to remote servers + +## Room Server + + * Reads new and backfilled room events from the logs written by FS, FB and CRS. + * Tracks the current state of the room and the state at each event. + * Probably does auth checks on the incoming events. + * Handles state resolution as part of working out the current state and the + * state at each event. + * Writes updates to the current state and new events to logs. + * Shards by room ID. + +## Receipt Server + + * Reads new updates to receipts from the logs written by the FS and CRU. + * Somehow learns enough information from the room server to workout how the + current receipt markers move with each update. + * Writes the new marker positions to logs + * Shards by room ID? + * It may be impossible to implement without folding it into the Room Server + forever coupling the components together. + +## Typing Server + + * Reads new updates to typing from the logs written by the FS and CTS. + * Updates the current list of people typing in a room. + * Writes the current list of people typing in a room to the logs. + * Shards by room ID? + +## Presence Server + + * Reads the current state of the rooms from the logs to track the intersection + of room membership between users. + * Reads updates to presence from the logs writen by the FS and the CPS. + * Reads when clients sync from the logs from the Client Sync. + * Tracks any timers for users. + * Writes the changes to presence state to the logs. + * Shards by user ID somehow? + +## Client Sync + + * Handle /client/v2/sync requests. + * Reads new events and the current state of the rooms from logs written by the Room Server. + * Reads new receipts positions from the logs written by the Receipts Server. + * Reads changes to presence from the logs written by the Presence Server. + * Reads changes to typing from the logs written by the Typing Server. + * Writes when a client starts and stops syncing to the logs. + +## Client Search + + * Handle whatever the client API path for event search is? + * Reads new events and the current state of the rooms from logs writeen by the Room Server. + * Maintains a full text search index of somekind. + +## Client Push + + * Pushes unread messages to remote push servers. + * Reads new events and the current state of the rooms from logs writeen by the Room Server. + * Reads the position of the read marker from the Receipts Server. + * Makes outbound HTTP hits to the push server for the client device.