dendrite/roomserver
Till 5f872f4a82
Fix panic in QueryNextRoomHierarchyPage (#3253)
Sentry reported the following panic:
```
time="2023-11-01T01:33:56.220583478Z" level=error msg="Request panicked!
goroutine 43763845 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:24 +0x5e
github.com/matrix-org/dendrite/internal/httputil.MakeExternalAPI.MakeJSONAPI.Protect.func3.1()
	github.com/matrix-org/util@v0.0.0-20221111132719-399730281e66/json.go:98 +0x13e
panic({0x15b5540?, 0x2453560?})
	runtime/panic.go:914 +0x21f
github.com/matrix-org/dendrite/internal/httputil.MakeAuthAPI.func1.1()
	github.com/matrix-org/dendrite/internal/httputil/httpapi.go:91 +0x4a
panic({0x15b5540?, 0x2453560?})
	runtime/panic.go:914 +0x21f
github.com/matrix-org/dendrite/roomserver/internal/query.(*Queryer).QueryNextRoomHierarchyPage(0x413185?, {0x1a576e0, 0xc0436705a0}, {{{0xc01e5fd260, 0x1f}, {0xc01e5fd261, 0x12}, {0xc01e5fd274, 0xb}}, {0xc145cb5200, ...}, ...}, ...)
	github.com/matrix-org/dendrite/roomserver/internal/query/query_room_hierarchy.go:116 +0xbfe
github.com/matrix-org/dendrite/clientapi/routing.QueryRoomHierarchy(0xc0be13b200, 0xc144e65dd0, {0xc01e5fd260?, 0x6?}, {0x7faf140639c8, 0xc00059af20}, 0xc08adca000?)
	github.com/matrix-org/dendrite/clientapi/routing/room_hierarchy.go:141 +0x68b
github.com/matrix-org/dendrite/clientapi/routing.Setup.func35(0xc03e7d5c20?, 0x17c3a57?)
	github.com/matrix-org/dendrite/clientapi/routing/routing.go:534 +0xbe
github.com/matrix-org/dendrite/internal/httputil.MakeAuthAPI.func1(0xc0bd097300)
	github.com/matrix-org/dendrite/internal/httputil/httpapi.go:108 +0x5ed
github.com/matrix-org/util.(*jsonRequestHandlerWrapper).OnIncomingRequest(0xc0bd097200?, 0xc13b7d6fc0?)
	github.com/matrix-org/util@v0.0.0-20221111132719-399730281e66/json.go:79 +0x19
github.com/matrix-org/dendrite/internal/httputil.MakeExternalAPI.MakeJSONAPI.func2({0x1a54880, 0xc138f28b60}, 0xc0bd097200?)
	github.com/matrix-org/util@v0.0.0-20221111132719-399730281e66/json.go:141 +0xaa
github.com/matrix-org/dendrite/internal/httputil.MakeExternalAPI.MakeJSONAPI.Protect.func3({0x1a54880?, 0xc138f28b60?}, 0x17c01d9?)
	github.com/matrix-org/util@v0.0.0-20221111132719-399730281e66/json.go:103 +0x63
net/http.HandlerFunc.ServeHTTP(...)
	net/http/server.go:2136
github.com/matrix-org/dendrite/internal/httputil.MakeExternalAPI.func1({0x1a54880?, 0xc138f28b60?}, 0xc0bd097100)
	github.com/matrix-org/dendrite/internal/httputil/httpapi.go:191 +0x411
net/http.HandlerFunc.ServeHTTP(0xc0bd097000?, {0x1a54880?, 0xc138f28b60?}, 0xbe1348905308878e?)
	net/http/server.go:2136 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000000000, {0x1a54880, 0xc138f28b60}, 0xc0bd096f00)
	github.com/gorilla/mux@v1.8.0/mux.go:210 +0x1c5
github.com/matrix-org/dendrite/setup/base.SetupAndServeHTTP.(*Handler).Handle.(*Handler).handle.func5({0x1a54880, 0xc138f28b60}, 0xc0bd096e00)
	github.com/getsentry/sentry-go@v0.14.0/http/sentryhttp.go:103 +0x298
net/http.HandlerFunc.ServeHTTP(0xc0bd096a00?, {0x1a54880?, 0xc138f28b60?}, 0x7fae6812f5d0?)
	net/http/server.go:2136 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000000a80, {0x1a54880, 0xc138f28b60}, 0xc0bd096900)
	github.com/gorilla/mux@v1.8.0/mux.go:210 +0x1c5
net/http.serverHandler.ServeHTTP({0xc02884c4e0?}, {0x1a54880?, 0xc138f28b60?}, 0x6?)
	net/http/server.go:2938 +0x8e
net/http.(*conn).serve(0xc1926922d0, {0x1a576e0, 0xc024a6ec90})
	net/http/server.go:2009 +0x5f4
created by net/http.(*Server).Serve in goroutine 16979
	net/http/server.go:3086 +0x5cb
" context=missing panic="runtime error: invalid memory address or nil pointer dereference"
```

[skip ci]
2023-11-08 14:22:02 +01:00
..
acls Update gmsl to use new validated RoomID on PDUs (#3200) 2023-09-15 14:39:06 +00:00
api Update gmsl to use new validated RoomID on PDUs (#3200) 2023-09-15 14:39:06 +00:00
auth Update gmsl to use new validated RoomID on PDUs (#3200) 2023-09-15 14:39:06 +00:00
internal Fix panic in QueryNextRoomHierarchyPage (#3253) 2023-11-08 14:22:02 +01:00
producers PDU Sender split (#3100) 2023-06-06 20:55:18 +00:00
state Check event is not rejected (#3243) 2023-10-25 09:47:21 +02:00
storage Check event is not rejected (#3243) 2023-10-25 09:47:21 +02:00
types de-MSC-ifying space summaries (MSC2946) (#3134) 2023-07-20 15:06:05 +01:00
version Add config key for default room version (#3171) 2023-08-08 14:20:05 +01:00
README.md use go module for dependencies (#594) 2019-05-21 21:56:55 +01:00
roomserver_test.go Some tweaks for the device list updater (#3251) 2023-10-31 16:39:45 +01:00
roomserver.go de-MSC-ifying space summaries (MSC2946) (#3134) 2023-07-20 15:06:05 +01:00

RoomServer

RoomServer Internals

Numeric IDs

To save space matrix string identifiers are mapped to local numeric IDs. The numeric IDs are more efficient to manipulate and use less space to store. The numeric IDs are never exposed in the API the room server exposes. The numeric IDs are converted to string IDs before they leave the room server. The numeric ID for a string ID is never 0 to avoid being confused with go's default zero value. Zero is used to indicate that there was no corresponding string ID. Well-known event types and event state keys are preassigned numeric IDs.

State Snapshot Storage

The room server stores the state of the matrix room at each event. For efficiency the state is stored as blocks of 3-tuples of numeric IDs for the event type, event state key and event ID. For further efficiency the state snapshots are stored as the combination of up to 64 these blocks. This allows blocks of the room state to be reused in multiple snapshots.

The resulting database tables look something like this:

+-------------------------------------------------------------------+
| Events                                                            |
+---------+-------------------+------------------+------------------+
| EventNID| EventTypeNID      | EventStateKeyNID | StateSnapshotNID |
+---------+-------------------+------------------+------------------+
|       1 | m.room.create   1 | ""             1 | <nil>          0 |
|       2 | m.room.member   2 | "@user:foo"    2 | <nil>          0 |
|       3 | m.room.member   2 | "@user:bar"    3 | {1,2}          1 |
|       4 | m.room.message  3 | <nil>          0 | {1,2,3}        2 |
|       5 | m.room.member   2 | "@user:foo"    2 | {1,2,3}        2 |
|       6 | m.room.message  3 | <nil>          0 | {1,3,6}        3 |
+---------+-------------------+------------------+------------------+

+----------------------------------------+
| State Snapshots                        |
+-----------------------+----------------+
| EventStateSnapshotNID | StateBlockNIDs |
+-----------------------+----------------|
|                     1 |           {1}  |
|                     2 |         {1,2}  |
|                     3 |       {1,2,3}  |
+-----------------------+----------------+

+-----------------------------------------------------------------+
| State Blocks                                                    |
+---------------+-------------------+------------------+----------+
| StateBlockNID | EventTypeNID      | EventStateKeyNID | EventNID |
+---------------+-------------------+------------------+----------+
|             1 | m.room.create   1 | ""             1 |        1 |
|             1 | m.room.member   2 | "@user:foo"    2 |        2 |
|             2 | m.room.member   2 | "@user:bar"    3 |        3 |
|             3 | m.room.member   2 | "@user:foo"    2 |        6 |
+---------------+-------------------+------------------+----------+