Commit Graph

289 Commits

Author SHA1 Message Date
Till
f69ebc6af2
Add roomserver tests (1/?) (#2434)
* Add EventJSONTable tests

* Add eventJSON tests

* Add EventStateKeysTable tests

* Add EventTypesTable tests

* Add Events Table tests
Move variable declaration outside loops
Switch to testify/assert for tests

* Move variable declaration outside loop

* Remove random data

* Fix issue where the EventReferenceSHA256 is not set

* Add more tests

* Revert "Fix issue where the EventReferenceSHA256 is not set"

This reverts commit 8ae34c4e5f78584f0edb479f5a893556d2b95d19.

* Update GMSL

* Add tests for duplicate entries

* Test what happens if we select non-existing NIDs

* Add test for non-existing eventType

* Really update GMSL
2022-05-09 15:30:32 +02:00
Neil Alexander
79e2fbc663
Update to matrix-org/gomatrixserverlib#307 2022-05-09 13:53:51 +01:00
Neil Alexander
85c00208c5
Fix power level event auth bugs (update to matrix-org/gomatrixserverlib#306) 2022-05-06 15:41:16 +01:00
Neil Alexander
a1a5357f79
Produce more useful event auth errors (update to matrix-org/gomatrixserverlib#305) 2022-05-06 12:46:01 +01:00
Neil Alexander
e4da04e75b
Update to matrix-org/gomatrixserverlib#303 2022-05-05 14:06:05 +01:00
Neil Alexander
d9e71b93b6
Use gomatrixserverlib.Client instead of http.Client (#2421)
* Update to matrix-org/gomatrixserverlib#303

* Use `gomatrixserverlib.Client` for phone-home stats

* Use `gomatrixserverlib.Client` for push notifications

* Use `gomatrixserverlib.Client` for appservices

* Use `gomatrixserverlib.Client` for three-PID invites
2022-05-05 11:33:16 +01:00
Till
f023cdf8c4
Add UserAPI storage tests (#2384)
* Add tests for parts of the userapi storage

* Add tests for keybackup

* Add LoginToken tests

* Add OpenID tests

* Add profile tests

* Add pusher tests

* Add ThreePID tests

* Add notification tests

* Add more device tests, fix numeric localpart query

* Fix failing CI

* Fix numeric local part query
2022-04-27 15:05:49 +02:00
Neil Alexander
d7cc187ec0
Prevent JetStream from handling OS signals, allow running as a Windows service (#2385)
* Prevent JetStream from handling OS signals, allow running as a Windows service (fixes #2374)

* Remove double import
2022-04-27 13:36:40 +01:00
Neil Alexander
144c060fa7
Remove libp2p demos (#2337)
* Remove libp2p demos

* Put it back
2022-04-21 13:31:43 +01:00
Neil Alexander
a9f0a390c6
Update to NATS Server v2.8.0 and nats.go v1.14.0 (#2359) 2022-04-20 15:13:04 +01:00
Neil Alexander
711e377b9c
Update go-sqlite3-js to matrix-org/go-sqlite3-js#2 (SQLite 3.36.0) 2022-04-19 10:34:33 +01:00
Till
69f2ff7c82
Correctly use provided filters (#2339)
* Apply filters correctly

* Fix issues; Use prepareWithFilters

* Update gmsl & tests

* go.mod..

* PR comments
2022-04-11 09:05:23 +02:00
Neil Alexander
986d27a128
Update to matrix-org/pinecone@2999ea2 2022-04-08 16:39:09 +01:00
Neil Alexander
9bd9f2beba
Update to matrix-org/pinecone@9b3963248c 2022-04-08 16:23:50 +01:00
Neil Alexander
4312b87ab0
Update dependencies (#2336) 2022-04-08 12:24:49 +01:00
Neil Alexander
4d9d9cc9b1
Update to matrix-org/gomatrixserverlib#300 2022-04-05 14:43:44 +01:00
Neil Alexander
562d742240
Update to matrix-org/gomatrixserverlib#299 2022-04-04 19:04:17 +01:00
Neil Alexander
c69159bda7
Update to matrix-org/pinecone@e526fa8 2022-04-04 15:16:02 +01:00
Neil Alexander
8213b2ba30
Update Pinecone P2P demo 2022-03-30 15:01:22 +01:00
Neil Alexander
6c95f9b535
Update to matrix-org/pinecone@6fb0773 2022-03-23 14:45:22 +00:00
Neil Alexander
c54d88aecf
Update to matrix-org/pinecone@8687b0d 2022-03-18 15:32:14 +00:00
Neil Alexander
0fb94fc781
Update to matrix-org/gomatrixserverlib@0980b7f 2022-03-17 16:48:41 +00:00
Neil Alexander
1894f34868
Update to matrix-org/gomatrixserverlib@a62d9f3 2022-03-17 09:58:44 +00:00
Neil Alexander
047147a3af
Update Yggdrasil P2P demo (also remove QUIC dependency) 2022-03-16 16:49:10 +00:00
Neil Alexander
0ad665fabc
Update to NATS Server 2.7.4 (#2282)
* Update to NATS Server v2.7.4-beta2

* Bump to release v2.7.4

* Listen for TCP

* Revert "Listen for TCP"

This reverts commit 28dff0ab4d807417a9eda8feab9f8efdf13837c7.
2022-03-16 10:47:18 +00:00
Neil Alexander
2696f36738
Federation client improvements (HTTP/2, reaping federation trippers, TLS client session cache, keepalive tweaks, update to matrix-org/gomatrixserverlib#297) 2022-03-16 10:27:16 +00:00
Neil Alexander
e1881627d1
Implement MSC3383 (update to matrix-org/gomatrixserverlib#296) 2022-03-10 12:44:31 +00:00
Neil Alexander
030b995636
Update to matrix-org/pinecone@cfde1f8 2022-03-09 10:19:49 +00:00
Neil Alexander
0297929b76
Revert NATS server upgrade in 00b3545b14 2022-03-04 15:56:31 +00:00
Dan
f05ce478f0
Implement Push Notifications (#1842)
* Add Pushserver component with Pushers API

Co-authored-by: Tommie Gannert <tommie@gannert.se>
Co-authored-by: Dan Peleg <dan@globekeeper.com>

* Wire Pushserver component

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>

* Add PushGatewayClient.

The full event format is required for Sytest.

* Add a pushrules module.

* Change user API account creation to use the new pushrules module's defaults.

Introduces "scope" as required by client API, and some small field
tweaks to make some 61push Sytests pass.

* Add push rules query/put API in Pushserver.

This manipulates account data over User API, and fires sync messages
for changes. Those sync messages should, according to an existing TODO
in clientapi, be moved to userapi.

Forks clientapi/producers/syncapi.go to pushserver/ for later extension.

* Add clientapi routes for push rules to Pushserver.

A cleanup would be to move more of the name-splitting logic into
pushrules.go, to depollute routing.go.

* Output rooms.join.unread_notifications in /sync.

This is the read-side. Pushserver will be the write-side.

* Implement pushserver/storage for notifications.

* Use PushGatewayClient and the pushrules module in Pushserver's room consumer.

* Use one goroutine per user to avoid locking up the entire server for
  one bad push gateway.
* Split pushing by format.
* Send one device per push. Sytest does not support coalescing
  multiple devices into one push. Matches Synapse. Either we change
  Sytest, or remove the group-by-url-and-format logic.
* Write OutputNotificationData from push server. Sync API is already
  the consumer.

* Implement read receipt consumers in Pushserver.

Supports m.read and m.fully_read receipts.

* Add clientapi route for /unstable/notifications.

* Rename to UpsertPusher for clarity and handle pusher update

* Fix linter errors

* Ignore body.Close() error check

* Fix push server internal http wiring

* Add 40 newly passing 61push tests to whitelist

* Add next 12 newly passing 61push tests to whitelist

* Send notification data before notifying users in EDU server consumer

* NATS JetStream

* Goodbye sarama

* Fix `NewStreamTokenFromString`

* Consume on the correct topic for the roomserver

* Don't panic, NAK instead

* Move push notifications into the User API

* Don't set null values since that apparently causes Element upsetti

* Also set omitempty on conditions

* Fix bug so that we don't override the push rules unnecessarily

* Tweak defaults

* Update defaults

* More tweaks

* Move `/notifications` onto `r0`/`v3` mux

* User API will consume events and read/fully read markers from the sync API with stream positions, instead of consuming directly

Co-authored-by: Piotr Kozimor <p1996k@gmail.com>
Co-authored-by: Tommie Gannert <tommie@gannert.se>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-03-03 11:40:53 +00:00
Neil Alexander
00b3545b14
Update NATS Server to v2.7.3 2022-03-02 10:36:20 +00:00
Kegan Dougal
1a79060b46 Bump GMSL version 2022-03-01 14:16:47 +00:00
kegsay
f1b92de017
MSC2946: Spaces Summary (round 2) (#2232)
* Initial cut at fixing up MSC2946 to work with latest spec

* bugfix: send response back correctly

* Initial working version of MSC2946

* msc2946: handle suggested_only; remove custom database

As the MSC doesn't require reverse lookups, we can just pull
the room state and inspect via the roomserver database. To
handle this, expand QueryCurrentState to support wildcards.

Use all this and handle `?suggested_only`.

* Sort child rooms

* msc2946: Make TestClientSpacesSummary pass

* msc2946: allow invited rooms to be spidered

* msc2946: support basic federation requests

* fix up go mod
2022-03-01 13:40:07 +00:00
Neil Alexander
a23fda6626
Update Events call-sites which now don't return an error, update parsedRespState to sort (#2227)
* Topologically sort with `SendEventWithState`, so that earlier events should satisfy auth for later ones

* Revert "Topologically sort with `SendEventWithState`, so that earlier events should satisfy auth for later ones"

This reverts commit b0cd706012b4c9b6724b11e16f19c4cb732ab286.

* Update to matrix-org/gomatrixserverlib#293

* `Events` no longer returns an error, other tweaks

* Make sure `Events` is sorted for `parsedRespState` too
2022-02-28 14:51:40 +00:00
Neil Alexander
4c07374c42
Reduce allocations significantly in state res v2, which should help to keep memory down when joining rooms too (update to matrix-org/gomatrixserverlib@f6ab9c5) 2022-02-24 17:05:49 +00:00
Neil Alexander
b8a97b6ee0
Update to matrix-org/pinecone@0f0afd1a46 2022-02-23 10:45:07 +00:00
Neil Alexander
934491eda5
Update NATS Server to v2.7.2 (#2193)
* Update NATS JetStream to v2.7.2

* Remove deprecated option
2022-02-17 13:15:35 +00:00
Neil Alexander
4c8c53244e
Update prometheus 2022-02-15 16:27:22 +00:00
Neil Alexander
e22e87c012
Update to matrix-org/gomatrixserverlib@20632dd 2022-02-15 11:14:43 +00:00
kegsay
aa5c3b88de
Unmarshal events at the Dendrite level not GMSL level (#2164)
* Use new event json types in gmsl

* Fix EventJSON to actually unmarshal events

* Update GMSL

* Bump GMSL and improve error messages

* Send back the correct RespState

* Update GMSL
2022-02-09 20:31:24 +00:00
Neil Alexander
a2b4777ae5
Update to matrix-org/gomatrixserverlib@a05e156fd8 2022-02-08 09:30:21 +00:00
S7evinK
9de7efa0b0
Remove sarama/saramajetstream dependencies (#2138)
* Remove dependency on saramajetstream & sarama

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Remove internal.ContinualConsumer from federationapi

* Remove internal.ContinualConsumer from syncapi

* Remove internal.ContinualConsumer from keyserver

* Move to new Prepare function

* Remove saramajetstream & sarama dependency

* Delete unneeded file

* Remove duplicate import

* Log error instead of silently irgnoring it

* Move `OffsetNewest` and `OffsetOldest` into keyserver types, change them to be more sane values

* Fix comments

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-02-04 13:08:13 +00:00
Neil Alexander
532f445c4e
Remove roomserver input deadlines (#2144)
It isn't really clear that the deadlines actually help in any way. Currently we can use up our 2 minutes doing something, run out of context time and then return an error which causes the transaction to rollback and forgetting everything we've done. If the message came to us from NATS then we probably will end up retrying just to be in the same situation. We'd be really a lot better if we just spent the time reconciling the problem in the first place, and then we're much less likely to need to fetch those missing auth or prev events in the future.

Also includes matrix-org/gomatrixserverlib#287 so we don't wait so long for servers that are obviously dead.
2022-02-04 12:13:07 +00:00
Neil Alexander
567fd04428
Update to matrix-org/gomatrixserverlib#286 2022-01-31 14:29:13 +00:00
Neil Alexander
dac762d025
Revert Prometheus client upgrades altogether 2022-01-31 11:47:07 +00:00
Neil Alexander
2995f73ae0
Update prometheus client 2022-01-31 11:16:21 +00:00
Neil Alexander
5367e7ed2c
Update to matrix-org/gomatrixserverlib@801c51af9f 2022-01-31 10:51:01 +00:00
Neil Alexander
ba1a9b98b7
Tweak some logging (#2130)
* Modify some log levels

* Update gomatrixserverlib to matrix-org/gomatrixserverlib@336334f

* Update gomatrixserverlib to matrix-org/gomatrixserverlib@cde7ac8

* Demote warning about key change producer

* Add more useful roomserver logging

* Further tweaking
2022-01-31 10:48:28 +00:00
Neil Alexander
4281976df9
Update Sarama to fix 32-bit builds (#2120) 2022-01-28 17:31:54 +00:00
Neil Alexander
2c3dd48bb2
Require Go 1.16 (#2122) 2022-01-28 16:24:01 +00:00