* Don't use more than 999 variables in SQLite querys.
Solve this problem in a more general and reusable way.
Also fix#1369
Add some unit tests.
Signed-off-by: Henrik Sölver <henrik.solver@gmail.com>
* Don't rely on testify for basic assertions
* Readability improvements and linting
Co-authored-by: Henrik Sölver <henrik.solver@gmail.com>
* Add support for database migrations
Closes#1246
This PR does NOT add any migrations as an example. I have
manually tested that the library works with SQL and Go based
upgrades correctly. Documentation should be sufficient for
devs to add migrations.
* Clarifications
* Linting
* Use state from RespSendJoin post-check
* Don't create input events for invalid events
* Let's try this again
* Update gomatrixserverlib
* Update gomatrixserverlib to matrix-org/gomatrixserverlib@38f437f
* Use federation sender for backfill and getting missing events
* Fix internal URL paths
* Update go.mod/go.sum for matrix-org/gomatrixserverlib#218
* Add missing server implementations in HTTP interface
* Add FederationClient interface to federationsender
- Use a shim struct in HTTP mode to keep the same API as `FederationClient`.
- Use `federationsender` instead of `FederationClient` in `keyserver`.
* Pointers not values
* Review comments
* Fix unit tests
* Rejig backoff
* Unbreak test
* Remove debug logs
* Review comments and linting
* Initial pass at refactoring config (not finished)
* Don't forget current state and EDU servers
* More shifting around
* Update server key API tests
* Fix roomserver test
* Fix more tests
* Further tweaks
* Fix current state server test (sort of)
* Maybe fix appservices
* Fix client API test
* Include database connection string in database options
* Fix sync API build
* Update config test
* Fix unit tests
* Fix federation sender build
* Fix gobind build
* Set Listen address for all services in HTTP monolith mode
* Validate config, reinstate appservice derived in directory, tweaks
* Tweak federation API test
* Set MaxOpenConnections/MaxIdleConnections to previous values
* Update generate-config
* Add device list updater which manages updating remote device lists
- Doesn't persist stale lists to the database yet
- Doesn't have tests yet
* Mark device lists as fresh when we persist
* Add QueryDeviceMessages to serve up device keys and stream IDs
* Consume key change events in fedsender
Don't yet send them to destinations as we haven't worked them out yet
* Send device list updates to all required servers
* Glue it all together
* Perform outbound federation hits for querying/claiming E2E keys
Untested currently because we need the receiving end to work
before sytest will be happy.
* Linting
Squashed commit of the following:
commit 86c2388e13ffdbabdd50cea205652dccc40e1860
Merge: b0a3ee6c f5e7e751
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Thu Jul 16 13:47:10 2020 +0100
Merge branch 'master' into neilalexander/yggbarequic
commit b0a3ee6c5c063962384bb91c59ec753ddc8cfe5f
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Thu Jul 16 13:42:22 2020 +0100
Add support for broadcasting wake-up EDUs to known hosts
commit 8a5c2020b3a4b705b5d5686a9e71990a49e6d471
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Thu Jul 16 13:42:10 2020 +0100
Bare QUIC demo working
commit d3939b3d6568cf4262c0391486a5203873b68bfc
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Wed Jul 15 11:42:43 2020 +0100
Support bare Yggdrasil sessions with encrypted QUIC
* Add a bit more logging to the fedsender
* bugfix: continue sending PDUs if ones are added whilst sending another PDU
Without this, the queue goes back to sleep on `<-oq.notifyPDUs` which won't
fire because `pendingPDUs` is already > 0. This should fix a flakey sytest.
* Break if no txn is sent
* WIP syncapi work
* More debugging
* Bump GMSL version to pull in working Event.Redact
* Remove logging
* Make redactions work on v3+
* Fix more tests
* Emit redacted_event from the roomserver when redactions are validated
- Consume them in the currentstateserver and act accordingly.
- Add integration test for the roomserver to check that injecting
`m.room.redaction` events result in `redacted_event` being emitted.
* Linting
* Ignore events that redact themselves
* Don't return null to public directory request
* Initial support for finding public rooms in Yggdrasil demo (incomplete)
* Increase QUIC idle time to 15 minutes
* Initial QUIC work
* Update Yggdrasil demo
* Make sure that the federation sender knows how many pending events are in the database when the worker starts
* QUIC tunables
* pprof
* Don't spin
* Set build info for Yggdrasil
* Use content_value instead of membership
* Fix build
* Replace publicroomsapi with a combination of clientapi/roomserver/currentstateserver
- All public rooms paths are now handled by clientapi
- Requests to (un)publish rooms are sent to the roomserver via `PerformPublish`
which are stored in a new `published_table.go`
- Requests for public rooms are handled in clientapi by:
* Fetch all room IDs which are published using `QueryPublishedRooms` on the roomserver.
* Apply pagination parameters to the slice.
* Do a `QueryBulkStateContent` request to the currentstateserver to pull out
required state event *content* (not entire events).
* Aggregate and return the chunk.
Mostly but not fully implemented (DB queries on currentstateserver are missing)
* Fix pq query
* Make postgres work
* Make sqlite work
* Fix tests
* Unbreak pagination tests
* Linting
* Return remote errors from FS.PerformJoin
Follows the same pattern as PerformJoin on roomserver (no error return).
Also return the right format for incompatible room version errors.
Makes a bunch of tests pass!
* Handle network errors better when returning remote HTTP errors
* Linting
* Fix tests
* Update whitelist, pass network errors through in API=1 mode
* More key tweaks
* Start testing stuff
* Move responsibility for generating local keys into server key API, don't register prom in caches unless needed, start tests
* Don't store our own keys in the database
* Don't store our own keys in the database
* Don't run tests for now
* Tweak caching behaviour, update tests
* Update comments, add fixes from forward-merge
* Debug logging
* Debug logging
* Perform final comparison against original set of requests
* oops
* Fetcher timeouts
* Fetcher timeouts
* missing func
* Tweaks
* Update gomatrixserverlib
* Fix Federation API test
* Break up FetchKeys
* Add comments to caching
* Add URL check in test
* Partially revert "Move responsibility for generating local keys into server key API, don't register prom in caches unless needed, start tests"
This reverts commit d7eb54c5b30b2f6a9d6514b643e32e6ad2b602f3.
* Fix federation API test
* Fix internal cache stuff again
* Fix server key API test
* Update comments
* Update comments from review
* Fix lint
* Use MissingAuthEventHandler on performjoin to try and speed up cases where we have missing events
* Update gomatrixserverlib
* Use supplied room version
* Use AuthChainProvider
* Tweaks
* Update gomatrixserverlib
* Signature checks
* Return bad request on CS API /send if bad JSON
* Return some more M_BAD_JSON in the right places
* nolint because damnit gocyclo all I added was a type check for an error
* Update gomatrixserverlib
* Update gomatrixserverlib
* Update sytest-whitelist
* Update gomatrixserverlib
* Update sytest-whitelist
* NotJSON -> BadJSON
* Groundwork for send-to-device messaging
* Update sample config
* Add unstable routing for now
* Send to device consumer in sync API
* Start the send-to-device consumer
* fix indentation in dendrite-config.yaml
* Create send-to-device database tables, other tweaks
* Add some logic for send-to-device messages, add them into sync stream
* Handle incoming send-to-device messages, count them with EDU stream pos
* Undo changes to test
* pq.Array
* Fix sync
* Logging
* Fix a couple of transaction things, fix client API
* Add send-to-device test, hopefully fix bugs
* Comments
* Refactor a bit
* Fix schema
* Fix queries
* Debug logging
* Fix storing and retrieving of send-to-device messages
* Try to avoid database locks
* Update sync position
* Use latest sync position
* Jiggle about sync a bit
* Fix tests
* Break out the retrieval from the update/delete behaviour
* Comments
* nolint on getResponseWithPDUsForCompleteSync
* Try to line up sync tokens again
* Implement wildcard
* Add all send-to-device tests to whitelist, what could possibly go wrong?
* Only care about wildcard when targeted locally
* Deduplicate transactions
* Handle tokens properly, return immediately if waiting send-to-device messages
* Fix sync
* Update sytest-whitelist
* Fix copyright notice (need to do more of this)
* Comments, copyrights
* Return errors from Do, fix dendritejs
* Review comments
* Comments
* Constructor for TransactionWriter
* defletions
* Update gomatrixserverlib, sytest-blacklist
* Server key API (works for monolith but not for polylith yet)
* Re-enable caching on server key API component
* Groundwork for HTTP APIs for server key API
* Hopefully implement HTTP for server key API
* Simplify public key request marshalling from map keys
* Update gomatrixserverlib
* go mod tidy
* Common -> internal
* remove keyring.go
* Update Docker Hub for server key API
* YAML is funny about indentation
* Wire in new server key API into hybrid monolith mode
* Create maps
* Route server key API endpoints on internal API mux
* Fix server key API URLs
* Add fetcher behaviour into server key API implementation
* Return error if we failed to fetch some keys
* Return results anyway
* Move things about a bit
* Remove unused code
* Fix comments, don't use federation sender URL in polylith mode
* Add server_key_api to sample config
* Review comments
* HTTP API to cache keys that have been requested
* Overwrite server_key_api listen in monolith hybrid mode
* Make backfill work for shared history visibility
* fetch missing state on backfill to remember snapshots correctly
* Fix gmsl to not mux in auth events into room state
* Whoops
* Linting
* Initial cut for backfilling
The syncserver now asks the roomserver via QueryBackfill (which already
existed to *handle* backfill requests) which then makes federation requests
via gomatrixserverlib.RequestBackfill.
Currently, tests fail on subsequent /messages requests because we don't know
which servers are in the room, because we are unable to get state snapshots
from a backfilled event because that code doesn't exist yet.
* WIP backfill, doesn't work
* Make initial backfill pass checks
* Persist backfilled events with state snapshots
* Remove debug lines
* Linting
* Review comments
* Update gomatixserverlib
* Try to build invite stripped state if not given to us
* SendInvite improvements
* Transpose invite_room_state into invite_state.events for sync API
* Remove syncapi debugging output
* Use RespInviteV2
* Update gomatrixserverlib
* Send the invite event as a normal roomserver event too, for incorporating into room (should this be done by the roomserver automatically for invite inputs?)
* Federation sender use invite_room_state, room server try to insert membership state
* Check supported room versions on the invite endpoint
* Prevent roomserver query API from trying to handle requests for stub rooms
* Adding a nolint
* Replace IsRoomStub with RoomNIDExcludingStubs, fix query API to use that instead
* Review comments
* Update gomatrixserverlib
* Test matrix.org as perspective key server
* Base64 decode better
* Optional strict validity checking in gmsl
* Update gomatrixserverlib
* Attempt to find missing auth events over federation (this shouldn't happen but I am guessing there is a synapse bug involved where we don't get all of the auth events)
* Update gomatrixserverlib, debug logging
* Remove debugging output
* More verbose debugging
* Print outliers
* Increase timeouts for testing, observe contexts before trying to join over more servers
* Don't block on roomserver (experimental)
* Don't block on roomserver
* Update gomatrixserverlib
* Update gomatrixserverlib
* Configurable perspective key fetchers
* Output number of configured keys for perspective
* Example perspective config included
* Undo debug stack trace
* Undo debug stack trace
* Restore original HTTP listener in monolith
* Fix lint
* Review comments
* Set default HTTP server timeout to 5 minutes now, block again when joining
* Don't use HTTP address for HTTPS whoops
* Update gomatrixserverlib
* Update gomatrixserverlib
* Update gomatrixserverlib
* Actually add perspectives
* Actually add perspectives
* Update gomatrixserverlib
* Improve error handling in federation /send endpoint a bit
* Remove unknownRoomError, use unmarshalError when unable to get room ID
* Swap out a couple more internal server errors
* Update gomatrixserverlib
* Update gomatrixserverlib
* Update gomatrixserverlib
* Update gomatrixserverlib
* Update gomatrixserverlib
* Update gomatrixserverlib
* Return bad limit in error
* Same with domain/userid
* Set default room version to 4
* Default to v1 when room_version key missing
Signed-off-by: Alex Chen <minecnly@gmail.com>
* Squashed commit of the following:
commit c1bca95adb
Author: Kegsay <kegan@matrix.org>
Date: Thu Apr 16 10:06:55 2020 +0100
Add SQL tracing via DENDRITE_TRACE_SQL (#968)
* Add SQL tracing via DENDRITE_TRACE_SQL
Add this to `internal/sqlutil` in preparation for #897
* Not entirely
commit c2ea961909
Author: Kegsay <kegan@matrix.org>
Date: Wed Apr 15 17:48:40 2020 +0100
Add HTTP trace logging (#965)
* Dump all requests/response server-side
* Wrap in DENDRITE_TRACE
* DENDRITE_TRACE_HTTP is better
* Bugfix for response body and linting
* False is true and true is false
* Linting
* How did this get missed
* More linting
* Update gomatrixserverlib
* Remove unneeded imports
Co-authored-by: Cnly <minecnly@gmail.com>
* Dump all requests/response server-side
* Wrap in DENDRITE_TRACE
* DENDRITE_TRACE_HTTP is better
* Bugfix for response body and linting
* False is true and true is false
* Linting
* How did this get missed
* More linting
* Add libp2p-go
* Some tweaks, tidying up
(cherry picked from commit 1a5bb121f8121c4f68a27abbf25a9a35a1b7c63e)
* Move p2p dockerfile
(cherry picked from commit 8d3bf44ea1bf37f950034e73bcdc315afdabe79a)
* Remove containsBackwardsExtremity
* Fix some linter errors, update some libp2p packages/calls, other tidying up
* Add -port for dendrite-p2p-demo
* Use instance name as key ID
* Remove P2P demo docker stuff, no longer needed now that we have SQLite
* Remove Dockerfile-p2p too
* Remove p2p logic from dendrite-monolith-server
* Inject publicRoomsDB in publicroomsapi
Inject publicRoomsDB instead of switching on base.libP2P.
See: https://github.com/matrix-org/dendrite/pull/956/files?file-filters%5B%5D=.go#r406276914
* Fix lint warning
* Extract mDNSListener from base.go
* Extract CreateFederationClient into demo
* Create P2PDendrite from BaseDendrite
Extract logic specific to P2PDendrite from base.go
* Set base.go to upstream/master
* Move pubsub to demo cmd
* Move PostgreswithDHT to cmd
* Remove unstable features
* Add copyrights
* Move libp2pvalidator into p2pdendrite
* Rename dendrite-p2p-demo -> dendrite-demo-libp2p
* Update copyrights
* go mod tidy
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Update gomatrixserverlib
* Default to room version 4
* Update gomatrixserverlib
* Limit prev_events and auth_events
* Fix auth_events, prev_events
* Fix linter issues
* Update gomatrixserverlib
* Fix getState
* Update sytest-whitelist
* Squashed commit of the following:
commit 067b875063
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date: Fri Apr 3 14:29:06 2020 +0100
Invites v2 endpoint (#952)
* Start converting v1 invite endpoint to v2
* Update gomatrixserverlib
* Early federationsender code for sending invites
* Sending invites sorta happens now
* Populate invite request with stripped state
* Remodel a bit, don't reflect received invites
* Handle invite_room_state
* Handle room versions a bit better
* Update gomatrixserverlib
* Tweak order in destinationQueue.next
* Revert check in processMessage
* Tweak federation sender destination queue code a bit
* Add comments
commit 955244c092
Author: Ben B <benne@klimlive.de>
Date: Fri Apr 3 12:40:50 2020 +0200
use custom http client instead of the http DefaultClient (#823)
This commit replaces the default client from the http lib with a custom one.
The previously used default client doesn't come with a timeout. This could cause
unwanted locks.
That solution chosen here creates a http client in the base component dendrite
with a constant timeout of 30 seconds. If it should be necessary to overwrite
this, we could include the timeout in the dendrite configuration.
Here it would be a good idea to extend the type "Address" by a timeout and
create an http client for each service.
Closes#820
Signed-off-by: Benedikt Bongartz <benne@klimlive.de>
Co-authored-by: Kegsay <kegan@matrix.org>
* Update sytest-whitelist, sytest-blacklist
* Update go.mod/go.sum
* Add some error wrapping for debug
* Add a NOTSPEC to common/events.go
* Perform state resolution at send_join
* Set default room version to v2 again
* Tweak GetCapabilities
* Add comments to ResolveConflictsAdhoc
* Update sytest-blacklist
* go mod tidy
* Update sytest-whitelist, sytest-blacklist
* Update versions
* Updates from review comments
* Update sytest-blacklist, sytest-whitelist
* Check room versions compatible at make_join, add some comments, update gomatrixserverlib, other tweaks
* Set default room version back to v2
* Update gomatrixserverlib, sytest-whitelist
* Start converting v1 invite endpoint to v2
* Update gomatrixserverlib
* Early federationsender code for sending invites
* Sending invites sorta happens now
* Populate invite request with stripped state
* Remodel a bit, don't reflect received invites
* Handle invite_room_state
* Handle room versions a bit better
* Update gomatrixserverlib
* Tweak order in destinationQueue.next
* Revert check in processMessage
* Tweak federation sender destination queue code a bit
* Add comments
* Room version 2 by default, other wiring updates, update gomatrixserverlib
* Fix nil pointer exception
* Fix some more nil pointer exceptions hopefully
* Update gomatrixserverlib
* Send all room versions when joining, not just stable ones
* Remove room version cquery
* Get room version when getting events from the roomserver database
* Reset default back to room version 2
* Don't generate event IDs unless needed
* Revert "Remove room version cquery"
This reverts commit a170d5873360dd059614460acc8b21ab2cda9767.
* Query room version in federation API, client API as needed
* Improvements to make_join send_join dance
* Make room server producers use headered events
* Lint tweaks
* Update gomatrixserverlib
* Versioned SendJoin
* Query room version in syncapi backfill
* Handle transaction marshalling/unmarshalling within Dendrite
* Sorta fix federation (kinda)
* whoops commit federation API too
* Use NewEventFromTrustedJSON when getting events from the database
* Update gomatrixserverlib
* Strip headers on federationapi endpoints
* Fix bug in clientapi profile room version query
* Update gomatrixserverlib
* Return more useful error if room version query doesn't find the room
* Update gomatrixserverlib
* Update gomatrixserverlib
* Maybe fix federation
* Fix formatting directive
* Update sytest whitelist and blacklist
* Temporarily disable room versions 3 and 4 until gmsl is fixed
* Fix count of EDUs in logging
* Update gomatrixserverlib
* Update gomatrixserverlib
* Update gomatrixserverlib
* Rely on EventBuilder in gmsl to generate the event IDs for us
* Some review comments fixed
* Move function out of common and into gmsl
* Comment in federationsender destinationqueue
* Update gomatrixserverlib
* Use HeaderedEvent in syncapi
* Update notifier test
* Fix persisting headered event
* Clean up unused API function
* Fix overshadowed err from linter
* Write headered JSON to invites table too
* Rename event_json to headered_event_json in syncapi database schemae
* Fix invites_table queries
* Update QueryRoomVersionCapabilitiesResponse comment
* Fix syncapi SQLite
* Create and glue ExternalPublicRoomsProvider into the public rooms component
This is how we will link p2p stuff to dendrite proper.
* Use gmsl structs rather than our own
* Implement federated public rooms
- Make thirdparty endpoint r0 so riot-web loads the public room list
* Typo
* Missing callsites
* Add room version into createRoomReq
* Extract room version from m.room.create event when persisting
* Reduce cyclomatic complexity
* Update whitelist, gomatrixserverlib, tweaks to roomserver
* Update sytest-whitelist again
* Implement gomatrixserverlib.HeaderedEvent, which should allow us to store room version headers along with the event across API boundaries and consumers/producers, and intercept unmarshalling to get the event structure right
* Add federationsender to previous
* Update room version descriptors, add error handling
* Fix database queries
* Drop Get from version package
* Fix database wrapping, add comments for version descriptions
* Don't set default room_version value in SQL
* p2p: Use JSServer for comms rather than GoJsConn
This has several benefits:
- it fixes a bug whereby you could not transmit >4k bytes to/from JS/Go land.
- it more clearly exposes the interface point between Go and JS: a single
global function call.
- it presents a nicer API shape than the previous `net.Conn`.
- it doesn't needlessly 'stream' data which is already sitting in-memory.
This is currently only active for local CS API traffic, another PR will
add Federation P2P support.
* Typo
* Pass in the private key seed to NewP2PLocalNode to use the same key for p2p
* Extract the ed25519 key from the server name and use it for event checks
This allows HSes which no longer exist to still have events verified for
them. This means that new members can join rooms that has missing servers.
* Remove no-op ipfs uri which we don't use anymore
* Try to recursively find auth events (to a point) if they are missing
* Remove recursion limit for now and other review fixes
* Simplify error handling for recursion
* Pass room version 1 only to MakeJoin until room version support comes later
* Use a fork of pq which supports userCurrent on wasm
* Use sqlite3_js driver when running in JS
* Add cmd/dendritejs to pull in sqlite3_js driver for wasm only
* Update to latest go-sqlite-js version
* Replace prometheus with a stub. sigh
* Hard-code a config and don't use opentracing
* Latest go-sqlite3-js version
* Generate a key for now
* Listen for fetch traffic rather than HTTP
* Latest hacks for js
* libp2p support
* More libp2p
* Fork gjson to allow us to enforce auth checks as before
Previously, all events would come down redacted because the hash
checks would fail. They would fail because sjson.DeleteBytes didn't
remove keys not used for hashing. This didn't work because of a build
tag which included a file which no-oped the index returned.
See https://github.com/tidwall/gjson/issues/157
When it's resolved, let's go back to mainline.
* Use gjson@1.6.0 as it fixes https://github.com/tidwall/gjson/issues/157
* Use latest gomatrixserverlib for sig checks
* Fix a bug which could cause exclude_from_sync to not be set
Caused when sending events over federation.
* Use query variadic to make lookups actually work!
* Latest gomatrixserverlib
* Add notes on getting p2p up and running
Partly so I don't forget myself!
* refactor: Move p2p specific stuff to cmd/dendritejs
This is important or else the normal build of dendrite will fail
because the p2p libraries depend on syscall/js which doesn't work
on normal builds.
Also, clean up main.go to read a bit better.
* Update ho-http-js-libp2p to return errors from RoundTrip
* Add an LRU cache around the key DB
We actually need this for P2P because otherwise we can *segfault*
with things like: "runtime: unexpected return pc for runtime.handleEvent"
where the event is a `syscall/js` event, caused by spamming sql.js
caused by "Checking event signatures for 14 events of room state" which
hammers the key DB repeatedly in quick succession.
Using a cache fixes this, though the underlying cause is probably a bug
in the version of Go I'm on (1.13.7)
* breaking: Add Tracing.Enabled to toggle whether we do opentracing
Defaults to false, which is why this is a breaking change. We need
this flag because WASM builds cannot do opentracing.
* Start adding conditional builds for wasm to handle lib/pq
The general idea here is to have the wasm build have a `NewXXXDatabase`
that doesn't import any postgres package and hence we never import
`lib/pq`, which doesn't work under WASM (undefined `userCurrent`).
* Remove lib/pq for wasm for syncapi
* Add conditional building to remaining storage APIs
* Update build script to set env vars correctly for dendritejs
* sqlite bug fixes
* Docs
* Add a no-op main for dendritejs when not building under wasm
* Use the real prometheus, even for WASM
Instead, the dendrite-sw.js must mock out `process.pid` and
`fs.stat` - which must invoke the callback with an error (e.g `EINVAL`)
in order for it to work:
```
global.process = {
pid: 1,
};
global.fs.stat = function(path, cb) {
cb({
code: "EINVAL",
});
}
```
* Linting
