Commit Graph

2240 Commits

Author SHA1 Message Date
Neil Alexander
f7b2a5866e
Update dependencies (#1372)
* Other updates

* Update naffka

* naffka not indirect

* Revert quic-go
2020-09-01 10:20:31 +01:00
Neil Alexander
737802fc06
Update Complement dockerfile to not wait for apt-get input 2020-08-28 15:19:47 +01:00
Neil Alexander
c42c70597c
Add some error wrapping to sync API, use background context for sync (#1363)
* Add some error wrapping to sync API

* Don't use request context for BeginTx until mattn/go-sqlite3#764 is fixed
2020-08-28 13:28:40 +01:00
Neil Alexander
fee1c22790
Fix #1361 (#1362) 2020-08-28 09:46:32 +01:00
Kegsay
9af2f5f1f2
Remove device DB from clientapi (#1352)
* Remove device DB from clientapi

* Remove device DB from startup configuration

It's all an impl detail now in user API
2020-08-27 18:53:40 +01:00
Neil Alexander
c0f28845f8
Try to protect GetNextTransactionPDUs (#1350) 2020-08-27 15:27:12 +01:00
Neil Alexander
034e5d5a55
Speed up accepting invites from remote users if we're already in the room (#1349) 2020-08-27 11:47:13 +01:00
Neil Alexander
7466e6b718
Fix lock errors in federation sender (#1347)
* Fix lock errors in federation sender

* Additional fix to writers
2020-08-27 11:05:41 +01:00
Kegsay
421b6b2313
Add OpenTracing documentation (#1345)
* Add OpenTracing documentation

Closes #274

* Reshuffle

* Update setup.md
2020-08-27 10:24:56 +01:00
Neil Alexander
42183bef50
Allow join to accept server_name in query parameters (#1346) 2020-08-26 18:23:08 +01:00
Kegsay
3802efe301
Port of #945 (#1343)
- Make sure we always cleanup the temp directory on error.
- Complain about it having an error prone API shape.
2020-08-26 15:38:34 +01:00
Kegsay
29d6481842
Wait for 8h between device list updates for blacklisted servers (#1344) 2020-08-26 15:38:21 +01:00
Kegsay
abd16ff4a0
Modify DeviceListUpdater to retry requests according to RetryAfter (#1342)
* Modify DeviceListUpdater to retry requests according to RetryAfter

* Reduce wait time for sytest test pollution
2020-08-26 12:03:09 +01:00
oliverpool
3205b9212d
[readability] use event.StateKeyEquals where relevant and minor for-loop refactoring (#1339)
* use event.StateKeyEquals where relevant

Signed-off-by: Olivier Charvin <git@olivier.pfad.fr>

* use userID
2020-08-25 21:04:35 +01:00
Kegsay
55498c8deb
Fix 'Invited user can reject invite over federation several times' (#1341) 2020-08-25 18:59:00 +01:00
Neil Alexander
c60270eea3
Enforce history visibility etc for /rooms/{roomID}/state (#1340)
* Enforce history visibility etc for /rooms/{roomID}/state

* Deduplicate OnIncomingStateRequest and OnIncomingStateTypeRequest

* Revert "Deduplicate OnIncomingStateRequest and OnIncomingStateTypeRequest"

This reverts commit 335035d66e629022232abc682d6631e3cf669e23.
2020-08-25 18:43:56 +01:00
Neil Alexander
48850d9995
Update /versions (#1338) 2020-08-25 15:44:33 +01:00
Neil Alexander
819b926820
Make redactions permanent (#1337)
* Make redactions permanent

* Update storage.go
2020-08-25 15:44:19 +01:00
Neil Alexander
287700baee
Randomly generated media IDs (#1336)
* Filenames based on hashes but media IDs randomly generated and per-upload metadata

* Lint

* Comment generateMediaID
2020-08-25 15:08:37 +01:00
oliverpool
a4db43e096
Don't overwrite global err before return (#1293)
Signed-off-by: Olivier Charvin <git@olivier.pfad.fr>
2020-08-25 13:11:52 +01:00
Neil Alexander
c8b873abc8
Roomserver NID caches (#1335)
* Initial work on roomserver NID caches

* Give caches to roomserver storage

* Populate caches

* Fix bugs

* Fix WASM build

* Don't hit cache twice in RoomNIDExcludingStubs

* Store reverse room ID-room NID mapping, consult caches when assigning NIDs
2020-08-25 12:32:29 +01:00
Kegsay
05242096a1
Update README.md 2020-08-25 12:09:34 +01:00
Kegsay
733abe912b
Add spec compliant path for redaction (#1334)
Possibly fixes #1194
2020-08-25 10:39:30 +01:00
Neil Alexander
720ddce0a8
Use Writer in shared package (#1296) 2020-08-25 10:29:45 +01:00
Kegsay
3b14119aff
Update README/CONTRIBUTING/sytest for AWSY issues (#1333)
* Update README/CONTRIBUTING/sytest for AWSY issues

* More docs
2020-08-25 08:11:41 +01:00
oliverpool
466d795375
Fix scripts path in documentation (#1294)
Broken by 24d8df664c

Signed-off-by: Olivier Charvin <git@olivier.pfad.fr>
2020-08-24 18:13:43 +01:00
Neil Alexander
4b2db1dff5
Update Naffka (#1295)
* Update Naffka

* Fix Naffka setup

* Update Naffka

* Update Naffka
2020-08-24 13:49:20 +01:00
Neil Alexander
10461b8870
Fix embedding Element Web into demo-libp2p and demo-yggdrasil 2020-08-21 11:01:30 +01:00
Neil Alexander
9d53351dc2
Component-wide TransactionWriters (#1290)
* Offset updates take place using TransactionWriter

* Refactor TransactionWriter in current state server

* Refactor TransactionWriter in federation sender

* Refactor TransactionWriter in key server

* Refactor TransactionWriter in media API

* Refactor TransactionWriter in server key API

* Refactor TransactionWriter in sync API

* Refactor TransactionWriter in user API

* Fix deadlocking Sync API tests

* Un-deadlock device database

* Fix appservice API

* Rename TransactionWriters to Writers

* Move writers up a layer in sync API

* Document sqlutil.Writer interface

* Add note to Writer documentation
2020-08-21 10:42:08 +01:00
Neil Alexander
5aaf32bbed
Select distinct on room memberships in sync API (#1292) 2020-08-21 09:57:52 +01:00
Kegsay
ec95d331a5
Add support for running Complement on current working directories (#1291)
This will be used in the future by Buildkite to run on CI.
2020-08-20 18:35:04 +01:00
Kegsay
6d6bb75137
Add FederationClient interface to federationsender (#1284)
* Add FederationClient interface to federationsender

- Use a shim struct in HTTP mode to keep the same API as `FederationClient`.
- Use `federationsender` instead of `FederationClient` in `keyserver`.

* Pointers not values

* Review comments

* Fix unit tests

* Rejig backoff

* Unbreak test

* Remove debug logs

* Review comments and linting
2020-08-20 17:03:07 +01:00
Neil Alexander
068a3d3c9f
Roomserver per-room input parallelisation (Postgres) (#1289)
* Per-room input mutex

* GetMembership should use transaction when assigning state key NID

* Actually use writer transactions rather than ignoring them

* Limit per-room mutexes to Postgres

* Flip the check in InputRoomEvents
2020-08-20 16:24:33 +01:00
Neil Alexander
0fea056db4
Change backoff behaviour so that Failure returns planned end time (#1288) 2020-08-20 14:58:53 +01:00
Neil Alexander
5ad47d3b3d
Fix more roomserver transactions/locks (#1287)
* Fix transaction to InsertTransaction

* Remove unnecessary txn, add txns around setting up updaters
2020-08-20 09:24:52 +01:00
anandv96
f5edfb9659
#903: Client API: mutex on (user_id, room_id) (#1286)
* Client API: mutex on (user_id, room_id)

* Client API: mutex on (user_id, room_id)

Changed variable name used for the mutexes map
Changed the place where the mutex is locked
Changed unlock to a defered call instead of manually calling it
2020-08-20 08:27:43 +01:00
Neil Alexander
b24747b305
Transaction writer changes, move roomserver writers (#1285)
* Updated TransactionWriters, moved locks in roomserver, various other tweaks

* Fix redaction deadlocks

* Fix lint issue

* Rename SQLiteTransactionWriter to ExclusiveTransactionWriter

* Fix us not sending transactions through in latest events updater
2020-08-19 15:38:27 +01:00
Neil Alexander
775b04d776
Roomserver updater changes (#1283)
* Take input transaction when setting up updaters

* Fix nil pointer exceptions

* Rename room recent events updater to latest events updater

* Contd rename room recent events updater to latest events updater

* Remove unnecessary interfaces for latest events and membership updaters
2020-08-19 13:24:54 +01:00
Neil Alexander
3d58417555
Roomserver database-wide TransactionWriters (#1282)
* Database-wide TransactionWriter

* Fix deadlocking Sync API tests

* Undo non-roomserver changes for now
2020-08-19 10:57:29 +01:00
Kegsay
e571e196ce
Summarise key change logs (#1278) 2020-08-18 11:14:37 +01:00
Kegsay
02a8515e99
Only emit key changes which are different from what we had before (#1279)
We did this already for local `/keys/upload` but didn't for
remote `/users/devices`. This meant any resyncs would spam produce
events, hammering disk i/o and spamming the logs.
2020-08-18 11:14:20 +01:00
Neil Alexander
a5a85c6a11
Make PerformJoin responsible for sending invite to RS input (#1277)
* Make PerformJoin send input membership event

* Invite input room events in separate goroutine

* Don't limit roomserver input events using request context

* Synchronous input room events

* Nope, that didn't work

* oops send state key to GetMembership

* Don't generate stripped state in client API more times than necessary, generate output events on receiving end of federated invite

* Commit membership updater changes

* Tweaks
2020-08-17 17:33:19 +01:00
Neil Alexander
e7d450adb8
Add SkipClean and UseEncodedPaths to intermediate routers (#1275) 2020-08-17 12:28:20 +01:00
Neil Alexander
6cb1a65809
Synchronous invites (#1273)
* Refactor invites to be synchronous

* Fix synchronous invites

* Fix client API return type for send invite error

* Linter

* Restore PerformError on rsAPI.PerformInvite

* Update sytest-whitelist

* Don't override PerformError with normal errors

* Fix error passing

* Un-whitelist a couple of tests

* Update sytest-whitelist

* Try to handle multiple invite rejections better

* nolint

* Update gomatrixserverlib

* Fix /v1/invite test

* Remove replace from go.mod
2020-08-17 11:40:49 +01:00
Kegsay
6820b3e024
AWSY: new groups and add AS API section (#1270)
Really we should never have put the AS API bits under CS API to
begin with, as it is a completely different versioned spec: split
it out (which bumps the CS API % slightly).

Also, split out Device Keys into 2 extra sections: Device Key Backup
and Cross-signing Keys. The point of AWSY is for it to be easy for
anyone to see which features are implemented in Dendrite, and bundling
three distinct features under the umbrella of Device Keys was just
unclear.
2020-08-14 11:26:06 +01:00
Neil Alexander
c1f4faf308
Fix Docker builds, polylith components (#1269) 2020-08-13 18:27:19 +01:00
Kegsay
20c8f252a7
Make 'Device list doesn't change if remote server is down' pass (#1268)
- As a last resort, query the DB when exhausting all possible remote query
  endpoints, but keep the field in `failures` so clients can detect that this
  is stale data.
- Unblock `DeviceListUpdater.Update` on failures rather than timing out.
- Use a mutex when writing directly to `res`, not just for failures.
2020-08-13 16:43:27 +01:00
Neil Alexander
4c4732a9c9
Don't send to ACL'd servers (#1267)
* Don't send to ACL'd servers

* Use gjson to look for room_id in EDU
2020-08-13 14:23:37 +01:00
Neil Alexander
9677a95afc
API setup refactoring (#1266)
* Start HTTP endpoint refactoring

* Update SetupAndServeHTTP

* Fix builds

* Don't set up external listener if no address configured

* TLS HTTP setup

* Break apart client/federation/key/media muxes

* Tweaks

* Fix P2P demos

* Fix media API routing

* Review comments @Kegsay

* Update sample config

* Fix gobind build

* Fix External -> Public in federation API test
2020-08-13 12:16:37 +01:00
Kegsay
820c56c165
Fix more E2E sytests (#1265)
* WIP: Eagerly sync device lists on /user/keys/query requests

Also notify servers when a user's device display name changes. Few
caveats:
 - sytest `Device deletion propagates over federation` fails
 - `populateResponseWithDeviceKeysFromDatabase` is called from multiple
   goroutines and hence is unsafe.

* Handle deleted devices correctly over federation
2020-08-12 22:43:02 +01:00