Commit Graph

2438 Commits

Author SHA1 Message Date
Neil Alexander
6ee8507955
Correct account data position mapping 2022-04-27 15:10:10 +01:00
Till
f023cdf8c4
Add UserAPI storage tests (#2384)
* Add tests for parts of the userapi storage

* Add tests for keybackup

* Add LoginToken tests

* Add OpenID tests

* Add profile tests

* Add pusher tests

* Add ThreePID tests

* Add notification tests

* Add more device tests, fix numeric localpart query

* Fix failing CI

* Fix numeric local part query
2022-04-27 15:05:49 +02:00
Neil Alexander
d7cc187ec0
Prevent JetStream from handling OS signals, allow running as a Windows service (#2385)
* Prevent JetStream from handling OS signals, allow running as a Windows service (fixes #2374)

* Remove double import
2022-04-27 13:36:40 +01:00
Neil Alexander
54ff4cf690
Don't try to federated-join via ourselves (#2383) 2022-04-27 12:23:55 +01:00
Neil Alexander
dca4afd2f0
Don't send account data or receipts for left/forgotten rooms (#2382)
* Only include account data and receipts for rooms in a complete sync that we care about

* Fix global account data
2022-04-27 12:03:34 +01:00
Neil Alexander
66b397b3c6
Don't create fictitious presence entries (#2381)
* Don't create fictitious presence entries for users that don't have any

* Update whitelist, since that test probably shouldn't be passing

* Fix panics
2022-04-27 11:25:07 +01:00
Neil Alexander
6c5c6d73d7
Use a value that is Go 1.16-friendly 2022-04-26 17:05:31 +01:00
Neil Alexander
b527e33c16
Send all account data on complete sync by default
Squashed commit of the following:

commit 0ec8de57261d573a5f88577aa9d7a1174d3999b9
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Apr 26 16:56:30 2022 +0100

    Select filter onto provided target filter

commit da40b6fffbf5737864b223f49900048f557941f9
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Apr 26 16:48:00 2022 +0100

    Specify other field too

commit ffc0b0801f63bb4d3061b6813e3ce5f3b4c8fbcb
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Tue Apr 26 16:45:44 2022 +0100

    Send as much account data as possible during complete sync
2022-04-26 16:58:20 +01:00
Neil Alexander
f6d07768a8
Fix account data position 2022-04-26 16:08:01 +01:00
Neil Alexander
6892e0f0e0
Start account data ID from from 2022-04-26 16:02:21 +01:00
Till
4c19f22725
Fix account_data not correctly send in a complete sync (#2379)
* Return the StreamPosition from the database and not the latest

* Fix linter issue
2022-04-26 14:50:56 +01:00
Neil Alexander
5306c73b00
Fix bug when uploading device signatures (#2377)
* Find the complete key ID when uploading signatures

* Try that again

* Try splitting the right thing

* Don't do it for device keys

* Refactor `QuerySignatures`

* Revert "Refactor `QuerySignatures`"

This reverts commit c02832a3e92569f64f180dec1555056dc8f8c3e3.

* Both requested key IDs and master/self/user keys

* Fix uniqueness

* Try tweaking GMSL

* Update GMSL again

* Revert "Update GMSL again"

This reverts commit bd6916cc379dd8d9e3f38d979c6550bd658938aa.

* Revert "Try tweaking GMSL"

This reverts commit 2a054524da9d64c6a2a5228262fbba5fde28798c.

* Database migrations
2022-04-26 13:08:54 +01:00
Till
e8be2b234f
Add heroes to the room summary (#2373)
* Implement room summary heroes

* Add passing tests

* Move MembershipCount to addRoomSummary

* Add comments, close Statement
2022-04-26 10:53:17 +02:00
Till
feac9db43f
Add transactionsCache to redact endpoint (#2375) 2022-04-26 10:28:41 +02:00
Till Faelligen
7df5d69a5b Checkout correct branch for Sytest 2022-04-26 08:07:27 +02:00
Till
e8ab2154aa
Return M_NOT_FOUND for rejected events (#2371)
* Return M_NOT_FOUND for rejected events

* Add passing tests
2022-04-25 19:05:01 +02:00
Till
e95fc5c5e3
Use provided filter for account_data (#2372)
* Reuse IncrementalSync, use provided filter

* Inform SyncAPI about newly created push_rules
2022-04-25 19:04:46 +02:00
Neil Alexander
aad81b7b4d
Only call key update process functions if there are updates, don't send things to ourselves over federation 2022-04-25 14:22:46 +01:00
Till
446819e4ac
Store the EDU type in the database (#2370) 2022-04-25 11:56:50 +02:00
Till Faelligen
67fb086c13 Update README 2022-04-24 20:26:20 +02:00
Neil Alexander
6d78c4d67d
Fix retrieving cross-signing signatures in /user/devices/{userId} (#2368)
* Fix retrieving cross-signing signatures in `/user/devices/{userId}`

We need to know the target device IDs in order to get the signatures and we weren't populating those.

* Fix up signature retrieval

* Fix SQLite

* Always include the target's own signatures as well as the requesting user
2022-04-22 14:58:24 +01:00
Till
c07f347f00
Reuse the existing lazyload cache on /context and /messages (#2367) 2022-04-22 11:38:29 +02:00
kegsay
0eb5bd1e13
Update AWSY test groups (#2365)
* Update AWSY test groups

* Better names
2022-04-21 17:17:52 +01:00
Neil Alexander
144c060fa7
Remove libp2p demos (#2337)
* Remove libp2p demos

* Put it back
2022-04-21 13:31:43 +01:00
Neil Alexander
2258387d39
Update test list 2022-04-20 16:55:24 +01:00
Neil Alexander
54e7ea41c6
Eliminate more SQL no row errors in sync API (#2363)
* Handle `sql.ErrNoRows` in main `/sync` codepaths

* Catch more
2022-04-20 16:51:37 +01:00
Neil Alexander
bb987cd64b
Lazy loading fixes (#2362)
* Return some more usefully wrapped errors when doing sync

* Remove unnecessary error check

* Couple of guards around `sql.ErrNoRows`

* Nolint
2022-04-20 16:06:46 +01:00
Neil Alexander
a9f0a390c6
Update to NATS Server v2.8.0 and nats.go v1.14.0 (#2359) 2022-04-20 15:13:04 +01:00
Kegan Dougal
073972646f Use unix not second 2022-04-19 13:57:02 +01:00
Neil Alexander
85b1631ecf
Add newly passing test to list 2022-04-19 10:48:32 +01:00
Neil Alexander
711e377b9c
Update go-sqlite3-js to matrix-org/go-sqlite3-js#2 (SQLite 3.36.0) 2022-04-19 10:34:33 +01:00
Neil Alexander
7e745665a4
Change pushkey_ts to be seconds (fix #2354) (#2358) 2022-04-19 09:51:02 +01:00
fcwoknhenuxdfiyv-nextcloud
abf71649b0
Make sure resp.Username is defined before hashing. Fixes #2356 (#2357)
Co-authored-by: Jason Quigley <jason@onecha.net>
2022-04-19 09:46:54 +01:00
Till
57e3622b85
Implement lazy loading on /sync (#2346)
* Initial work on lazyloading

* Partially implement lazy loading on /sync

* Rename methods

* Make missing tests pass

* Preallocate slice, even if it will end up with fewer values

* Let the cache handle the user mapping

* Linter

* Cap cache growth
2022-04-19 09:46:45 +01:00
Till
3ddbffd59e
Refactor media storage layer, add tests (#2352)
* Refactor mediaapi storage layer

* Verify filetype before trying to create thumbnails

* Add media api storage tests

* Fix returned values
2022-04-14 13:32:48 +01:00
Neil Alexander
3a5e9a0f28
Use default sync filter if specified filter is not found (should fix #2350) (#2351) 2022-04-13 16:41:22 +01:00
Neil Alexander
1140f39993
Precompute values for userIDSet in sync notifier (#2348)
* Precompute values for `userIDSet` in sync notifier

* Mutexes

* Fixes

* Sensible initial value

* Update syncapi/notifier/notifier.go

Co-authored-by: Till <2353100+S7evinK@users.noreply.github.com>

* Placate the almighty linter

Co-authored-by: Till <2353100+S7evinK@users.noreply.github.com>
2022-04-13 12:35:30 +01:00
Till
29f2168789
Make /messages filterable (#2347)
* Make /messages filterable
Fix bug when determining if an event contains an URL

* Add newly passing test

* Fix test
2022-04-13 13:16:02 +02:00
kegsay
ea92f80c12
Add database namespacing for unit tests (#2340)
* Add database namespacing for unit tests

Background: Running `go test ./...` will run tests in different packages concurrently.
This can be stopped or limited by using `-p 1` (no concurrency). We want concurrency,
but this causes problems when running Postgres DBs in CI. The problem is that, in CI,
we have 1x postgres server exposing 1x postgres DB, which we wipe clean at the end of
each test via `defer close()`. When tests run concurrently, calls to `close()` will
delete data/tables which other tests are currently using, causing havoc.

Fix this by:
 - Creating a database per package.
 - Namespacing the database name by a hash of the current working directory (the directory containing those `_test.go` files)

This is exactly what SQLite does, quite unintentionally, via the use of `file:dendrite_test.db`,
which dumps the file into the current working directory which is the package running the tests,
hence deleting the file is safe when running concurrently.

* Linting

* Don't create the database in a txn

* dupe db is not an error
2022-04-11 10:23:01 +01:00
Till
69f2ff7c82
Correctly use provided filters (#2339)
* Apply filters correctly

* Fix issues; Use prepareWithFilters

* Update gmsl & tests

* go.mod..

* PR comments
2022-04-11 09:05:23 +02:00
kegsay
b4b2fbc36b Remove dead code in the sync api (#2341) 2022-04-09 00:37:50 +01:00
kegsay
6d25bd6ca5
syncapi: add more tests; fix more bugs (#2338)
* syncapi: add more tests; fix more bugs

bugfixes:
 - The postgres impl of TopologyTable.SelectEventIDsInRange did not use the provided txn
 - The postgres impl of EventsTable.SelectEvents did not preserve the ordering of the input event IDs in the output events slice
 - The sqlite impl of EventsTable.SelectEvents did not use a bulk `IN ($1)` query.

Added tests:
 - `TestGetEventsInRangeWithTopologyToken`
 - `TestOutputRoomEventsTable`
 - `TestTopologyTable`

* -p 1 for now
2022-04-08 17:53:24 +01:00
Neil Alexander
986d27a128
Update to matrix-org/pinecone@2999ea2 2022-04-08 16:39:09 +01:00
Neil Alexander
9bd9f2beba
Update to matrix-org/pinecone@9b3963248c 2022-04-08 16:23:50 +01:00
Neil Alexander
6907e42167
Use connection manager in Pinecone demos 2022-04-08 13:55:46 +01:00
Neil Alexander
4312b87ab0
Update dependencies (#2336) 2022-04-08 12:24:49 +01:00
Till
e8dd37d533
Add metrics for internal API requests (#2310)
* Add response size and requests total to internal handler

* Move MustRegister calls to New* funcs

* Move MustRegister back to init

* Init at some place, minimize changes
2022-04-08 12:24:40 +02:00
Neil Alexander
25d99c4473
Remove client-api-proxy and federation-api-proxy (closes #928) (#2335) 2022-04-08 10:59:51 +01:00
Neil Alexander
f299f97e0a
Fix data race in TestCorrectStreamWakeup (#2334) 2022-04-08 10:46:36 +01:00
Neil Alexander
b8c97431b9
Prevent setting push rules via account data endpoint (#2333) 2022-04-08 10:46:23 +01:00