Commit Graph

30 Commits

Author SHA1 Message Date
Till Faelligen
39581af3ba
CI update 2022-10-10 15:49:56 +02:00
Neil Alexander
085bf5e28b
Revert Docker changes 2022-10-04 11:33:05 +01:00
Till
e6c992ba8b
Update Dockerfile (#2342)
Updates/adds a new multistage (build-kit) Dockerfile. (if accepted,
could make `Dockerfile.monolith` and `Dockerfile.polylith` in
`build/docker` obsolete)
There's no huge difference between the dockerfiles, except this uses a
non-root user when running the container, also doesn't copy the working
directory to the image when building.
Also adds vulnerabilities scans using
[Trivy](https://github.com/aquasecurity/trivy) for the created docker
images. (untested)

Building images is done using 
```
docker build . --target image-monolith -t dendrite-monolith
docker build . --target image-polylith -t dendrite-polylith
```

As noted in the comments, only adds `dendrite-polylith-multi` to the
polylith image and all required binaries to the monolith image.
Probably needs some docs updating, if this is accepted.

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-10-04 10:41:06 +01:00
Till
95a509757a
Complement QoL changes (#2663)
This PR does the following:
- adds a `keysize` parameter to `generate-keys`, so we can use lower sized keys when running in CI
- updates the Complement docker files to use BuildKit (requires Docker >18.09)
- uses `exec` when executing `dendrite-monotlith-server`, making it PID 1 inside docker, which results in Dendrite actually receiving the `SIGTERM` signal send by Docker. (Making it faster when running tests with Complement, as we don't take 10 seconds to timeout)
2022-08-23 13:10:29 +02:00
Neil Alexander
606cb67506
Enable workflow_dispatch in GHA 2022-08-18 13:50:58 +01:00
Tak Wai Wong
fad3ac8e78
Protect user_interactive reads and writes with locks (#2635)
* Protect user_interactive reads and writes with locks

* Ignore golangci-lint false positive

* fix lint

Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
2022-08-12 09:12:05 +01:00
Neil Alexander
2b352915a1
Update golangci-lint component in GHA workflow 2022-08-11 17:40:57 +01:00
Neil Alexander
bbff41b44b
Disable stack protector on Linux CI build pipelines for now (to avoid relocation target __stack_chk_fail_local not defined errors) 2022-08-03 10:50:45 +01:00
Neil Alexander
f7f2453a85
Test Go 1.19 in CI 2022-08-03 10:35:57 +01:00
Till
081f5e7226
Update database migrations, remove goose (#2264)
* Add new db migration

* Update migrations
Remove goose

* Add possibility to test direct upgrades

* Try to fix WASM test

* Add checks for specific migrations

* Remove AddMigration
Use WithTransaction
Add Dendrite version to table

* Fix linter issues

* Update tests

* Update comments, outdent if

* Namespace migrations

* Add direct upgrade tests, skipping over one version

* Split migrations

* Update go version in CI

* Fix copy&paste mistake

* Use contexts in migrations

Co-authored-by: kegsay <kegan@matrix.org>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-07-25 10:39:22 +01:00
Neil Alexander
3ea21273bc
Ristretto cache (#2563)
* Try Ristretto cache

* Tweak

* It's beautiful

* Update GMSL

* More strict keyable interface

* Fix that some more

* Make less panicky

* Don't enforce mutability checks for now

* Determine mutability using deep equality

* Tweaks

* Namespace keys

* Make federation caches mutable

* Update cost estimation, add metric

* Update GMSL

* Estimate cost for metrics better

* Reduce counters a bit

* Try caching events

* Some guards

* Try again

* Try this

* Use separate caches for hopefully better hash distribution

* Fix bug with admitting events into cache

* Try to fix bugs

* Check nil

* Try that again

* Preserve order jeezo this is messy

* thanks VS Code for doing exactly the wrong thing

* Try this again

* Be more specific

* aaaaargh

* One more time

* That might be better

* Stronger sorting

* Cache expiries, async publishing of EDUs

* Put it back

* Use a shared cache again

* Cost estimation fixes

* Update ristretto

* Reduce counters a bit

* Clean up a bit

* Update GMSL

* 1GB

* Configurable cache sizees

* Tweaks

* Add `config.DataUnit` for specifying friendly cache sizes

* Various tweaks

* Update GMSL

* Add back some lazy loading caching

* Include key in cost

* Include key in cost

* Tweak max age handling, config key name

* Only register prometheus metrics if requested

* Review comments @S7evinK

* Don't return errors when creating caches (it is better just to crash since otherwise we'll `nil`-pointer exception everywhere)

* Review comments

* Update sample configs

* Update GHA Workflow

* Update Complement images to Go 1.18

* Remove the cache test from the federation API as we no longer guarantee immediate cache admission

* Don't check the caches in the renewal test

* Possibly fix the upgrade tests

* Update to matrix-org/gomatrixserverlib#322

* Update documentation to refer to Go 1.18
2022-07-11 14:31:31 +01:00
Neil Alexander
086f182e24
Disable WebAssembly builds for now 2022-07-01 09:50:06 +01:00
Till Faelligen
7df5d69a5b Checkout correct branch for Sytest 2022-04-26 08:07:27 +02:00
kegsay
ea92f80c12
Add database namespacing for unit tests (#2340)
* Add database namespacing for unit tests

Background: Running `go test ./...` will run tests in different packages concurrently.
This can be stopped or limited by using `-p 1` (no concurrency). We want concurrency,
but this causes problems when running Postgres DBs in CI. The problem is that, in CI,
we have 1x postgres server exposing 1x postgres DB, which we wipe clean at the end of
each test via `defer close()`. When tests run concurrently, calls to `close()` will
delete data/tables which other tests are currently using, causing havoc.

Fix this by:
 - Creating a database per package.
 - Namespacing the database name by a hash of the current working directory (the directory containing those `_test.go` files)

This is exactly what SQLite does, quite unintentionally, via the use of `file:dendrite_test.db`,
which dumps the file into the current working directory which is the package running the tests,
hence deleting the file is safe when running concurrently.

* Linting

* Don't create the database in a txn

* dupe db is not an error
2022-04-11 10:23:01 +01:00
kegsay
6d25bd6ca5
syncapi: add more tests; fix more bugs (#2338)
* syncapi: add more tests; fix more bugs

bugfixes:
 - The postgres impl of TopologyTable.SelectEventIDsInRange did not use the provided txn
 - The postgres impl of EventsTable.SelectEvents did not preserve the ordering of the input event IDs in the output events slice
 - The sqlite impl of EventsTable.SelectEvents did not use a bulk `IN ($1)` query.

Added tests:
 - `TestGetEventsInRangeWithTopologyToken`
 - `TestOutputRoomEventsTable`
 - `TestTopologyTable`

* -p 1 for now
2022-04-08 17:53:24 +01:00
kegsay
7499147550
Add test infrastructure code for dendrite unit/integ tests (#2331)
* Add test infrastructure code for dendrite unit/integ tests

Start re-enabling some syncapi storage tests in the process.

* Linting

* Add postgres service to unit tests

* dendrite not syncv3

* Skip test which doesn't work

* Linting

* Add `jetstream.PrepareForTests`

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-04-08 10:12:30 +01:00
Till
c84937b852
Add Are We Synapse Yet to GHA (#2321)
* Add Are We Synapse Yet to GHA

* Better output & add comments
2022-04-05 15:32:30 +02:00
Neil Alexander
1554d51b37
Pass DOCKER_TOKEN secret into Docker workflow 2022-03-25 14:27:41 +00:00
Neil Alexander
565b5423ea
One final tweak to the GHA pipeline 2022-03-25 13:41:28 +00:00
Neil Alexander
5b5e6a59b6
Give packages permission to update-docker-images 2022-03-25 13:31:41 +00:00
Neil Alexander
7f3d42bb46
Use correct path name 2022-03-25 13:29:09 +00:00
Neil Alexander
62bd559275
Factor Docker step into own job 2022-03-25 13:28:20 +00:00
Neil Alexander
c8e1ad5997
Specify branch name in Docker flow 2022-03-25 13:26:18 +00:00
Neil Alexander
28642683fc
Fix Docker flow 2022-03-25 13:25:02 +00:00
Neil Alexander
5e780d3ca2
Chain Docker update onto main 2022-03-25 10:08:13 +00:00
Neil Alexander
ad818a4370
Update dendrite.yml 2022-03-24 12:41:30 +00:00
Neil Alexander
c2a27efc36
Update on section of GHA workflow 2022-03-24 12:40:44 +00:00
Neil Alexander
fe5148c12d
Check for success of initial tests on GHA 2022-03-24 12:14:35 +00:00
Neil Alexander
3bdda65bc4
Hopefully fail Initial tests passed if they actually failed (#2298) 2022-03-24 12:05:36 +00:00
S7evinK
01f863d248
Move CI to Github Actions (#2297)
* Initial test

* Move CI to GHA

* Naming

* Always report all linter issues

* Remove if true

* Test complement in different variations

* Try again

* Move Complement back after initial tests and readd timeout
Make linting fail further checks
Remove CodeQL

* Update and rename tests.yml to dendrite.yml

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2022-03-24 12:52:51 +01:00