mirror of
https://github.com/1f349/dendrite.git
synced 2024-11-09 22:42:58 +00:00
Refine config and docs for sytest (#714)
This PR adds the "passed but expected fail" tests to testfile, as well as instructions on finding which tests to add for new PRs.
This commit is contained in:
parent
bc382bba46
commit
adaec7c8e7
@ -1,8 +1,78 @@
|
|||||||
# SyTest
|
# SyTest
|
||||||
|
|
||||||
Dendrite uses [SyTest](https://github.com/matrix-org/sytest) for its
|
Dendrite uses [SyTest](https://github.com/matrix-org/sytest) for its
|
||||||
integration testing. When creating a new PR, add the test IDs that your PR
|
integration testing. When creating a new PR, add the test IDs (see below) that
|
||||||
should allow to pass to `testfile` in dendrite's root directory. Not all PRs
|
your PR should allow to pass to `testfile` in dendrite's root directory. Not all
|
||||||
need to make new tests pass. If we find your PR should be making a test pass we
|
PRs need to make new tests pass. If we find your PR should be making a test pass
|
||||||
may ask you to add to that file, as generally Dendrite's progress can be
|
we may ask you to add to that file, as generally Dendrite's progress can be
|
||||||
tracked through the amount of SyTest tests it passes.
|
tracked through the amount of SyTest tests it passes.
|
||||||
|
|
||||||
|
## Finding out which tests to add
|
||||||
|
|
||||||
|
We recommend you run the tests locally by manually setting up SyTest or using a
|
||||||
|
SyTest docker image. After running the tests, a script will print the tests you
|
||||||
|
need to add to `testfile` for you.
|
||||||
|
|
||||||
|
You should proceed after you see no build problems for dendrite after running:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
./build.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
### Manually Setting up SyTest
|
||||||
|
|
||||||
|
Make sure you have Perl v5+ installed, and get SyTest with:
|
||||||
|
|
||||||
|
(Note that this guide assumes your SyTest checkout is next to your
|
||||||
|
`dendrite` checkout.)
|
||||||
|
|
||||||
|
```sh
|
||||||
|
git clone -b develop https://github.com/matrix-org/sytest
|
||||||
|
cd sytest
|
||||||
|
./install-deps.pl
|
||||||
|
```
|
||||||
|
|
||||||
|
Set up the database:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
sudo -u postgres psql -c "CREATE USER dendrite PASSWORD 'itsasecret'"
|
||||||
|
sudo -u postgres psql -c "CREATE DATABASE sytest_template OWNER dendrite"
|
||||||
|
mkdir -p "server-0"
|
||||||
|
cat > "server-0/database.yaml" << EOF
|
||||||
|
args:
|
||||||
|
user: dendrite
|
||||||
|
database: dendrite
|
||||||
|
host: 127.0.0.1
|
||||||
|
type: pg
|
||||||
|
EOF
|
||||||
|
```
|
||||||
|
|
||||||
|
Run the tests:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
./run-tests.pl -I Dendrite::Monolith -d ../dendrite/bin -W ../dendrite/testfile -O tap --all | tee results.tap
|
||||||
|
```
|
||||||
|
|
||||||
|
where `tee` lets you see the results while they're being piped to the file.
|
||||||
|
|
||||||
|
Once the tests are complete, run the helper script to see if you need to add
|
||||||
|
any newly passing test names to `testfile` in the project's root directory:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
../dendrite/show-expected-fail-tests.sh results.tap
|
||||||
|
```
|
||||||
|
|
||||||
|
If the script prints nothing/exits with 0, then you're good to go.
|
||||||
|
|
||||||
|
### Using a SyTest Docker image
|
||||||
|
|
||||||
|
Ensure you have the latest image for SyTest, then run the tests:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
docker pull matrixdotorg/sytest-dendrite
|
||||||
|
docker run --rm -v /path/to/dendrite/:/src/ matrixdotorg/sytest-dendrite
|
||||||
|
```
|
||||||
|
|
||||||
|
where `/path/to/dendrite/` should be replaced with the actual path to your
|
||||||
|
dendrite source code. The output should tell you if you need to add any tests to
|
||||||
|
`testfile`.
|
||||||
|
29
show-expected-fail-tests.sh
Executable file
29
show-expected-fail-tests.sh
Executable file
@ -0,0 +1,29 @@
|
|||||||
|
#! /bin/bash
|
||||||
|
|
||||||
|
results_file=$1
|
||||||
|
passed_but_expected_fail=$(grep ' # TODO passed but expected fail' ${results_file} | sed -E 's/^ok [0-9]+ (\(expected fail\) )?//' | sed -E 's/( \([0-9]+ subtests\))? # TODO passed but expected fail$//')
|
||||||
|
tests_to_add=""
|
||||||
|
already_in_testfile=""
|
||||||
|
|
||||||
|
fail_build=0
|
||||||
|
while read -r test_id; do
|
||||||
|
grep "${test_id}" testfile > /dev/null 2>&1
|
||||||
|
if [ "$?" != "0" ]; then
|
||||||
|
tests_to_add="${tests_to_add}${test_id}\n"
|
||||||
|
fail_build=1
|
||||||
|
else
|
||||||
|
already_in_testfile="${already_in_testfile}${test_id}\n"
|
||||||
|
fi
|
||||||
|
done <<< "${passed_but_expected_fail}"
|
||||||
|
|
||||||
|
if [ -n "${tests_to_add}" ]; then
|
||||||
|
echo "ERROR: The following passed tests are not present in testfile. Please append them to the file:"
|
||||||
|
echo -e "${tests_to_add}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ -n "${already_in_testfile}" ]; then
|
||||||
|
echo "WARN: Tests in testfile still marked as expected fail:"
|
||||||
|
echo -e "${already_in_testfile}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
exit ${fail_build}
|
142
testfile
142
testfile
@ -1,4 +1,144 @@
|
|||||||
GET /register yields a set of flows
|
GET /register yields a set of flows
|
||||||
POST /register can create a user
|
POST /register can create a user
|
||||||
POST /register downcases capitals in usernames
|
POST /register downcases capitals in usernames
|
||||||
|
POST /register rejects registration of usernames with '!'
|
||||||
|
POST /register rejects registration of usernames with '"'
|
||||||
|
POST /register rejects registration of usernames with ':'
|
||||||
|
POST /register rejects registration of usernames with '?'
|
||||||
|
POST /register rejects registration of usernames with '\'
|
||||||
|
POST /register rejects registration of usernames with '@'
|
||||||
|
POST /register rejects registration of usernames with '['
|
||||||
|
POST /register rejects registration of usernames with ']'
|
||||||
|
POST /register rejects registration of usernames with '{'
|
||||||
|
POST /register rejects registration of usernames with '|'
|
||||||
|
POST /register rejects registration of usernames with '}'
|
||||||
|
POST /register rejects registration of usernames with '£'
|
||||||
|
POST /register rejects registration of usernames with 'é'
|
||||||
|
POST /register rejects registration of usernames with '\n'
|
||||||
|
POST /register rejects registration of usernames with '''
|
||||||
|
GET /login yields a set of flows
|
||||||
|
POST /login can log in as a user
|
||||||
|
POST /login can log in as a user with just the local part of the id
|
||||||
|
POST /login as non-existing user is rejected
|
||||||
|
POST /login wrong password is rejected
|
||||||
|
GET /events initially
|
||||||
|
GET /initialSync initially
|
||||||
|
Version responds 200 OK with valid structure
|
||||||
|
PUT /profile/:user_id/displayname sets my name
|
||||||
|
GET /profile/:user_id/displayname publicly accessible
|
||||||
|
PUT /profile/:user_id/avatar_url sets my avatar
|
||||||
|
GET /profile/:user_id/avatar_url publicly accessible
|
||||||
|
GET /device/{deviceId} gives a 404 for unknown devices
|
||||||
|
PUT /device/{deviceId} gives a 404 for unknown devices
|
||||||
|
POST /createRoom makes a public room
|
||||||
|
POST /createRoom makes a private room
|
||||||
|
POST /createRoom makes a private room with invites
|
||||||
|
POST /createRoom makes a room with a name
|
||||||
|
POST /createRoom makes a room with a topic
|
||||||
|
Can /sync newly created room
|
||||||
|
GET /rooms/:room_id/state/m.room.member/:user_id fetches my membership
|
||||||
|
GET /rooms/:room_id/state/m.room.power_levels fetches powerlevels
|
||||||
|
POST /join/:room_alias can join a room
|
||||||
|
POST /join/:room_id can join a room
|
||||||
|
POST /join/:room_id can join a room with custom content
|
||||||
|
POST /join/:room_alias can join a room with custom content
|
||||||
|
POST /rooms/:room_id/leave can leave a room
|
||||||
|
POST /rooms/:room_id/invite can send an invite
|
||||||
|
POST /rooms/:room_id/ban can ban a user
|
||||||
|
POST /rooms/:room_id/send/:event_type sends a message
|
||||||
|
PUT /rooms/:room_id/send/:event_type/:txn_id sends a message
|
||||||
|
PUT /rooms/:room_id/send/:event_type/:txn_id deduplicates the same txn id
|
||||||
|
GET /rooms/:room_id/state/m.room.power_levels can fetch levels
|
||||||
|
PUT /rooms/:room_id/state/m.room.power_levels can set levels
|
||||||
|
PUT power_levels should not explode if the old power levels were empty
|
||||||
|
Both GET and PUT work
|
||||||
|
POST /rooms/:room_id/read_markers can create read marker
|
||||||
|
User signups are forbidden from starting with '_'
|
||||||
|
Can logout all devices
|
||||||
|
Request to logout with invalid an access token is rejected
|
||||||
|
Request to logout without an access token is rejected
|
||||||
|
Room creation reports m.room.create to myself
|
||||||
|
Room creation reports m.room.member to myself
|
||||||
|
New room members see their own join event
|
||||||
|
Existing members see new members' join events
|
||||||
|
setting 'm.room.power_levels' respects room powerlevel
|
||||||
|
Unprivileged users can set m.room.topic if it only needs level 0
|
||||||
|
Users cannot set ban powerlevel higher than their own
|
||||||
|
Users cannot set kick powerlevel higher than their own
|
||||||
|
Users cannot set redact powerlevel higher than their own
|
||||||
|
Can get rooms/{roomId}/members for a departed room (SPEC-216)
|
||||||
|
3pid invite join with wrong but valid signature are rejected
|
||||||
|
3pid invite join valid signature but revoked keys are rejected
|
||||||
|
3pid invite join valid signature but unreachable ID server are rejected
|
||||||
|
Room members can override their displayname on a room-specific basis
|
||||||
|
Room members can join a room with an overridden displayname
|
||||||
|
displayname updates affect room member events
|
||||||
|
avatar_url updates affect room member events
|
||||||
|
Real non-joined user cannot call /events on shared room
|
||||||
|
Real non-joined user cannot call /events on invited room
|
||||||
|
Real non-joined user cannot call /events on joined room
|
||||||
|
Real non-joined user cannot call /events on default room
|
||||||
|
Real non-joined users can get state for world_readable rooms
|
||||||
|
Real non-joined users can get individual state for world_readable rooms
|
||||||
|
Real non-joined users can get individual state for world_readable rooms after leaving
|
||||||
|
Real non-joined users cannot send messages to guest_access rooms if not joined
|
||||||
|
Real users can sync from world_readable guest_access rooms if joined
|
||||||
|
Real users can sync from default guest_access rooms if joined
|
||||||
|
Can't forget room you're still in
|
||||||
|
Can get rooms/{roomId}/members
|
||||||
|
Can create filter
|
||||||
|
Can download filter
|
||||||
|
Can sync
|
||||||
|
Can sync a joined room
|
||||||
|
Newly joined room is included in an incremental sync
|
||||||
|
User is offline if they set_presence=offline in their sync
|
||||||
|
Changes to state are included in an incremental sync
|
||||||
|
A change to displayname should appear in incremental /sync
|
||||||
|
Current state appears in timeline in private history
|
||||||
|
Current state appears in timeline in private history with many messages before
|
||||||
|
Rooms a user is invited to appear in an initial sync
|
||||||
|
Rooms a user is invited to appear in an incremental sync
|
||||||
|
Sync can be polled for updates
|
||||||
|
Sync is woken up for leaves
|
||||||
|
Newly left rooms appear in the leave section of incremental sync
|
||||||
|
We should see our own leave event, even if history_visibility is restricted (SYN-662)
|
||||||
|
We should see our own leave event when rejecting an invite, even if history_visibility is restricted (riot-web/3462)
|
||||||
|
Newly left rooms appear in the leave section of gapped sync
|
||||||
|
Previously left rooms don't appear in the leave section of sync
|
||||||
|
Left rooms appear in the leave section of full state sync
|
||||||
|
Newly banned rooms appear in the leave section of incremental sync
|
||||||
|
Newly banned rooms appear in the leave section of incremental sync
|
||||||
|
local user can join room with version 1
|
||||||
|
User can invite local user to room with version 1
|
||||||
|
local user can join room with version 2
|
||||||
|
User can invite local user to room with version 2
|
||||||
|
local user can join room with version 3
|
||||||
|
User can invite local user to room with version 3
|
||||||
|
local user can join room with version 4
|
||||||
|
User can invite local user to room with version 4
|
||||||
|
Should reject keys claiming to belong to a different user
|
||||||
|
Can add account data
|
||||||
|
Can add account data to room
|
||||||
|
Latest account data appears in v2 /sync
|
||||||
|
New account data appears in incremental v2 /sync
|
||||||
|
Checking local federation server
|
||||||
|
Inbound federation can query profile data
|
||||||
|
Outbound federation can send room-join requests
|
||||||
|
Outbound federation can send events
|
||||||
|
Inbound federation can backfill events
|
||||||
|
Backfill checks the events requested belong to the room
|
||||||
|
Can upload without a file name
|
||||||
|
Can download without a file name locally
|
||||||
|
Can upload with ASCII file name
|
||||||
|
Can send image in room message
|
||||||
|
AS cannot create users outside its own namespace
|
||||||
|
Regular users cannot register within the AS namespace
|
||||||
|
AS can't set displayname for random users
|
||||||
|
AS user (not ghost) can join room without registering, with user_id query param
|
||||||
|
Changing the actions of an unknown default rule fails with 404
|
||||||
|
Changing the actions of an unknown rule fails with 404
|
||||||
|
Enabling an unknown default rule fails with 404
|
||||||
|
Trying to get push rules with unknown rule_id fails with 404
|
||||||
|
Events come down the correct room
|
||||||
|
local user can join room with version 5
|
||||||
|
User can invite local user to room with version 5
|
||||||
|
Loading…
Reference in New Issue
Block a user