dendrite/docs/development/coverage.md

title	parent	nav_order	permalink
Coverage	Development	3	/development/coverage

Running unit tests with coverage enabled

Running unit tests with coverage enabled can be done with the following commands, this will generate a integrationcover.log

go test -covermode=atomic -coverpkg=./... -coverprofile=integrationcover.log $(go list ./... | grep -v '/cmd/')
go tool cover -func=integrationcover.log

Running Sytest with coverage enabled

To run Sytest with coverage enabled:

docker run --rm --name sytest -v "/Users/kegan/github/sytest:/sytest" \
  -v "/Users/kegan/github/dendrite:/src" -v "$(pwd)/sytest_logs:/logs" \
  -v "/Users/kegan/go/:/gopath" -e "POSTGRES=1" \
  -e "COVER=1" \
  matrixdotorg/sytest-dendrite:latest
  
# to get a more accurate coverage you may also need to run Sytest using SQLite as the database:
docker run --rm --name sytest -v "/Users/kegan/github/sytest:/sytest" \
  -v "/Users/kegan/github/dendrite:/src" -v "$(pwd)/sytest_logs:/logs" \
  -v "/Users/kegan/go/:/gopath" \
  -e "COVER=1" \
  matrixdotorg/sytest-dendrite:latest

This will generate a folder covdatafiles in each server's directory, e.g server-0/covdatafiles. To parse them, ensure your working directory is under the Dendrite repository then run:

go tool covdata func -i="$(find -name 'covmeta*' -type f -exec dirname {} \; | uniq | paste -s -d ',' -)"

which will produce an output like:

...
github.com/matrix-org/util/json.go:132:   MakeJSONAPI         70.0%
github.com/matrix-org/util/json.go:151:   respond             84.6%
github.com/matrix-org/util/json.go:180:   WithCORSOptions      0.0%
github.com/matrix-org/util/json.go:191:   SetCORSHeaders     100.0%
github.com/matrix-org/util/json.go:202:   RandomString       100.0%
github.com/matrix-org/util/json.go:210:   init               100.0%
github.com/matrix-org/util/unique.go:13:  Unique              91.7%
github.com/matrix-org/util/unique.go:48:  SortAndUnique      100.0%
github.com/matrix-org/util/unique.go:55:  UniqueStrings      100.0%
total                                    (statements)         64.0%

(after running Sytest for Postgres and SQLite)

The total coverage for this run is the last line at the bottom. However, this value is misleading because Dendrite can run in different configurations, which will never be tested in a single test run (e.g sqlite or postgres). To get a more accurate value, you'll need run Sytest for Postgres and SQLite (see commands above). Additional processing is required also to remove packages which will never be tested and extension MSCs:

# If you executed both commands from above, you can get the total coverage using the following commands  
go tool covdata textfmt -i="$(find -name 'covmeta*' -type f -exec dirname {} \; | uniq | paste -s -d ',' -)" -o sytest.cov
grep -Ev 'relayapi|setup/mscs' sytest.cov > final.cov
go tool cover -func=final.cov

# If you only executed the one for Postgres:
go tool covdata textfmt -i="$(find -name 'covmeta*' -type f -exec dirname {} \; | uniq | paste -s -d ',' -)" -o sytest.cov
grep -Ev 'relayapi|sqlite|setup/mscs' sytest.cov > final.cov
go tool cover -func=final.cov

# If you only executed the one for SQLite:
go tool covdata textfmt -i="$(find -name 'covmeta*' -type f -exec dirname {} \; | uniq | paste -s -d ',' -)" -o sytest.cov
grep -Ev 'relayapi|postgres|setup/mscs' sytest.cov > final.cov
go tool cover -func=final.cov

Getting coverage from Complement

Getting the coverage for Complement runs is a bit more involved.

First you'll need a docker image compatible with Complement, one can be built using

docker build -t complement-dendrite -f build/scripts/Complement.Dockerfile .

from within the Dendrite repository.

Clone complement to a directory of your liking:

git clone https://github.com/matrix-org/complement.git
cd complement

Next we'll need a script to execute after a test finishes, create a new file posttest.sh, make the file executable (chmod +x posttest.sh) and add the following content:

#!/bin/bash

mkdir -p /tmp/Complement/logs/$2/$1/
docker cp $1:/tmp/covdatafiles/. /tmp/Complement/logs/$2/$1/

This will copy the covdatafiles files from each container to something like /tmp/Complement/logs/TestLogin/94f9c428de95779d2b62a3ccd8eab9d5ddcf65cc259a40ece06bdc61687ffed3/. ($1 is the containerID, $2 the test name)

Now that we have set up everything we need, we can finally execute Complement:

COMPLEMENT_BASE_IMAGE=complement-dendrite \
COMPLEMENT_SHARE_ENV_PREFIX=COMPLEMENT_DENDRITE_ \
COMPLEMENT_DENDRITE_COVER=1 \
COMPLEMENT_POST_TEST_SCRIPT=$(pwd)/posttest.sh \
  go test -tags dendrite_blacklist ./tests/... -count=1 -v -timeout=30m -failfast=false

Once this is done, you can copy the resulting covdatafiles files to your Dendrite repository for the next step.

cp -pr /tmp/Complement/logs PathToYourDendriteRepository

You can also run the following to get the coverage for Complement runs alone:

go tool covdata func -i="$(find /tmp/Complement -name 'covmeta*' -type f -exec dirname {} \; | uniq | paste -s -d ',' -)"

Combining the results of (almost) all runs

Now that we have all our covdatafiles files within the Dendrite repository, you can now execute the following command, to get the coverage overall (excluding unit tests):

go tool covdata func -i="$(find -name 'covmeta*' -type f -exec dirname {} \; | uniq | paste -s -d ',' -)"