smhk

GitLab: Enable coverage reporting with pytest

GitLab supports coverage reporting. There are essential two types of coverage:

  • Total coverage, which is a single percentage displayed next to a job (or in Merge Requests, or badges).
  • Detailed coverage, which is a line-by-line coverage visualisation.

Following are the steps for configuring pytest to generate coverage statistics, and hook both of these coverage types up to GitLab.

Update tox.ini to generate terminal and XML coverage §

Assuming your Python package is called my_package, update your pytest call in tox.ini to use --cov-report as follows:

tox.ini
[testenv]
deps =
    pytest
    pytest-cov
commands =
    pytest \
        --cov-report=term \
        --cov-report=xml:coverage/{envname}/coverage.xml \
        --cov=my_package \
        tests

The important parts are:

  • For the total coverage:
    • --cov-report=term - to print the coverage out to stdout. This will be parsed by GitLab to display the total coverage percentage in the GitLab UI.
  • For the detailed coverage:
    • --cov-report=xml:<dir> - to write the coverage out in XML format. This will be used for detailed coverage reports in the GitLab UI.

Update .gitlab-ci.yml to generate coverage artifacts §

Then, in .gitlab-ci.yml, use the coverage_report field with the following configuration to get GitLab to extract the coverage:

.gitlab-ci.yml
unit_test:py310:
    stage: test
    artifacts:
        reports:
            coverage_report:
                coverage_format: cobertura
                path: 'coverage/py310/coverage.xml'
    coverage: '/TOTAL\s+\d+\s+\d+\s+(\d+%)/'

The coverage_report is a special field used by GitLab Covertura Coverage to extract the coverage results.1

The important parts are:

  • For the total coverage:
    • coverage: <regex> - this defines the regex used to extract the total coverage percentage from stdout.
  • For the detailed coverage:
    • artifacts: ... reports: ... coverage_report: - this provides the coverage report to GitLab.
    • path: '<dir>' - this must be the same directory as in tox.ini.

For example, your stdout might look like:

---------- coverage: platform linux, python 3.10.12-final-0 ----------
Name                                    Stmts   Miss  Cover
-----------------------------------------------------------
src/my_package/__init__.py                  0      0   100%
src/my_package/file_1.py                  102     12    88%
src/my_package/file_2.py                  128     59    54%
src/my_package/file_3.py                   24      4    83%
...
src/my_package/test/__init__.py             0      0   100%
src/my_package/test/test_1.py             103      0   100%
src/my_package/test/test_2.py              62      0   100%
src/my_package/test/test_3.py              47      0   100%
...
-----------------------------------------------------------
TOTAL                                     987    145    85%
Coverage XML written to file coverage/py310/coverage.xml
=================== 98 passed, 7 skipped in 77.14s (0:01:17) ===================
  py310: OK (160.28=setup[71.12]+cmd[9.29,79.88] seconds)
  congratulations :) (160.58 seconds)

Then the regex is extracting the line starting with TOTAL.2

Bonus: separate coverage directory for each Python version §

We can use $(echo $CI_JOB_NAME | cut -d : -f 2) to extract the Python version from the job name. For example, this will convert unit_test:py310 to py310.

This can be useful if you wish to have a separate coverage directory for each Python version:

.gitlab-ci.yml
.unit_test_template: &unit_test_template
    stage: test
    artifacts:
        reports:
            coverage_report:
                coverage_format: cobertura
                path: 'coverage/$(echo $CI_JOB_NAME | cut -d : -f 2)/coverage.xml'
    coverage: '/TOTAL\s+\d+\s+\d+\s+(\d+%)/'

unit_test:py310:
    <<: *unit_test_template
    script:
        - tox -e py310

unit_test:py311:
    <<: *unit_test_template
    script:
        - tox -e py311

  1. At time of writing, GitLab supports two types of detailed overage: Cobertura and JaCoCo. These notes use Cobertura. ↩︎

  2. The GitLab documentation includes many example regexes, including one for pytest, but it did not manage to match for my pytest output. ↩︎