Platform Dependent Python Coverage Test with Tox

Last updated on December 20, 2020

When testing Python programs, coverage.py is often used in measuring code coverage, and enforcing 100% code coverage is regarded as a good practice:

# .coveragerc
[coverage:report]
# Enforce 100% coverage test
fail_under = 100
show_missing = True

However, if there are some lines of code that are platform dependent (i.e., they are never executed on at least one platform), code coverage tests usually fail. For example, the following code snippet would always lead to a coverage that is less than 100% on a platform other than Windows:

if os.name != 'nt':
    # Do something if the OS is not Windows...

You can ask coverage.py to ignore this block by adding a comment # pragma: no cover, but then coverage.py would ignore it on all platforms, including all non-Windows platforms. If you use tox for testing, this issue can be resolved cleanly.

First, mark your Python code that you would like to exclude with some platform-specific coverage comments:

if os.name != 'nt':  # pragma: no cover windows. No execution of this branch on Windows
    # Do something if the OS is not Windows...

For Linux and MacOS exclusions, use # pragma: no cover linux and # pragma: no cover macos, respectively. We will now try to make coverage.py ignore these specifically marked lines. For the example above, we will make coverage.py ignore the marked if-block on Windows and only on Windows.

In tox.ini, add the following part to make sure this tox configuration is platform-aware (see the tox documentation for how this works):

# tox.ini
[tox]
envlist = py{36,37,38,39}-{linux,macos,windows}
[testenv]
platform = linux: linux
           macos: darwin
           windows: win32
setenv =
    linux: PLATFORM = linux
    macos: PLATFORM = macos
    windows: PLATFORM = windows

Then, create a template coveragerc file, named .coveragerc.in, including the following section along with your other coverage.py settings:

# .coveragerc.in
[coverage:report]
exclude_lines =
    pragma: no cover {platform}

These lines mean that coverage.py should ignore a line if pragma: no cover {platform} is present.

Back to tox.ini, include the following lines:

# tox.ini
[testenv:py{36,37,38,39}-{linux,macos,windows}]
setenv =
    COVERAGE_RCFILE = {envtmpdir}/coveragerc
commands_pre =
    {envpython} -c 'from pathlib import Path; Path(r"{env:COVERAGE_RCFILE}").write_text(Path(".coveragerc.in").read_text().format(platform="{env:PLATFORM}"))'
commands =
    coverage run -m pytest -v

These lines read from .coveragerc.in, substitute {platform} with the appropriate platform name, and write it to a temporary file that coverage.py recognizes as the configuration file for this run.

Now, running tox -e py39-windows (or other Python versions) will ignore the lines marked with # pragma: no cover windows for coverage purposes, and tox -e py39-linux (or other Python versions and OSes) will not ignore those lines.

For a live example, check out the flake8-executable code repository, specifically tox.ini, .coveragerc.in, and the Python source file with the marks.

Leave a Reply

Your email address will not be published. Required fields are marked *