Skip to content

The regtest fixture

Write a test

The pytest-regtest plugin provides multiple fixtures. To record output, use the fixture regtest that works like a file handle:

# test_squares.py

def test_squares(regtest):

    result = [i*i for i in range(10)]

    # one way to record output:
    print(result, file=regtest)

    # alternative method to record output:
    regtest.write("done")

Run the test

If you run this test script with pytest the first time there is no recorded output for this test function so far and thus the test will fail with a message including a diff:

$ pytest -v test_squares.py
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-9.0.3, pluggy-1.6.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-regtest/checkouts/latest/.venv/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmpajxjux7u
plugins: regtest-2.5.0, cov-7.1.0
collecting ... collected 1 item

test_squares.py::test_squares FAILED                                     [100%]

=================================== FAILURES ===================================
_________________________________ test_squares _________________________________

regression test output not recorded yet for test_squares.py::test_squares:

[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
done
---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 1
total number of failed snapshot tests  : 0
=========================== short test summary info ============================
FAILED test_squares.py::test_squares
============================== 1 failed in 0.12s ===============================

This is a diff of the current output is to a previously recorded output tobe. Since we did not record output yet, the diff contains no lines marked +.

Reset the test

To record the current output, we run pytest with the --regtest-reset flag:

$ pytest -v --regtest-reset test_squares.py
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-9.0.3, pluggy-1.6.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-regtest/checkouts/latest/.venv/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmpajxjux7u
plugins: regtest-2.5.0, cov-7.1.0
collecting ... collected 1 item

test_squares.py::test_squares RESET                                      [100%]

---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 0
total number of failed snapshot tests  : 0
the following output files have been reset:
  _regtest_outputs/test_squares.test_squares.out
============================== 1 passed in 0.01s ===============================

You can also see from the output that the recorded output is in the _regtest_outputs folder which in the same folder as the test script. Don't forget to commit this folder to your version control system!

Run the test again

$ pytest -v test_squares.py
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-9.0.3, pluggy-1.6.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-regtest/checkouts/latest/.venv/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmpajxjux7u
plugins: regtest-2.5.0, cov-7.1.0
collecting ... collected 1 item

test_squares.py::test_squares PASSED                                     [100%]

---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 0
total number of failed snapshot tests  : 0
============================== 1 passed in 0.01s ===============================

Break the test

Let us break the test by changing the test function to compute 11 instead of 10 square numbers:

# test_squares.py

def test_squares(regtest):

    result = [i*i for i in range(11)]

    # one way to record output:
    print(result, file=regtest)

    # alternative method to record output:
    regtest.write("done")

The next run of pytest delivers a nice diff of the current and expected output from this test function:

$ pytest -v test_squares.py
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-9.0.3, pluggy-1.6.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-regtest/checkouts/latest/.venv/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmpajxjux7u
plugins: regtest-2.5.0, cov-7.1.0
collecting ... collected 1 item

test_squares.py::test_squares FAILED                                     [100%]

=================================== FAILURES ===================================
_________________________________ test_squares _________________________________

regression test output differences for test_squares.py::test_squares:
    (recorded output from _regtest_outputs/test_squares.test_squares)

    >   --- current
    >   +++ expected
    >   @@ -1,2 +1,2 @@
    >   -[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
    >   +[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
    >    done

---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 1
total number of failed snapshot tests  : 0
=========================== short test summary info ============================
FAILED test_squares.py::test_squares
============================== 1 failed in 0.12s ===============================

In case the change was intended, you can reset the test again:

$ pytest -v --regtest-reset test_squares.py
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-9.0.3, pluggy-1.6.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-regtest/checkouts/latest/.venv/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmpajxjux7u
plugins: regtest-2.5.0, cov-7.1.0
collecting ... collected 1 item

test_squares.py::test_squares RESET                                      [100%]

---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 0
total number of failed snapshot tests  : 0
the following output files have been reset:
  _regtest_outputs/test_squares.test_squares.out
============================== 1 passed in 0.01s ===============================

Multiple outputs in one test

You can record multiple independent outputs in a single test by using regtest.version (or its alias regtest.identifier):

def test_multi_stage(regtest):
    regtest.version = "stage_1"
    print("Results from stage 1", file=regtest)

    regtest.version = "stage_2"
    print("Results from stage 2", file=regtest)
This will create two separate output files in the _regtest_outputs folder.

Platform specific output

Another use case for versions is to implement platform-specific regression testing. This is useful when the output of a function is expected to differ between operating systems (e.g., due to different path separators or floating point implementations):

import sys

def test_platform_specific(regtest):
    regtest.version = sys.platform
    print(f"Running on {sys.platform}", file=regtest)
    # ... platform specific results ...

To use this, you need to run pytest --regtest-reset on each supported platform and commit all resulting files in the _regtest_outputs folder to your repository.