Skip to content

Taking snapshots of NumPy arrays

pytest-regtest implements snapshot testing for NumPy arrays of arbitrary dimension. snapshot.check offers options for managing numerical accuracies and implements tailored diagnostics for failing tests.

Write a snapshot test for a NumPy matrix

The following test checks a 3 times 3 matrix:

# test_squares.py

import numpy as np

def test_squares(snapshot):
    result = np.arange(9.0).reshape(3, 3)
    snapshot.check(result)

Run the test

If you run this test script with pytest the first time there is no recorded output for this test function so far and thus the test will fail with a message including a diff:

$ pytest -v test_squares.py
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-9.0.3, pluggy-1.6.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-regtest/checkouts/latest/.venv/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmpan6p154i
plugins: regtest-2.5.0, cov-7.1.0
collecting ... collected 1 item

test_squares.py::test_squares FAILED                                     [100%]

=================================== FAILURES ===================================
_________________________________ test_squares _________________________________

snapshot error(s) for test_squares.py::test_squares:

snapshot not recorded yet:
    > test_squares.py +5
    > snapshot.check(result)
    [[0. 1. 2.]
     [3. 4. 5.]
     [6. 7. 8.]]
---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 0
total number of failed snapshot tests  : 1
=========================== short test summary info ============================
FAILED test_squares.py::test_squares
============================== 1 failed in 0.12s ===============================

This is a diff of the current output is to a previously recorded output tobe. Since we did not record output yet, the diff contains no lines marked +.

Reset the test

To record the current output, we run pytest with the --regtest-reset flag:

$ pytest -v --regtest-reset test_squares.py
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-9.0.3, pluggy-1.6.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-regtest/checkouts/latest/.venv/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmpan6p154i
plugins: regtest-2.5.0, cov-7.1.0
collecting ... collected 1 item

test_squares.py::test_squares RESET                                      [100%]

---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 0
total number of failed snapshot tests  : 0
the following output files have been reset:
  _regtest_outputs/test_squares.test_squares__0
============================== 1 passed in 0.01s ===============================

You can also see from the output that the recorded output is in the _regtest_outputs folder which in the same folder as the test script. Don't forget to commit this folder to your version control system!

Run the test again

$ pytest -v test_squares.py
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-9.0.3, pluggy-1.6.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-regtest/checkouts/latest/.venv/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmpan6p154i
plugins: regtest-2.5.0, cov-7.1.0
collecting ... collected 1 item

test_squares.py::test_squares PASSED                                     [100%]

---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 0
total number of failed snapshot tests  : 0
============================== 1 passed in 0.01s ===============================

Break the test

Let us break the test by changing the test function to compute 11 instead of 10 square numbers:

# test_squares.py

import numpy as np

def test_squares(snapshot):
    result = np.arange(12.0).reshape(4, 3)
    result[:, 0] += 1
    snapshot.check(result)

The next run of pytest delivers a nice diff of the current and expected output from this test function:

$ pytest -v test_squares.py
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-9.0.3, pluggy-1.6.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-regtest/checkouts/latest/.venv/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmpan6p154i
plugins: regtest-2.5.0, cov-7.1.0
collecting ... collected 1 item

test_squares.py::test_squares FAILED                                     [100%]

=================================== FAILURES ===================================
_________________________________ test_squares _________________________________

snapshot error(s) for test_squares.py::test_squares:

snapshot mismatch:
    > test_squares.py +6:
    > snapshot.check(result)
    shape mismatch: current shape: (4, 3)
                   recorded shape: (3, 3)
    --- current
    +++ expected
    row   0: -[1. 1. 2.]
             +[0. 1. 2.]
    row   1: -[4. 4. 5.]
             +[3. 4. 5.]
    row   2: -[7. 7. 8.]
             +[6. 7. 8.]
    row   3: -[10. 10. 11.]
---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 0
total number of failed snapshot tests  : 1
=========================== short test summary info ============================
FAILED test_squares.py::test_squares
============================== 1 failed in 0.12s ===============================

In case the change was intended, you can reset the test again:

$ pytest -v --regtest-reset test_squares.py
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-9.0.3, pluggy-1.6.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-regtest/checkouts/latest/.venv/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmpan6p154i
plugins: regtest-2.5.0, cov-7.1.0
collecting ... collected 1 item

test_squares.py::test_squares RESET                                      [100%]

---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 0
total number of failed snapshot tests  : 0
the following output files have been reset:
  _regtest_outputs/test_squares.test_squares__0
============================== 1 passed in 0.01s ===============================

Testing with numerical tolerances

The snapshot implementation for NumPy arrays also supports absolute and numerical tolerances using the optional arguments rtol and atol in the snapshot.check call. If not explicitly specified, both parameters are set to 0.0 which is often to prudent and must be adapted by the user:

# test_squares_with_tolerance.py

import numpy as np

def test_squares(snapshot):
    result = np.arange(9.0).reshape(3, 3)
    snapshot.check(result, atol=1e-8, rtol=1e-6)

We record this 3 x 3 matrix:

$ pytest -v --regtest-reset test_squares_with_tolerance.py
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-9.0.3, pluggy-1.6.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-regtest/checkouts/latest/.venv/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmpan6p154i
plugins: regtest-2.5.0, cov-7.1.0
collecting ... collected 1 item

test_squares_with_tolerance.py::test_squares RESET                       [100%]

---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 0
total number of failed snapshot tests  : 0
the following output files have been reset:
  _regtest_outputs/test_squares_with_tolerance.test_squares__0
============================== 1 passed in 0.01s ===============================

Now we change our data within the specified limits:

# test_squares_with_tolerance.py

import numpy as np

def test_squares(snapshot):
    result = np.arange(9.0).reshape(3, 3) + 1e-8
    snapshot.check(result, atol=1e-8, rtol=1e-6)

As we can see, the test still passes:

$ pytest -v test_squares_with_tolerance.py
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-9.0.3, pluggy-1.6.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-regtest/checkouts/latest/.venv/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmpan6p154i
plugins: regtest-2.5.0, cov-7.1.0
collecting ... collected 1 item

test_squares_with_tolerance.py::test_squares PASSED                      [100%]

---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 0
total number of failed snapshot tests  : 0
============================== 1 passed in 0.01s ===============================

Changing the formatting of numbers

pytest-regtest uses per default the output settings from NumPy. Adapting print options can lead to more readable diagnostic messages:

# test_squares_with_printoptions.py

import numpy as np

def test_squares(snapshot):
    result = np.arange(9.0).reshape(3, 3) - 2.0
    with np.printoptions(formatter=dict(float="{:+.3e}".format)):
        snapshot.check(result)

You can see that the numbers are not formatted with {+.3e} which forces scientific notation with 3 digits after the decimal point and always shows the sign of the numbers:

$ pytest -v test_squares_with_printoptions.py
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-9.0.3, pluggy-1.6.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-regtest/checkouts/latest/.venv/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmpan6p154i
plugins: regtest-2.5.0, cov-7.1.0
collecting ... collected 1 item

test_squares_with_printoptions.py::test_squares FAILED                   [100%]

=================================== FAILURES ===================================
_________________________________ test_squares _________________________________

snapshot error(s) for test_squares_with_printoptions.py::test_squares:

snapshot not recorded yet:
    > test_squares_with_printoptions.py +6
    > snapshot.check(result)
    [[-2.000e+00 -1.000e+00 +0.000e+00]
     [+1.000e+00 +2.000e+00 +3.000e+00]
     [+4.000e+00 +5.000e+00 +6.000e+00]]
---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 0
total number of failed snapshot tests  : 1
=========================== short test summary info ============================
FAILED test_squares_with_printoptions.py::test_squares
============================== 1 failed in 0.12s ===============================