Regression Test

Many of the Python projects in the SciComp group use py.test for their testing needs. The codespeak py library, which contains py.test, can be installed by typing

easy_install -U py

Once you're ready to test, just type

python `which py.test`

This works in individual package directories as well as directories containing multiple projects.

Speeding Up the Test Run

py.test supports running tests in parallel. This appears to work well. On a quad-core machine, use the option -n 4 to achieve 4-way parallelism.

Writing your own tests

Any function named test_xxx or class named TestXxx in a file named test_xxx.py will automatically be discovered and run by py.test, regardless of where it lives in the source tree. The "xxx" represents any string of characters.

Conditions in tests should be verified using assert condition.

Email Notification of Regression Failures

Subscribe to the mailing list here.

Avoiding long-running tests

Similar to the mpi keyword above, there's also a long keyword. Using the option -k -long, you may avoid waiting for long-running tests if that isn't desired. (Note that long-running tests are no less important!)

When you're writing tests and wondering whether a given test should be marked "long": The guideline should be that anything longer than about 10 seconds is "long".

NOTE: See this issue.

Marking tests with keywords

Marking tests with keywords works like this:

   1 import py.test
   2 @py.test.mark.long
   3 def test_something():
   4     ...

This obviously will spoil the party if py.test is not available. If you can stomach a dependency on pytools, then you may do this instead:

   1 from pytools.test import mark_test
   2 @mark_test.long
   3 def test_something():
   4     ...

This will boil down to a no-op if py.test isn't available.

(You need py.test version 1.0 or newer for this to work--the syntax was different in prior versions.)

Bugs, Issues and Workarounds

MPI Tests

MPI tests are a bit special. It turns out that OpenMPI goes into a hang if you fork an mpirun from a process that is already an initialized MPI process. Unfortunately, this is easy to do: Any process that imports boostmpi is an initialized MPI process. And hedge imports MPI automatically when guessing a run context. This makes it necessary to run the tests in two batches: One *with* MPI and one without, where care is taken that the MPI test enumerator process never initializes MPI. This is accomplished as follows:

python `which py.test` -k mpi
python `which py.test` -k -mpi

Update: It appears this is fixed in OpenMPI 1.3 and newer.

py.test crashes if you enumerate modules with builtin functions

Bug Report