Testing Guidelines

Writing Unit Tests

Unit tests are a type of software testing where individual units or components of a software are tested in isolation to ensure that they are working as expected. Writing unit tests is an important aspect of code quality as it helps to identify and fix bugs early in the development process, ensures that new changes do not break existing functionality, and provides documentation for how the code is supposed to work.

For TARDIS, we use pytest as our testing framework. If you are adding a new feature or fixing a bug, it is important to write unit tests for your code to ensure that it works correctly and does not break existing functionality.

Unit tests should test meaningful functionality of the code, and not just test trivial things like whether a function runs without throwing an error. For example, if you are writing a function that adds two numbers, a unit test should check that the function returns the correct sum for different pairs of numbers. Use pytest parametrize to test multiple input/output pairs where individual or small input/output pairs are sufficient to test the functionality of the code.

Example:

@pytest.mark.parametrize(
["compton_opacity", "photoabsorption_opacity", "total_opacity", "expected"],
[
    (1, 0, 1, GXPacketStatus.COMPTON_SCATTER),
    (0, 1, 1, GXPacketStatus.PHOTOABSORPTION),
    (0, 0, 1, GXPacketStatus.PAIR_CREATION),
],
)
def test_scatter_type(
    compton_opacity, photoabsorption_opacity, total_opacity, expected
):
    """Test the scattering type

    Parameters
    ----------
    compton_opacity : float
    photoabsorption_opacity : float
    total_opacity : float
    expected : list
        Expected parameters
    """
    actual = scatter_type(compton_opacity, photoabsorption_opacity, total_opacity)
    assert actual == expected

Tests should make use of pytest fixtures where possible. For example, if you are testing a function that requires a certain input data structure, you can create a fixture that sets up that data structure and use it in your tests. Use existing fixtures where possible to reduce code duplication and make the tests faster to run. Search for fixtures that are already defined using your IDE.

Example:

@pytest.fixture(scope="session")
def simulation_verysimple_vpacket_tracking(config_verysimple, atomic_dataset):
    atomic_data = deepcopy(atomic_dataset)
    sim = Simulation.from_config(
        config_verysimple, atom_data=atomic_data, virtual_packet_logging=True
    )
    sim.last_no_of_packets = 4000
    sim.run_final()
    return sim

Using Regression Data

When writing tests, it is often helpful to include regression data tests to make sure that the output of that piece of code remains consistent. Regression data is a set of output values that are generated by the code at some point in time, which can then be checked against in the future.

TARDIS has a regression data repository that contains a variety of output data for different features of the code. Before merging a change to the code, we validate the changed code against these pre-written values.

We use a regression data class defined in tardisbase to access the regression data in our unit tests. This class provides methods to load the regression data from numpy arrays or pandas dataframes stored as HDF files.

Note

Try to reuse existing regression data where possible so that we minimize the amount of regression data storage required, and reduce duplication of data and thus confusion.

Example:

def test_transition_probabilities_regression(
    self, continuum_macro_atom_state, regression_data
):
    """Test transition probabilities using regression data.

    This test stores the transition probabilities in regression data
    for comparison across runs.
    """
    actual = continuum_macro_atom_state.transition_probabilities
    expected = regression_data.sync_dataframe(actual)
    pdt.assert_frame_equal(actual, expected)

Note that regression_data is loaded as a fixture. The sync_dataframe method is primarily used to load pre-existing regression data that the test can check against, such as with the pdt.assert_frame_equal line that immediately follows.

The sync_dataframe method allows the regression data to be updated or read depending on the --generate-reference flag that we have added to pytest. pytest tardis --tardis-regression-data=/path/to/tardis-regression-data --generate-reference will run all the tests in TARDIS, and every time that a sync method appears in a test, it will update the regression data with the actual dataframe instead of reading it. Then, pytest tardis --tardis-regression-data=/path/to/tardis-regression-data will read the expected dataframe from the regression data for comparison. For more information on how to update the regression data, please refer to the Update the Regression Data section of the documentation.

Regression data files belong in the dedicated tardis-regression-data repository. Do not commit regression data to the main TARDIS repository.

For guidance on when to update regression data and how those changes are reviewed, see TARDIS Development Playbook.