Update the Reference Data

A special kind of tests are executed only when pytest is called alongside the --refdata flag. These tests compares the output of the TARDIS code (mostly arrays) against the information stored in the reference data files.

TARDIS stores reference data in the tardis-refdata repository. This repository also has a mirror hosted in Azure Pipelines (synchronized automatically by a GitHub workflow) since this Microsoft service does not have limitations in bandwidth nor storage.

Sometimes, this data needs to be updated. The procedure to update these files manually is not trivial and has been automated recently thanks to the NumFOCUS support.

Default Procedure

Imagine you are working on a new feature (or fix) for TARDIS, you have opened a pull request and the reference data tests are failing in the testing pipeline. This could happen for many reasons:

  1. There’s a problem in your code.

  2. Your code is OK, but the reference data is outdated.

  3. The pipeline is broken.

If you think your could be dealing with scenario B, then:

  1. Write /azp run compare-refdata in a comment on your PR.

  2. Analyze the results and discuss if the reference data effectively requires an update.

  3. Update the reference data by writing /azp run update-refdata on a new comment.

Note

  • If you don’t have enough privileges to run the pipelines, tag a TARDIS developer capable of doing so.

  • If any of these two pipelines fail, please tag a TARDIS team member responsible for CI/CD.

If everything went well, the reference data will have been updated by the TARDIS bot and the commit message should include the pull request number that triggered the update.

Manual Procedure

The manual procedure is documented for debugging purposes and should not be used in general.

  1. Activate the tardis environment.

  2. Fork and clone the tardis-refdata repository.

  3. Follow the instructions at the top of the notebook tardis-refdata/notebooks/ref_data_compare.ipynb.

  4. Go to your local tardis repository and make sure you are working on the branch you want to generate new reference data from.

  5. Generate new reference data with pytest tardis --refdata=/path/to/tardis-refdata --generate-reference.

  6. Run the ref_data_compare.ipynb notebook and check the results.

  7. Make a new branch in tardis-refdata, push your new reference data and open a pull request.