ptc-catalog

Adopt Automated Correctness Testing in a Legacy Software Project

Download this PTC (Markdown)

Target

Adoption of automated testing for a legacy software project that has no automated tests at the beginning (e.g., where testing is done manually or only tested against customer codes).

User Story

As a developer on a legacy software project/product, I want my team to adopt increasingly more rigorous and effective automated correctness testing, so that development will be easier, users will see less defects, releases will go out faster, and the sustainability of my project will be improved.

Card

Stage Description Added Value
0 One or more automated simple system-level tests that simply run the code for a few different inputs and look for 0 return code (i.e., does not crash) are created Major breakages in the code causing segfaults and other crashes will be caught when code is changed or added.
1 Comparison of the code outputs to previous outputs are used to create system-level no-change tests. Changes in the behavior of the code will be caught when code is changed or added. (But such tests can be very fragile.)
2 System-level verification tests that don’t just compare to previous output but instead look for physical or mathematical properties of the underlying equations or algorithms are added. Good changes (like solving the equations to a higher tolerance faster) will not cause the tests to fail and these tests will be more robust compared to no-change tests.
3 Developers use the Legacy Software Change Algorithm to make changes to existing code which tests larger integrated components creating integration-level no-change/characterization tests. Changes to the code can be made faster and with less risk compared to only running system-level tests.
4 Developers use the Legacy Software Change Algorithm to make changes to existing code for for smaller units creating unit-level non-change/characterization tests in addition to integration-level tests where appropriate. Changes to the code can be made faster and problems will be easier to debug when compared to only writing tests at the courser-grained integration level.
5 Developers add automated unit-level and integration-level verification tests for new code as it is written (or before it is written using Test Driven Development). New code will have automated finer-grained tests which will make it easier and safer to extend that code in the future (and new code will no longer be “Legacy Code”).

Comments

Acknowledgement

This Project Tracking Card originated from the PSIP PTC Catalog. The development of the PSIP PTC Catalog was supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the U.S. Department of Energy Office of Science and the National Nuclear Security Administration.