How to Find Out Which Source Changes Broke an Example

The process of finding the source changes that caused a test example to fail usually starts with two steps: determining when the test failed and finding the source changes that where made at that time.

It is also worth checking Summary Notes on Failed cm Tests to see if someone has already starting trying to determine the cause the problem. Please update that page if you find out any information and are unable to fix the problem immediately.

Determining When a Test Failed

To narrow down the cause of the problem, bounds can be found for the time interval during which the problem occurred. Useful information for this is available in the "Testing status by version" section of the example web page.

The "last success" time is the most useful piece of information as it gives a definite lower bound on the time for the change that caused the failure. If there is no "last success" time reported for a version then that test version has never passed.

History (log) Files

More detailed information can be obtained from history files, which can be accessed by following the links attached to each version name.

These pages include a line for each time the version has been tested. The "break" status indicates that the output has changed and that it is not the expected output. "broken" indicates that the output is still the same as the previous break.

Times are recorded in universal time.

Each interval from each "break" back to the last "success" or previous "broken" status is a possible interval during which a source change contributed to the failure of the test. (If there is more than one "break" since the last "success", then there is more than one possible interval. It is usually worth checking the earlier intervals first as the latter "break"s may have been fixed.)

Listing the Source Changes in a Specific Time Interval

The 'cmisschanges.pl' script (in 'cmiss_utils/scripts' but there is a link under '$CMISS_ROOT/bin') is useful here. It is worth widening the time interval of interest a little at the lower end to take into account time to compile cm (and it is possible that the cm executable wasn't even updated that day due to compilation failure).

Basic use of the script is something like:

cmisschanges.pl -after 2003-11-03 -before 2003-11-04,12:54 -tree .

If this produces too much output, then you may like to change the compulsory '-tree' `option' value to be something more specific like:

'examples/5/55/552/5521|cm(/|$)'

But be aware that changes to libraries such as linear_solvers may also affect cm output.

Determining the Maintainer of an Example

Running 'cvs log' on the cm command input file for the example gives a good indication of who created the example or has made significant changes to it.

e.g. the following command can be run from '$CMISS_ROOT':

cvs log examples/cellml/JRW_1998/cmiss_input/JRW_1998.com