History for Overview of test examples
changed:
-
<h2>1 Testing examples</h2>
<p>The example testing relies on an environment variable CMISS_EXAMPLES
which should indicate the top of the example tree.
i.e. currently setenv CMISS_EXAMPLES /hpc/cmiss/examples
Using such a variable means that there are no explicit paths inside
the actual example tree and the example tree or parts thereof can be
checked out on different machines and in different directories.
</p><p>Many parameters use regular expressions to allow you to select versions or programmers etc.
In each case the regular expression must match the entire word (it is wrapped in /^ $/ delimiters)
so that the word cmgui will match only cmgui and not cmgui_linux, cmgui_perl etc.
</p><h2>1.1 A single example</h2>
<p>This testing runs an example with any executable you choose and then
performs the appropriate diffs specified by the compare_version. All
the files remain in the directory where you run it and the test result
is written to the screen. Your directory should start empty.
</p><p>To test a single example run
<br>$CMISS_EXAMPLES/path_to_example/control/test -e path_to_example -s operating_system -x executable
</p><p>i.e. For example 146 a command could be
<br>$CMISS_EXAMPLES/1/14/146/control/test -e 1/14/146 -s irix -x /product/cmiss/cm/bin/cmo_n32
</p><p></p><table border="2" width="90%">
<caption><h3>$CMISS_EXAMPLES/path_to_example/control/test options</h3></caption>
<tbody><tr>
<td><b>-h</b></td><td>Writes usage.</td></tr>
<tr><td><b>-e</b></td><td>Relative path from example root to single example.</td></tr>
<tr><td><b>-s</b></td><td>Operating system, used to choose executables from $CMISS_EXAMPLES/commmon/cmiss_type1/${operating_system}_bin.</td></tr>
<tr><td><b>-x</b></td><td>The executable you want to test.</td></tr>
<tr><td><b>-c</b></td><td>Answers to compare with. (default is generic) This can be multiple versions using an &.</td></tr>
<tr><td><b>-k</b></td><td>Keep everyfile generated while testing.</td></tr>
<tr><td><b>-o</b></td><td>Where you want the result files copied too.
Normally this is only used by multiple versions testing. Also
suppresses writing of answers to the screen.</td></tr>
</tbody></table>
<h3>More examples</h3>
<p>$CMISS_EXAMPLES/1/14/146/control/test -e 1/14/146 -s linux -x
/product/cmiss/cm/bin/cm_linux -c cm_linux&generic
<br>Runs a linux executable and compares answers with those found in
cm_linux and in generic. The cm_linux answers have priority as that
name is specified first. Again your directory should start empty.
</p><h2>1.2 Running normal testing</h2>
<p>This testing runs testing on a part of the tree. The executables and
versions tested are controlled by the versions file. The results are
kept in the output directories and viewable in html. This is what the
overnight tests run.
</p><p>If you are running with the default example tree
(CMISS_EXAMPLES is /product/cmiss/examples) and you aren't the cmiss
user then you won't be able to copy your results into the tree. Either
run by overriding the executable with -x as shown in the More Examples
section just below or work on your own copy of the example tree checked
out with CVS.
</p><p>To test examples run
<br>$CMISS_EXAMPLES/common/cmiss_type1/test_tree.pl -p program
</p><p>i.e. To run all cm tests
<br>$CMISS_EXAMPLES/common/cmiss_type1/test_tree.pl -p cm
</p><p></p><table border="2" width="90%">
<caption><h3>$CMISS_EXAMPLES/common/cmiss_type1/test_tree.pl options</h3></caption>
<tbody><tr><td><b>-h</b></td><td>Writes usage.</td></tr>
<tr><td><b>-e</b></td><td>Relative path from example root below which test examples will be run.</td></tr>
<tr><td><b>-s</b></td><td>Operating system, used to choose executables
from $CMISS_EXAMPLES/commmon/cmiss_type1/${operating_system}_bin. This
option is normally automatically detected by the script.</td></tr>
<tr><td><b>-p</b></td><td>Regexp for the program you want to test. i.e. cm, cmgui, "cm.*" or filt.</td></tr>
<tr><td><b>-v</b></td><td>Regexp for the versions you want to test. i.e. cm_n32, cm_linux or "cm.*".</td></tr>
<tr><td><b>-x</b></td><td>The executable you want to use. Overrides the one specified in the versions file.</td></tr>
<tr><td><b>-c</b></td><td>Answers to compare with. (default is specified in versions file for each version).</td></tr>
<tr><td><b>--check-all-broken-versions</b></td><td>Check all the broken versions of each example that are valid for the OS and program as well as those selected with -v.</td></tr>
<tr><td><b>-k</b></td><td>Keep everyfile generated while testing.</td></tr>
<tr><td><b>-z</b></td><td>Maximum number of processes to use simultaneously. Examples do not run in order when executed this way.</td></tr>
<tr><td><b>-o</b></td><td>The output version name used for storing the output in the example tree.</td></tr>
</tbody></table>
<p>Test output produced by the executables is usually stored as differences.
The unix command `patch' is useful for recovering the full output from the
correct answers in the `testing/diff' directory and the differences in the
`output' directory.
</p><h3>More examples</h3>
<p>$CMISS_EXAMPLES/common/cmiss_type1/test_tree.pl -p cm -e 1/11
<br>Tests all the cm examples below 1/11 in the tree.
</p><p>$CMISS_EXAMPLES/common/cmiss_type1/test_tree.pl -p "(cm|cmgui)" -e 1/11
<br>Runs all the cm and cmgui examples below 1/11 in the tree.
</p><p>$CMISS_EXAMPLES/common/cmiss_type1/test_tree.pl -p cm -v cmo_n32 -x ~/cm/mycmo_n32
<br>Runs all the cmo_n32 tests using my cmo_n32 executable. Whenever
you override the executable or the compare_version without specifying
an output_version the output is kept in a cmiss_output directory
created where the command is run. A directory structure mimicing the
actual example tree is created but only with output directories. A
summary of the testing is printed to the screen when the tests
complete.
</p><p>$CMISS_EXAMPLES/common/cmiss_type1/test_tree.pl -p cm -e 1/11 -z 2
<br>Tests all the cm examples below 1/11 in the tree using two processes.
</p><p>$CMISS_EXAMPLES/common/cmiss_type1/test_tree.pl -p cm -v cmo_n32 -x ~/cm/mycmo_n32 -o freds_cmo
<br>Runs all the cmo_n32 tests using my cmo_n32 executable. If you have
privelege to do so the results will be copied into the main example
tree output as "freds_cmo" version and will be included in both the
html generated and the mail sent to everyone.
<!---------------------------------------------------------------------------->
</p><h2>2 Generating HTML and Mail</h2>
<p>The HTML and Mail generated are produced based on the contents of
the output directories in each example. Only tests that copied their
results into the output directories will be shown or reported on.
</p><p>You can do these things while the testing is in progress to see
how it is getting along. Examples that haven't been tested yet will be
listed as pending.
</p><h2>2.1 Generating HTML</h2>
<p>To update the web pages, which is done automatically each night run
<br>$CMISS_EXAMPLES/common/cmiss_type1/html_tree.pl
</p><p>Only the cmiss user should be updating the main tree in
/product/cmiss/examples but any cmiss programmer can work on their copy
of the tree.
</p><p></p><table border="2" width="90%">
<caption><h3>$CMISS_EXAMPLES/common/cmiss_type1/html_tree.pl options</h3></caption>
<tbody><tr><td><b>-h</b></td><td>Writes usage.</td></tr>
<tr><td><b>-e</b></td><td>Relative path from example root below which html will be updated.</td></tr>
</tbody></table>
<h3>More examples</h3>
<p>$CMISS_EXAMPLES/common/cmiss_type1/html_tree.pl -e 1
<br>Updates the html below 1 in the tree.
</p><h2>2.2 Generating Summary Information</h2>
<p>You can generate summaries of your own tree and output these to the
screen or a file so you can look at it in a browser. Email will only be
sent out if you specify -p or -n.
</p><p>To generate a summary and write that to the console simply run mail_tree.
<br>$CMISS_EXAMPLES/common/cmiss_type1/mail_tree.pl
</p><p>To create a summary html file run, then view it with a browser.
You will probably want to update the web pages to any examples you are
interested in as in section 2.1
<br>$CMISS_EXAMPLES/common/cmiss_type1/mail_tree.pl -f /tmp/filename.html
</p><p>To send mail all programmers in the notify lists, which also is done automatically each night run
<br>$CMISS_EXAMPLES/common/cmiss_type1/mail_tree.pl -p '.*'
</p><p>To send a summary mail message to yourself run
<br>$CMISS_EXAMPLES/common/cmiss_type1/mail_tree.pl -n your_email_address
</p><p></p><table border="2" width="90%">
<caption><h3>$CMISS_EXAMPLES/common/cmiss_type1/mail_tree.pl options</h3></caption>
<tbody><tr><td><b>-h</b></td><td>Writes usage.</td></tr>
<tr><td><b>-e</b></td><td>Relative path from example root below which examples will be included in the mail.</td></tr>
<tr><td><b>-p</b></td><td>Regexp to match programmers to whom mail is sent.</td></tr>
<tr><td><b>-n</b></td><td>Email address which will be notified for every
example searched by this mail. This is a space separated list (use quotes if
specifying more than one address. These email addresses are not filtered by the -p regexp.</td></tr>
<tr><td><b>-f</b></td><td>This specifies a filename to which will be written the mail for every example in the -e path.</td></tr>
<tr><td><b>-o</b></td><td>This specifies a directory which contains the output
from testing the examples in the -e path. The default is to use the normal
example tree (${CMISS_EXAMPLES}). This option can be useful if you have run
your own example testing and want to generate a summary of the testing
results output but not copied to the default example tree.</td></tr>
<tr><td><b>-F</b></td><td>Include the Fortran check errors in the summary
information - possibly only useful when generating the daily programmer's
email?</td></tr>
</tbody></table>
<h3>More examples</h3>
<p>$CMISS_EXAMPLES/common/cmiss_type1/mail_tree.pl -e 1 -p "s.blackett@auckland.ac.nz"
<br>Sends mail for examples below 1 in the tree but only to
"s.blackett@auckland.ac.nz". This will only send mail about examples
"s.blackett" is usually notified.
</p><p>$CMISS_EXAMPLES/common/cmiss_type1/mail_tree.pl -e 1 -n "s.blackett@auckland.ac.nz"
<br>Sends mail for examples below 1 in the tree but only to
"s.blackett@auckland.ac.nz". This will send mail about every example in
the 1 tree, whether "s.blackett" is normally notified about it or not.
</p><p>$CMISS_EXAMPLES/common/cmiss_type1/mail_tree.pl -f notify_all.html
<br>Generates a file called notify_all.html which contains the mail output for all the examples.
</p><p>$CMISS_EXAMPLES/common/cmiss_type1/mail_tree.pl -e andre/tests -o
/tmp/andre/cmiss_output -n "andre@bob.com" -f andres_test_summary.html
<br>If you have run the test_tree.pl script above on your own subset of test examples
(the andre/tests branch of your local example tree) and had the output
generated into /tmp/andre/cmiss_output, this command will summarise the
testing results and generate a file called andres_test_summary.html and send
the summary the andre@bob.com via email.
<!---------------------------------------------------------------------------->
</p><h2>3 Your own Example Tree</h2>
<p>When working on the examples it is very useful to have your own copy
of part of the example tree. To do this you need to be a cmiss
programmer. Having your own copy allows you to commit changes only when
they are right, to try all the testing versions and even to view the
html generated pages for your own testing without affecting anyone
else, great for testing changes to cmiss before you commit them. The
main consideration is disk space as the full example tree is nearly
about 900Mb, so use some temporary space, not on /usr/people and only
check out the parts you need.
</p><p>To get your tree working with all the command in your document
you first need the common directory. Check your CVSROOT is correct. For
the esu* system we use /product/cmiss/cvs. Change to your chosen place
for your examples, say /var/tmp/shanes_examples/ then do
<br>cvs checkout examples/common
</p><p>Now checkout the part of the tree you want, say you are working
on the examples below 2/21/219 then you can just checkout this part of
the tree.
<br>cvs checkout examples/2/21/219
</p><p>Finally set you CMISS_EXAMPLES variable to point to this tree, if I have my files in /var/tmp/shanes_examples/ then I do
<br>setenv CMISS_EXAMPLES /var/tmp/shanes_examples/examples
</p><p>Now you can do any of the commands on this page on your own copy
of the tree. If you generate html as in Section 2.1 then these won't be
visible on the web but you can just view the files directly, to look at
the 2/21/219 directory I would just enter into Netscape
file:/var/tmp/shanes_examples/examples/2/21/219/index.html
</p><p>If you run tests as in section 1.2 on your own tree you can
generate html or summary information as described in section 2.2 even
while the tests are still running.
</p><p>If you make changes to your tree you want to put into the main tree then commit these changes through <a href="/cm/wiki/IntroductionToCVS">cvs</a>.
</p><h2>4 Creating examples</h2>
<p>So how do I make an example?
</p><p>Personally I copy an example that is closest to the one I am
adding, remove the CVS directories from all the subdirectories of the
copy, update all the files and then add it into the example tree. Only
commit the web_data, cmiss_input and testing parts of the example
though, as the index.html and output directories are generated each
night and so should not be in the source CVS.
</p><h2>4.1 Easy example creation</h2>
<p>To make a new example you should checkout the part of the tree you are
placing it in as in Section 3. You may like to use the -l option on checkout
to avoid checking out subdirectories. Do NOT forget to check out examples/common and to have $CMISS_EXAMPLES set to your own examples directory for this work.
</p><p>$CMISS_EXAMPLES/common/cmiss_type1/make_example.pl -e 1/5r/5rg
<br>Creates a "blank" example at 1/5r/5rg
</p><p>If you want to use the example for cm testing specify the option -t
</p><p>$CMISS_EXAMPLES/common/cmiss_type1/make_example.pl -e 1/5r/5rg -t
<br>Creates a "blank" cm testing example at 1/5r/5rg
</p><p>Now..
</p><ul><li>Edit web_data/name.txt</li><li>Edit web_data/description.html</li><li>Edit cmiss_input/example_*.com</li><li>Put any other files required into cmiss_input directory</li></ul>
<p>And for testing examples...
</p><ul><li>Edit cmiss_input/test_output.com.</li><li>Edit or replace testing/ndiff/generic/cmiss_test.out with the
expected output from test_output.com.</li><li>Add any other required output files into testing/ndiff/generic. The
ndiff directory is for answers that contain real numbers and will be
compared with test output by ndiff. If the output does not contain any
real numbers you can instead put output files in testing/diff/generic
for a literal comparison.</li><li>If you wish to specify any special options for ndiff or diff then
add an options file for each special output file. The names of the
options files should be the output file name appended with ".opt". For
example "cmiss_test.out.opt" specifies options for the diff performed
with "cmiss_test.out". An options file in the "generic" directory is
applied to all versions unless an options file exists in the specific
version directory. Each line of an options file is passed as an
argument to the diff program before the filename arguments.<br>
Some useful options for ndiff are mentioned below under Numerical
Difference Testing.
</li></ul>
<p>Finally commit this all into CVS.
</p><ul><li>In the main directory of your new example run 'cvs update'</li><li>All your new files that were automatically added should have an A beside them.</li><li>Most other files you have added will have ? for unknown. Add them to the repository by 'cvs add new_filename'.</li><li>Now
commit all this to the repository. In the main directory of your new
example run 'cvs commit -m "Added an example to test blah blah"'</li></ul>
<h2>4.2 Editing examples</h2>
<p>Just checkout the examples that you wish to edit, make your changes, test
the changes, and then check these into cvs. Running 'cvs update' always
ensures you are up to date with the repository but does not eliminate any
changes you have made. The symbols by each filename indicate its status in
your directory. A means the file is scheduled for addition, R means the file
is scheduled for removal, M means that you have made some modifications that
aren't checked in, ? means that the file is unknown by CVS, U means that your
file has been updated from the repository and C means your changes conflict
with those that someone else has made (Get help with conflicts unless you know
what to do).
</p><ul><li>To add a directory
<br>1. Simply add it to cvs 'cvs add dirname'</li><li>To add a file
<br>1. Add the file to cvs
<br>2. Run 'cvs update'
<br>3. Commit your changes with 'cvs commit -m "Added another file" filename'</li><li>To remove a file
<br>1. Remove the file using 'rm filename'
<br>2. Tell cvs that you are going to remove the file with 'cvs removie filename'
<br>3. Run 'cvs update'
<br>4. Commit your changes with 'cvs commit -m "File removed as superseded by another file" filename'</li><li>Checking in changes
<br>1. 'Run cvs update'
<br>2. Commit your changes with 'cvs commit -m "Added streamlines" filename'</li></ul>
<h2>4.3 Extending examples</h2>
<!-- <p>To add a test version. -->
<!-- <p>To add another diffing program. -->
<h2>4.4 Complicated example creation</h2>
<p>So the existing struture doesn't do what you want.
</p><p>By changing the control directory you can make an example do whatever you want.
</p><ul><li>control/html: Executed to generate the index.html</li><li>control/test:
Used to generate all output in output directory. The default
implementation control multiprocessing, any number of diff functions
and multiple versions and operating systems</li><li>control/test_run:
Run by the default test program to actually execute the example. This
is something that you may want to customise. The flags in the end of a
version file get passed here.</li></ul>
<h2>4.5 Numerical Difference Testing</h2>
<p> When testing an example, the generated answers are tested against a
correct or fixed set of answers contained within the "testing" directory. If
the answers are contained below the "diff" directory then they must be exactly
the same, while if they are contained within a "ndiff" directory the answers
are considered the same if they are within a specified numerical
tolerance.</p>
<p> One of the options that ndiff can take is a tolerance (specified by
--tolerance=number[%], ie a number followed by an optional % sign, used to
specify a relative tolerance) which can be used to tolerate any slight
numerical differences. Currently the tolerance has a default level of 0.0, so
any differences that are not simply variations in format (e.g. Fortran's D
format) will be caught.</p>
<p> The tolerance of the numerical differences and regular expressions for
lines on which differences should be ignored can be specified through the
creation of appropriate "*.opt" files (see example creation above). An
example of such a file is shown below:
</p><pre>--tolerance=1e-10<br>--ignore-matching-lines=^\ Condition\ number<br>--match-regexp=^\ Time<br></pre>
<p> If the tolerance is not zero, a change in the last digit is accepted
because, even though this change may be larger than the tolerance, it may
represent a difference that is smaller. If more accuracy is required in the
output then more significant figures should be specified in the expected
output.</p>
<p>
--ignore-matching-lines is a regexp diff option which excludes matching lines.
--match-regexp is an ndiff option which will exclude matching lines and not pass them on
to the diff program.
</p>
<!---------------------------------------------------------------------------->
<h2>5 Example Testing on hpc1</h2>
<p>
You might be wondering how to create an example that is run on hpc so you can
test whether your parallel program is giving you the right answers ?
</p>
<p>
Overnight on hpc a similar testing tree is run but it will only run the
cm program with the irix64 operating system. These two
things are specified as the -p and -s switches for the
test_tree.pl program. Thus the test_tree.pl program requires
that a versions file, (= /path/to/example/control/versions) keeps
a corresponding entry for the operating system (irix64) and program name (cm)
in it. This is done by adding line(s) in the versions file like:
</p><pre>irix64 cm cm_n32_mp_2 $CMISS_ROOT/cm/bin/cm_n32_mp generic -p 2<br>irix64 cm cm_n32_mp_4 $CMISS_ROOT/cm/bin/cm_n32_mp generic -p 4<br>irix64 cm cm_n32_mp_8 $CMISS_ROOT/cm/bin/cm_n32_mp generic -p 8<br></pre>
Thus the cm_n32_mp executable will be run on this example each night. It will have
3 different versions, one for 2,4 and 8 threads. The -p n flag will specify the number
of cm threads to be used for the cm_n32_mp executable.
<p>
So how do I add this line ? You simply check out the versions file using cvs
and then add this line to it. You can run different executables by
altering/adding these lines to the versions file.
Examples are updated on hpc via cvs each night,
and then the output directory is copied back to esu1 (via secure copy, scp)
once examples have finished on hpc1.
</p>
Forgot your password?
New user?