Overview of test examples

<h2>1 Testing examples</h2>

<p>The example testing relies on an environment variable CMISS_EXAMPLES which should indicate the top of the example tree. i.e. currently setenv CMISS_EXAMPLES /hpc/cmiss/examples Using such a variable means that there are no explicit paths inside the actual example tree and the example tree or parts thereof can be checked out on different machines and in different directories.

</p><p>Many parameters use regular expressions to allow you to select versions or programmers etc. In each case the regular expression must match the entire word (it is wrapped in /^ $/ delimiters) so that the word cmgui will match only cmgui and not cmgui_linux, cmgui_perl etc.

</p><h2>1.1 A single example</h2>

<p>This testing runs an example with any executable you choose and then performs the appropriate diffs specified by the compare_version. All the files remain in the directory where you run it and the test result is written to the screen. Your directory should start empty. </p><p>To test a single example run <br>$CMISS_EXAMPLES/path_to_example/control/test -e path_to_example -s operating_system -x executable

</p><p>i.e. For example 146 a command could be <br>$CMISS_EXAMPLES/1/14/146/control/test -e 1/14/146 -s irix -x /product/cmiss/cm/bin/cmo_n32

</p><p></p><table border="2" width="90%"> <caption><h3>$CMISS_EXAMPLES/path_to_example/control/test options</h3></caption> <tbody><tr>

<td><b>-h</b></td><td>Writes usage.</td></tr> <tr><td><b>-e</b></td><td>Relative path from example root to single example.</td></tr> <tr><td><b>-s</b></td><td>Operating system, used to choose executables from $CMISS_EXAMPLES/commmon/cmiss_type1/${operating_system}_bin.</td></tr> <tr><td><b>-x</b></td><td>The executable you want to test.</td></tr> <tr><td><b>-c</b></td><td>Answers to compare with. (default is generic) This can be multiple versions using an &.</td></tr> <tr><td><b>-k</b></td><td>Keep everyfile generated while testing.</td></tr> <tr><td><b>-o</b></td><td>Where you want the result files copied too. Normally this is only used by multiple versions testing. Also suppresses writing of answers to the screen.</td></tr> </tbody></table>

<h3>More examples</h3> <p>$CMISS_EXAMPLES/1/14/146/control/test -e 1/14/146 -s linux -x /product/cmiss/cm/bin/cm_linux -c cm_linux&generic <br>Runs a linux executable and compares answers with those found in cm_linux and in generic. The cm_linux answers have priority as that name is specified first. Again your directory should start empty. </p><h2>1.2 Running normal testing</h2>

<p>This testing runs testing on a part of the tree. The executables and versions tested are controlled by the versions file. The results are kept in the output directories and viewable in html. This is what the overnight tests run. </p><p>If you are running with the default example tree (CMISS_EXAMPLES is /product/cmiss/examples) and you aren't the cmiss user then you won't be able to copy your results into the tree. Either run by overriding the executable with -x as shown in the More Examples section just below or work on your own copy of the example tree checked out with CVS. </p><p>To test examples run <br>$CMISS_EXAMPLES/common/cmiss_type1/test_tree.pl -p program

</p><p>i.e. To run all cm tests <br>$CMISS_EXAMPLES/common/cmiss_type1/test_tree.pl -p cm

</p><p></p><table border="2" width="90%"> <caption><h3>$CMISS_EXAMPLES/common/cmiss_type1/test_tree.pl options</h3></caption> <tbody><tr><td><b>-h</b></td><td>Writes usage.</td></tr> <tr><td><b>-e</b></td><td>Relative path from example root below which test examples will be run.</td></tr> <tr><td><b>-s</b></td><td>Operating system, used to choose executables from $CMISS_EXAMPLES/commmon/cmiss_type1/${operating_system}_bin. This option is normally automatically detected by the script.</td></tr> <tr><td><b>-p</b></td><td>Regexp for the program you want to test. i.e. cm, cmgui, "cm.*" or filt.</td></tr> <tr><td><b>-v</b></td><td>Regexp for the versions you want to test. i.e. cm_n32, cm_linux or "cm.*".</td></tr> <tr><td><b>-x</b></td><td>The executable you want to use. Overrides the one specified in the versions file.</td></tr> <tr><td><b>-c</b></td><td>Answers to compare with. (default is specified in versions file for each version).</td></tr> <tr><td><b>--check-all-broken-versions</b></td><td>Check all the broken versions of each example that are valid for the OS and program as well as those selected with -v.</td></tr> <tr><td><b>-k</b></td><td>Keep everyfile generated while testing.</td></tr> <tr><td><b>-z</b></td><td>Maximum number of processes to use simultaneously. Examples do not run in order when executed this way.</td></tr> <tr><td><b>-o</b></td><td>The output version name used for storing the output in the example tree.</td></tr> </tbody></table>

<p>Test output produced by the executables is usually stored as differences. The unix command `patch' is useful for recovering the full output from the correct answers in the `testing/diff' directory and the differences in the `output' directory.

</p><h3>More examples</h3> <p>$CMISS_EXAMPLES/common/cmiss_type1/test_tree.pl -p cm -e 1/11 <br>Tests all the cm examples below 1/11 in the tree. </p><p>$CMISS_EXAMPLES/common/cmiss_type1/test_tree.pl -p "(cm|cmgui)" -e 1/11 <br>Runs all the cm and cmgui examples below 1/11 in the tree. </p><p>$CMISS_EXAMPLES/common/cmiss_type1/test_tree.pl -p cm -v cmo_n32 -x ~/cm/mycmo_n32 <br>Runs all the cmo_n32 tests using my cmo_n32 executable. Whenever you override the executable or the compare_version without specifying an output_version the output is kept in a cmiss_output directory created where the command is run. A directory structure mimicing the actual example tree is created but only with output directories. A summary of the testing is printed to the screen when the tests complete. </p><p>$CMISS_EXAMPLES/common/cmiss_type1/test_tree.pl -p cm -e 1/11 -z 2 <br>Tests all the cm examples below 1/11 in the tree using two processes. </p><p>$CMISS_EXAMPLES/common/cmiss_type1/test_tree.pl -p cm -v cmo_n32 -x ~/cm/mycmo_n32 -o freds_cmo <br>Runs all the cmo_n32 tests using my cmo_n32 executable. If you have privelege to do so the results will be copied into the main example tree output as "freds_cmo" version and will be included in both the html generated and the mail sent to everyone.  </p><h2>2 Generating HTML and Mail</h2> <p>The HTML and Mail generated are produced based on the contents of the output directories in each example. Only tests that copied their results into the output directories will be shown or reported on. </p><p>You can do these things while the testing is in progress to see how it is getting along. Examples that haven't been tested yet will be listed as pending. </p><h2>2.1 Generating HTML</h2> <p>To update the web pages, which is done automatically each night run <br>$CMISS_EXAMPLES/common/cmiss_type1/html_tree.pl </p><p>To update the web page which displays large thumbnails for front end graphics examples, which is also done nightly, run the following<br>$CMISS_EXAMPLES/common/cmiss_type1/generate_index_thumbs.pl</p><p>Only the cmiss user should be updating the main tree in /hpc/cmiss/examples but any cmiss programmer can work on their copy of the tree.</p> <p></p><table border="2" width="90%"> <caption><h3>$CMISS_EXAMPLES/common/cmiss_type1/html_tree.pl options</h3></caption> <tbody><tr><td><b>-h</b></td><td>Writes usage.</td></tr> <tr><td><b>-e</b></td><td>Relative path from example root below which html will be updated.</td></tr> </tbody></table> <h3>More examples</h3> <p>$CMISS_EXAMPLES/common/cmiss_type1/html_tree.pl -e 1 <br>Updates the html below 1 in the tree.

</p><h2>2.2 Generating Summary Information</h2> <p>You can generate summaries of your own tree and output these to the screen or a file so you can look at it in a browser. Email will only be sent out if you specify -p or -n. </p><p>To generate a summary and write that to the console simply run mail_tree. <br>$CMISS_EXAMPLES/common/cmiss_type1/mail_tree.pl </p><p>To create a summary html file run, then view it with a browser. You will probably want to update the web pages to any examples you are interested in as in section 2.1 <br>$CMISS_EXAMPLES/common/cmiss_type1/mail_tree.pl -f /tmp/filename.html </p><p>To send mail all programmers in the notify lists, which also is done automatically each night run <br>$CMISS_EXAMPLES/common/cmiss_type1/mail_tree.pl -p '.*' </p><p>To send a summary mail message to yourself run <br>$CMISS_EXAMPLES/common/cmiss_type1/mail_tree.pl -n your_email_address

</p><p></p><table border="2" width="90%"> <caption><h3>$CMISS_EXAMPLES/common/cmiss_type1/mail_tree.pl options</h3></caption> <tbody><tr><td><b>-h</b></td><td>Writes usage.</td></tr> <tr><td><b>-e</b></td><td>Relative path from example root below which examples will be included in the mail.</td></tr> <tr><td><b>-p</b></td><td>Regexp to match programmers to whom mail is sent.</td></tr> <tr><td><b>-n</b></td><td>Email address which will be notified for every example searched by this mail. This is a space separated list (use quotes if specifying more than one address. These email addresses are not filtered by the -p regexp.</td></tr> <tr><td><b>-f</b></td><td>This specifies a filename to which will be written the mail for every example in the -e path.</td></tr> <tr><td><b>-o</b></td><td>This specifies a directory which contains the output from testing the examples in the -e path. The default is to use the normal example tree (${CMISS_EXAMPLES}). This option can be useful if you have run your own example testing and want to generate a summary of the testing results output but not copied to the default example tree.</td></tr> <tr><td><b>-F</b></td><td>Include the Fortran check errors in the summary information - possibly only useful when generating the daily programmer's email?</td></tr> </tbody></table>

<h3>More examples</h3> <p>$CMISS_EXAMPLES/common/cmiss_type1/mail_tree.pl -e 1 -p "s.blackett@auckland.ac.nz" <br>Sends mail for examples below 1 in the tree but only to "s.blackett@auckland.ac.nz". This will only send mail about examples "s.blackett" is usually notified. </p><p>$CMISS_EXAMPLES/common/cmiss_type1/mail_tree.pl -e 1 -n "s.blackett@auckland.ac.nz" <br>Sends mail for examples below 1 in the tree but only to "s.blackett@auckland.ac.nz". This will send mail about every example in the 1 tree, whether "s.blackett" is normally notified about it or not. </p><p>$CMISS_EXAMPLES/common/cmiss_type1/mail_tree.pl -f notify_all.html <br>Generates a file called notify_all.html which contains the mail output for all the examples. </p><p>$CMISS_EXAMPLES/common/cmiss_type1/mail_tree.pl -e andre/tests -o /tmp/andre/cmiss_output -n "andre@bob.com" -f andres_test_summary.html <br>If you have run the test_tree.pl script above on your own subset of test examples (the andre/tests branch of your local example tree) and had the output generated into /tmp/andre/cmiss_output, this command will summarise the testing results and generate a file called andres_test_summary.html and send the summary the andre@bob.com via email.

</p><h2>3 Your own Example Tree</h2> <p>When working on the examples it is very useful to have your own copy of part of the example tree. To do this you need to be a cmiss programmer. Having your own copy allows you to commit changes only when they are right, to try all the testing versions and even to view the html generated pages for your own testing without affecting anyone else, great for testing changes to cmiss before you commit them. The main consideration is disk space as the full example tree is nearly about 900Mb, so use some temporary space, not on /usr/people and only check out the parts you need. </p><p>To get your tree working with all the command in your document you first need the common directory. Check your CVSROOT is correct. For the esu* system we use /hpc/cmiss/cvs. Change to your chosen place for your examples, say /var/tmp/shanes_examples/ then do <br>cvs checkout examples/common </p><p>Now checkout the part of the tree you want, say you are working on the examples below 2/21/219 then you can just checkout this part of the tree. <br>cvs checkout examples/2/21/219 </p><p>Finally set you CMISS_EXAMPLES variable to point to this tree, if I have my files in /var/tmp/shanes_examples/ then I do <br>setenv CMISS_EXAMPLES /var/tmp/shanes_examples/examples </p><p>Now you can do any of the commands on this page on your own copy of the tree. If you generate html as in Section 2.1 then these won't be visible on the web but you can just view the files directly, to look at the 2/21/219 directory I would just enter into Netscape file:/var/tmp/shanes_examples/examples/2/21/219/index.html </p><p>If you run tests as in section 1.2 on your own tree you can generate html or summary information as described in section 2.2 even while the tests are still running. </p><p>If you make changes to your tree you want to put into the main tree then commit these changes through <a href="/cm/wiki/IntroductionToCVS">cvs</a>.

</p><h2>4 Creating examples</h2> <p>So how do I make an example? </p><p>Personally I copy an example that is closest to the one I am adding, remove the CVS directories from all the subdirectories of the copy, update all the files and then add it into the example tree. Only commit the web_data, cmiss_input and testing parts of the example though, as the index.html and output directories are generated each night and so should not be in the source CVS. </p><h2>4.1 Easy example creation</h2> <p>To make a new example you should checkout the part of the tree you are placing it in as in Section 3. You may like to use the -l option on checkout to avoid checking out subdirectories.  Do NOT forget to check out examples/common and to have $CMISS_EXAMPLES set to your own examples directory for this work.

</p><p>$CMISS_EXAMPLES/common/cmiss_type1/make_example.pl -e 1/5r/5rg <br>Creates a "blank" example at 1/5r/5rg </p><p>If you want to use the example for cm testing specify the option -t </p><p>$CMISS_EXAMPLES/common/cmiss_type1/make_example.pl -e 1/5r/5rg -t <br>Creates a "blank" cm testing example at 1/5r/5rg

</p><p>Now.. </p><ul><li>Edit web_data/name.txt</li><li>Edit web_data/description.html</li><li>Edit cmiss_input/example_*.com</li><li>Put any other files required into cmiss_input directory</li></ul> <p>And for testing examples... </p><ul><li>Edit cmiss_input/test_output.com.</li><li>Edit or replace testing/ndiff/generic/cmiss_test.out with the

expected output from test_output.com.</li><li>Add any other required output files into testing/ndiff/generic. The ndiff directory is for answers that contain real numbers and will be compared with test output by ndiff. If the output does not contain any real numbers you can instead put output files in testing/diff/generic for a literal comparison.</li><li>If you wish to specify any special options for ndiff or diff then add an options file for each special output file. The names of the options files should be the output file name appended with ".opt". For example "cmiss_test.out.opt" specifies options for the diff performed with "cmiss_test.out". An options file in the "generic" directory is applied to all versions unless an options file exists in the specific version directory. Each line of an options file is passed as an argument to the diff program before the filename arguments.<br> </li><li>Some useful options for ndiff are mentioned below under Numerical Difference Testing. </li></ul>

<p>If it is a front end graphics example (a)... </p><ul><li>Add a 512x512 png screenshot with the same name as the example into the web_data directory. (If the screenshot is present it will automatically be displayed on the generated web pages).</li><li>Ensure that any exnode and exelem files you read in from your com file are prefixed by $example/ at the start of the filename path and that you include the file extension, eg 'gfx read node $example/cube.exnode'.  If you do not do this your example tests will fail and you will not be able to run your example using the -example a/path/to/example cmgui command line option.<br></li><li>Add licensing information to at least your comfile (and potentially other files) as you will be adding your example to the publicly visiable subversion repository.  We suggest the mozilla tri-license as a good option.<br></li></ul><p>Finally commit this all into subversion (new front end graphics examples) or CVS (backend examples). <br></p><p><i>For subversion:</i></p><ul><li>In the parent directory of your new example (eg a) run 'svn update'.<br></li><li>Add your example files to the repository by 'svn add example_dir_name'.</li><li>Now commit all this to the repository. In the parent directory of your new example run 'svn commit -m "Added an example to test blah blah" example_dir_name'<br></li></ul><p><i>For CVS:</i><br></p><ul><li>In the main directory of your new example run 'cvs update'</li><li>All your new files that were automatically added should have an A beside them.</li><li>Most other files you have added will have ? for unknown. Add them to the repository by 'cvs add new_filename'.</li><li>Now commit all this to the repository. In the main directory of your new example run 'cvs commit -m "Added an example to test blah blah"'</li></ul>

<h2>4.2 Editing examples</h2>

<p>Just checkout the examples that you wish to edit, make your changes, test the changes, and then check these into cvs. Running 'cvs update' always ensures you are up to date with the repository but does not eliminate any changes you have made. The symbols by each filename indicate its status in your directory. A means the file is scheduled for addition, R means the file is scheduled for removal, M means that you have made some modifications that aren't checked in, ? means that the file is unknown by CVS, U means that your file has been updated from the repository and C means your changes conflict with those that someone else has made (Get help with conflicts unless you know what to do).

</p><ul><li>To add a directory: <br>1. Simply add it to cvs 'cvs add dirname'</li><li>To add a file <br>1. Add the file to cvs <br>2. Run 'cvs update' <br>3. Commit your changes with 'cvs commit -m "Added another file" filename'</li><li>To remove a file <br>1. Remove the file using 'rm filename' <br>2. Tell cvs that you are going to remove the file with 'cvs removie filename' <br>3. Run 'cvs update' <br>4. Commit your changes with 'cvs commit -m "File removed as superseded by another file" filename'</li><li>Checking in changes <br>1. 'Run cvs update' <br>2. Commit your changes with 'cvs commit -m "Added streamlines" filename'</li></ul>

<h2>4.3 Extending examples</h2>

<h2>4.4 Complicated example creation</h2> <p>So the existing struture doesn't do what you want. </p><p>By changing the control directory you can make an example do whatever you want. </p><ul><li>control/html: Executed to generate the index.html</li><li>control/test: Used to generate all output in output directory. The default implementation control multiprocessing, any number of diff functions and multiple versions and operating systems</li><li>control/test_run: Run by the default test program to actually execute the example. This is something that you may want to customise. The flags in the end of a version file get passed here.</li></ul>

<h2>4.5 Numerical Difference Testing</h2>

<p> When testing an example, the generated answers are tested against a correct or fixed set of answers contained within the "testing" directory. If the answers are contained below the "diff" directory then they must be exactly the same, while if they are contained within a "ndiff" directory the answers are considered the same if they are within a specified numerical tolerance.</p>

<p> One of the options that ndiff can take is a tolerance (specified by --tolerance=number%, ie a number followed by an optional % sign, used to specify a relative tolerance) which can be used to tolerate any slight numerical differences. Currently the tolerance has a default level of 0.0, so any differences that are not simply variations in format (e.g. Fortran's D format) will be caught.</p>

<p> The tolerance of the numerical differences and regular expressions for lines on which differences should be ignored can be specified through the creation of appropriate "*.opt" files (see example creation above). An example of such a file is shown below: </p><pre>--tolerance=1e-10<br>--ignore-matching-lines=^Conditionnumber<br>--match-regexp=^Time<br></pre>

<p> If the tolerance is not zero, a change in the last digit is accepted because, even though this change may be larger than the tolerance, it may represent a difference that is smaller. If more accuracy is required in the output then more significant figures should be specified in the expected output.</p>

<p> --ignore-matching-lines is a regexp diff option which excludes matching lines. --match-regexp is an ndiff option which will exclude matching lines and not pass them on to the diff program. </p>

<h2>5 Example Testing on hpc1</h2>

<p> You might be wondering how to create an example that is run on hpc so you can test whether your parallel program is giving you the right answers ? </p> <p> Overnight on hpc a similar testing tree is run but it will only run the cm program with the irix64 operating system. These two things are specified as the -p and -s switches for the test_tree.pl program. Thus the test_tree.pl program requires that a versions file, (= /path/to/example/control/versions) keeps a corresponding entry for the operating system (irix64) and program name (cm) in it. This is done by adding line(s) in the versions file like: </p><pre>irix64 cm cm_n32_mp_2 $CMISS_ROOT/cm/bin/cm_n32_mp generic -p 2<br>irix64 cm cm_n32_mp_4 $CMISS_ROOT/cm/bin/cm_n32_mp generic -p 4<br>irix64 cm cm_n32_mp_8 $CMISS_ROOT/cm/bin/cm_n32_mp generic -p 8<br></pre> Thus the cm_n32_mp executable will be run on this example each night. It will have 3 different versions, one for 2,4 and 8 threads. The -p n flag will specify the number of cm threads to be used for the cm_n32_mp executable.

<p> So how do I add this line ? You simply check out the versions file using cvs and then add this line to it. You can run different executables by altering/adding these lines to the versions file. Examples are updated on hpc via cvs each night, and then the output directory is copied back to esu1 (via secure copy, scp) once examples have finished on hpc1. </p>

Personal tools

Overview of test examples