Cmgui nightly build

Brief overview

Cmiss is built daily for several different machine architectures. As part of this process various versions of cmgui are built. The executables built are then used to test the examples. If all the examples pass then the executables are copied over to $CMISS_ROOT/bin. Testing takes a long time to complete so the daily build is scheduled to start post midnight, when machine load should be lighter.

This nightly build process takes care of the following:

• back up of previous log files
• building cmgui
• building unemap
• building cm
• building zinc
• testing examples
• tidying up log files and emailing results of build

The jobs

The first cron job run updates the subversion source for cmgui. This is done via a cronjob as esu8 does not have svn access and we want to ensure the code base is up to date before any builds are performed. A cron job starts off the build process. All built executables reside on the hpc file system but different machines are used to build executables for different machine architectures. The nightly cronjob scripts are found in '/hpc/cmiss/cmiss_utils/cronjobs'. These jobs are all under CVS control. If you edit them you should check out a local copy of cmiss_utils from CVS and then commit your changes (as yourself rather than as cmiss). All cron jobs are now triggered from bioeng85. Note that crontab_bioeng85.file is a backup of the crontab file used on bioeng85. If you update the bioeng85 crontab file please also edit the crontab_bioeng85.file and check in your changes. To temporarily disable or change the scheduled run times of the jobs see Scheduling cronjobs. Currently four jobs are scheduled to run using the crontab file, with other jobs called from those scripts.

The cmiss_cronjobs_svn_update.sh job peforms an svn update of the cmgui source and is run just before midnight.

The main script is 'cmiss_cronjobs_daily_master.sh' which is run every day at 12 midnight. This starts jobs on several other machines which results in building cm, cmgui, unemap and zinc on multiple machines.

The master job spawns jobs on bioeng85, bioeng1031 and esu8. It takes care of running the scripts:

cmiss_cronjobs_daily_bioeng85.sh
cmiss_cronjobs_daily_bioeng1031.sh
cmiss_cronjobs_daily_esu8.sh

After compilation is finished on bioeng1031, the bioeng1031 script will run cmiss_cronjobs_daily_bioeng22.sh on bioeng22 to test the new updated 64bit linux executables using the examples tree.

At 4am a cron job is triggered to run scripts for cm documentation:

cmiss_cronjobs_daily_cm-docs.sh

At 5:50am, hopefully after the jobs have finished on esu8, bioeng22, bioeng85, bioeng1031 a final cron job is run which takes care of checking cmgui and zinc are up to date, updating log files, generating the web pages, emailing out the results of the tests and updating executables:

cmiss_cronjobs_daily_final.sh

This job is run on bioeng1031, so if bioeng1031 is switched off no notification emails will be sent and the website will note be updated. For cmgui the check that code is up to date is done using the line

make -C "$CMISS_ROOT/cmgui" cronjob 2>&1

The -C flag results in a change into the cmgui directory and then make is called with the parameter cronjob. Among other things this call results in a 'make update' which runs cmissmake if the user is cmiss. The zinc up to date check is done in a similar way.

The machines

Several machines are used in the overnight build process. They all need to be up and running for successful completion of the overnight build.

Cronjobs run on:

• bioeng85, PC running linux (32 bit chipset). In server room (level 6).
• esu8, SGI Octane running IRIX (64 bit chipset). In server room (level 6).

Further scripts run on:

• bioeng1031, PC running linux (64 bit chipset). Virtual machine in the server room (level 6).
• bioeng22, Large PC running linux (amd64 bit chipset). Outside server room by poster printer (Level 6).

cmissmake is run on bioeng46, bioeng85, bioeng1031, bioeng1032 and esu8. Overnight testing is run on bioeng85, bioeng22 and esu8.

The logs

To check whether the cron jobs have run correctly you can examine the logs. Each job that is run daily generates an output file which has the suffix '.day.out'. (Eg esu8.day.out) Before the previous days output is overwritten, the previous day's job output is concatenated onto the end of a log file for that week. This file has the suffix '.week.log' (Eg esu8.week.log). At the start of each new week the weekly log file is copied over to a file with the suffix '.lastweek.log' (eg esu8.lastweek.log)

Common compilation failure:

In most cases, compilation failure is caused by the occurrence of a lock file in the local svn respository. Normally this affects the cross compilation for i386-win32 and i686-linux in bioeng85.

Executable locations

If compilation succeeds on a nightly build, for a program on a particular architecture, then the executable can be found on hpc by looking in $CMISS_ROOT/program/bin/architecture. Eg the cmgui-wx executable for i686-linux machines can be found in '/hpc/cmiss/cmgui/bin/i686-linux'. This directory contains the last successfully compiled executable of cmgui-wx (and other cmgui versions) for i686-linux machines.

Note that if ALL the examples pass, for EVERY version of cmgui then the executables are copied to the appropriate $CMISS_ROOT/bin/architecture directory. Eg the cmgui-wx executable for i686-linux would be copied to '/hpc/cmiss/bin/i686-linux'. The $CMISS_ROOT/bin directory hence contains the last executables built when every version of cmgui successfully passed the tests, sorted by architecture. Note that /hpc/cmiss/bin/i686-linux has architecture specific executables for lots of different programs (eg cm, cmgui, cmgui-debug, cmgui-wx, unemap etc).