History for Cmgui nightly build

changed:
-
Brief overview

  Cmiss is built daily for several different machine architectures. As part of this process various versions of cmgui are built. The executables built are then used to test the examples.  If all the examples pass then the executables are copied over to $CMISS_ROOT/bin.  Testing takes a long time to complete so the daily build is scheduled to start post midnight, when machine load should be lighter.

  This nightly build process takes care of the following:

  - back up of previous log files

  - building cmgui

  - building unemap

  - building cm

  - tidying up log files and emailing results of build

The jobs

  A cron job starts off the build process.  All built executables reside on hpc but different machines are used to build executables for different machine architectures.
  The nightly cronjob scripts are found in '/hpc/cmiss/cmiss_utils/cronjobs' but some of the files in there may no longer be used.  There are four crontab files which indicate the crontab syntax used for when to run scripts.  The key crontab files are crontab_esu8.file and crontab_hpc.file.  These match the crontab files on esu8 and hpc respectively.  I am uncertain if the other two are used any more. To temporarily disable or change the scheduled run times of the jobs see [Scheduling cronjobs].

  The main script is 'cmiss_cronjobs_daily_esu8.sh' which is run every day at 1am.   This starts jobs on esu8 including [cmissmake].  Cmissmake builds cm, cmgui or unemap on multiple machines.  For cmgui it is called indirectly from the esu8 cronjob from the line ::

    make -C "$CMISS_ROOT/cmgui" cronjob 2>&1

  The -C flag results in a change into the cmgui directory and then make is called with the parameter cronjob.  Among other things this call results in a 'make update' which runs cmissmake if the user is cmiss.

  The esu8 job also spawns jobs on bioeng69, bioeng22 and esu20.  It takes care of running the scripts::

    cmiss_cronjobs_daily_bioeng69.sh 
    cmiss_cronjobs_daily_bioeng22.sh
    cmiss_cronjobs_daily_esu20.sh

  A job also starts on hpc at 2am::

    cmiss_cronjobs_daily_hpc.sh

  As far as I know the following job no longer runs::

    cmiss_cronjobs_daily_esp167.sh


  At 6:50am, hopefully after the jobs have finished on esu8, bioeng69, esu20 and hpc a final job is run which takes care of updating log files, generating the web pages, emailing out the results of the tests and updating executables::

    cmiss_cronjobs_daily_final.sh

  This job is run on bioeng22, so if bioeng22 is switched off no notification emails will be sent and the website will note be updated.


The machines

  Several machines are used in the overnight build process. They all need to be up and running for successful completion of the overnight build.

  Cronjobs run on:

  * esu8, SGI Octane running IRIX (64 bit chipset). In server room (level 6).

  * hpc, IBM super computer running AIX.  Owen G Glenn building.

  Further scripts run on:

  * bioeng69, PC running linux (i686 32 bit chipset). In server room (level 6)

  * bioeng22 Large PC running linux (amd64 bit chipset). Outside server room by poster printer (Level 6).

  * esu20, SGI Octane running IRIX (64 bit chipset). In server room (level 6).

  In addition cmissmake is run which uses the above (except for esu20) plus the two macs

  * bioeng21 Apple imac (power pc chipset) running OS X. In the wet lab, soon to be obsolete (level 6). 

  * bioeng46 Apple imac (intel chipset) running OS X. On Peter Bier's desk (level 6).


The logs

  To check whether the cron jobs have run correctly you can examine the logs.
  Each job that is run daily generates an output file which has the suffix '.day.out'.  (Eg esu8.day.out)
  Before the previous days output is overwritten, the previous day's job output is concatenated onto the end of a log file for that week.  This file has the suffix '.week.log' (Eg esu8.week.log).  At the start of each new week the weekly log file is copied over to a file with the suffix '.lastweek.log' (eg esu8.lastweek.log)

Common compilation failure:
  
  In most cases, compilation failure is caused by the occurrence of a lock file in the local svn respository. Normally this affects the cross compilation for i386-win32 and i686-linux in bioeng22.

Executable locations

  If compilation succeeds on a nightly build, for a program on a particular architecture, then the executable can be found on hpc by looking in $CMISS_ROOT/program/bin/architecture.  Eg the cmgui-wx executable for i686-linux machines can be found in '/hpc/cmiss/cmgui/bin/i686-linux'.  This directory contains the last successfully compiled executable of cmgui-wx (and other cmgui versions) for i686-linux machines.

  Note that if ALL the examples pass, for EVERY version of cmgui then the executables are copied to the appropriate  $CMISS_ROOT/bin/architecture directory.  Eg the cmgui-wx executable for i686-linux would be copied to '/hpc/cmiss/bin/i686-linux'.  The $CMISS_ROOT/bin directory hence contains the last executables built when every version of cmgui successfully passed the tests, sorted by architecture.  Note that /hpc/cmiss/bin/i686-linux has architecture specific executables for lots of different programs (eg cm, cmgui, cmgui-debug, cmgui-wx, unemap etc).