CMSSW
From UF HPC Wiki
Contents |
Introduction
This is a short tutorial designed to take you through the entire process of launching CMSSW jobs on the local queue at the HPC. Much of this information is non-HPC specific, but is added for completeness. It is intended for bash users (the default shell).
Setup Your Environment
To setup your environment at login, insert the following code snippet into your .bash_profile (csh users must modify appropriately and place in .login) under the "User specific environment ..." section:
export SCRAM_ARCH=slc4_ia32_gcc345
export OSG_APP=${UFHPC_SCRATCH}/osg/app
export VO_CMS_SW_DIR=${OSG_APP}/cmssoft/cms
export CMS_PATH=${VO_CMS_SW_DIR}
source ${CMS_PATH}/cmsset_default.sh;
cvsbase=":pserver:anonymous@cmscvs.cern.ch:/cvs_server/repositories"
CVSROOT="${cvsbase}/CMSSW"; export CVSROOT
Before continuing, be sure you have sourced this environment!
source ${HOME}/.bash_profile
Setup Your CMSSW Area
You are not supposed to install software or launch jobs from the home area on the HPC. You must use one of the scratch disks, mounted at $UFHPC_SCRATCH (/scratch/ufhpc) or CRN (/scratch/crn). Simply make yourself a user area in the scratch space.
if [ ! -d ${UFHPC_SCRATCH}/${LOGNAME} ]; then mkdir ${UFHPC_SCRATCH}/${LOGNAME}; fi;
cd ${UFHPC_SCRATCH}/${LOGNAME};
From here, you may install your CMSSW area as usual. To see the list of installed software, use:
scramv1 list
To make a local CMSSW area, use:
scramv1 project CMSSW CMSSW_2_2_5
Download Packages
Enter your CMSSW area and initialize SCRAM:
cd CMSSW_2_2_5/src eval `scramv1 runtime -sh`
A common starting point is to use the FastSimulation to generate a few monte carlo events. If this describes you:
addpkg FastSimulation/Configuration addpkg FastSimulation/EventProducer
You may also use standard CVS tools:
cvs co RecoMET/METProducers
Remember to use the -R flag unless you want to check packages from the head, and also that it is typically only necessary to check packages out of CVS if you intend to modify them, or if you need a particular configuration (cfg) file from CVS.
Running Interactively
You should never run jobs (eg., call cmsRun) on the submit machine. You may test small jobs interactively from one of the test machines (eg., test05).
Modifying Code
Note that the submit machine is not a suitable place to develop and recompile code.
[mschmitt@submit Analysis]$ scramv1 b Parsing BuildFiles Out of memory!
If you need to develop code on the HPC, log into one of the test machines:
[mschmitt@submit ~]$ ssh test05
Last login: Tue Mar 3 09:39:16 2009 from submit.ufhpc
[mschmitt@r21a-s19 ~]$ cd ${UFHPC_SCRATCH}/mschmitt/CMSSW/CMSSW_2_2_5/src
[mschmitt@r21a-s19 src]$ scramv1 b
Reading cached build data
>> Local Products Rules ..... started
>> Local Products Rules ..... done
>> Building CMSSW version CMSSW_2_2_5 ----
You could also create and submit a job (see the next section) to compile CMSSW.
Create a Job Submission File
The sample PBS script can now be found on the PBS Sample Job Scripts page.
A few important notes on the job submission file:
1. Replace the email listed with the -M flag with your email address, and (obviously) the scratch area listed with your scratch area.
2. Your shell environment is not, by default, exported to the worker node (including $PWD). This can be circumvented with an appropriate submit flag (see below); however it is advisable to specify the entire required environment in your job submission file. It is not recommend for your job to refer to environment/setup files outside of the scratch disk, such as your .bash_profile!
3. The above job is intended for the testing queue (several processors kept separate for dedicated debugging purposes). To send it to the general queue, remove the -q flag (the entire line) at the beginning of the script and increase walltime (it should be safely larger than the expected runtime of your job -- but not too long, or it will be pushed farther down the queue).
Of course, you should take some time to review all of the available PBS flags and modify your own submission script appropriately. See "man qsub" on submit.hpc.ufl.edu...
Submit and Monitor the Job
There are three essential commands you will need to use to manage your jobs on the HPC. They are: qsub, qstat, and qdel.
To run the given script on a worker node:
qsub my_job.sh
To run the given script with your environment:
qsub -V my_job.sh
To look at all jobs in the queue:
qstat
To look at only your jobs in the queue:
qstat -u ${LOGNAME}
To monitor your jobs, polling every 10 seconds:
watch -n10 qstat -u ${LOGNAME}
To remove a job:
qdel 8494710.torque.ufhpc
To remove many jobs at once, create a script such as this:
#!/bin/bash
if [ $# -ne 2 ]; then echo "Usage: $0 first last"; fi;
list=`seq -f%f $1 $2 | cut -f1 -d.`;
for i in $list; do echo "removing job $i"; qdel ${i}.torque.ufhpc; done;
Don't forget to run a small test job to test your configuration before submitting large runs to the general queue.
