University of Heidelberg
BIOQUANT

Using the BioQuant Compute Cluster

 

Description

The Bioquant Cluster is a collection of compute servers accessible through a common batch system and a scheduler. Anyone with a user account at bioquant can submit a job to the cluster. The cluster is primarily reserved for batch jobs which run non-interactively, but interactive sessions for testing and/or debugging purposes are also possible.

Here is a brief description of the available hardware and the installed software, a mini how-to about job submission and PBS Scripting. If you have any questions which are not addressed here, please do not hesitate to contact us at support-it@bioquant.uni-heidelberg.de

Hardware

4 x Sun Fire x4100 2x Dual-Core AMD Opteron 2218 8GB Memory
27 x Sun Fire x4100 2x Dual-Core AMD Opteron 2220 8GB Memory
1 x Sun Fire x4100 2 x Dual-Core AMD Opteron 2220 20GB Memory
3 x Sun Fire x4100 2 x Dual-Core AMD Opteron 2220 24GB Memory
2 x SunFire 4600 8 x Quad-Core AMD Opteron 8384 256GB Memory
14 x IBM xSeries 2 x Quad Intel Xeon E5530 16GB Memory
28 x IBM xSeries 2 x Quad Intel Xeon E5530 20GB Memory
NVIDIA Tesla S1070 accessible over two frontend nodes
20 x SuperMicro 2 x Quad Intel Xeon E5520 12GB Memory
14 x IBM xSeries 2 x Quad Intel Xeon E5507 24GB Memory
14 x IBM 2 x 6-Core Intel Xeon E5-2630 2.30GHz 64GB Memory
1 x uv2000, 256 CPUs, 2 TB Memory

Software

on all nodes

  • OS: CentOS 6.4 x86_64 Kernel 2.6.32
  • OpenMPI 1.6.4-gcc
  • Compiler Suites: gcc 4.4.7, ICC 11.0
  • Batch System: Torque
  • Scheduler: Maui

on selected nodes

  • Mathematica 10
  • Matlab 2015a (Toolboxes: Optimization Toolbox, Statistics Toolbox, Image Processing Toolbox, Bioinformatics Toolbox, Curve Fitting Toolbox, Fuzzy Logic Toolbox, Image Processing Toolbox, Neural Network Toolbox, Partial Differential Equation Toolbox, SimBiology, Simulink, Symbolic Math Toolbox, Wavelet Toolbox, Global Optimization Toolbox, Distributing Computing Toolbox)
  • Maple 16

Additional software can be installed on the cluster provided there is a stable version of it and it doesn't break any dependencies of the already installed software. Just drop us an email with the information on the desired software. We will test it and install it on success.

Using the cluster

General usage

Login to one of our application servers appl2, appl9 or appl10 with your BioQuant account. At the beginning you start an interactive session on one of the nodes to run a test. Perhaps you have to compile your programm at the node first. If your test was succesfull you should submitt jobs by a script.

Common commands for job submission and monitoring like qsub, qstat, pbstop, pbsnodes (see below) are available on all three machines.

Interactive Jobs

You can start an interactive job on a cluster node via the batch system

 qsub -I  

or with Xdisplay forwarding:

 qsub -X -I 

Please be aware that interactive jobs are only allowed for initial testing. The normal usage is submitting to the queueing system to ensure the optimum use of the cluster resources.

Job submission

To submit a job into the cluster, create a PBS script (see below: Example PBS Scripts) and feed it into the qsub command, e.g.

 qsub mypbsscript.sh 

You can ask for additional resources either directly on the qsub command or from within the PBS Script. Some examples for the forme

 qsub -l ncpus=4 mypbsscript.sh # ask for four tasks (processes).
qsub -l nodes=4:ppn=4 mypbsscript.sh # ask for four nodes with four processes on each node.
qsub -l nodes=1:matlab mypbsscript.sh # ask for a node that has the matlab feature.
qsub -l mem=200gb mypbsscript.sh # ask for a node with 200gb of memory
qsub -l nodes=2,mem=200gb,cput=00:15:00 mypbsscript.sh # ask for two nodes with a total of 200gb of memory for 15 minutes of processing time
qsub -q cuda mypbsscript.sh # ask that the job be submitted to the cuda queue (will be executed on a Tesla machine; see above: Hardware)

UV2000

If you need more than 250GB memory, you have to request the node uv2000. The uv2000 is divided into 16 virtuell nodes.
Each node can use 120 GB memory. Depending on how much memory you need you have to request the nodes.
For example, if you need 360 GB memory you have to request:

qsub -l nodes=3:ppn=2:uv2000

or

qsub -l nodes=3:uv2000,pmem=120gb

PBS Scripting

A PBS Script is a Shell script which also contains special directives which are interpreted by the PBS Server.
Those directives begin with #PBS. Like every shell script, PBS scripts begin with the interpreter declaration:

 #!/bin/sh

 # Following that, we can use the first PBS directive to set the name of our job: 
 # Job Name (this line is a comment and won't be processed) #PBS -N my_pbs_job # Now we can ask for some resources, e.g. a total time of 15 hours and one node with 4 processors: #PBS -l walltime=15:00:00 #PBS -l nodes=1:ppn=4 # A most important directive is the one which ensures that the Linux # environment for the job is the same as the one we're working in: #PBS -V

# Then we define an alternative e-mail address where PBS messages will be sent to (default is local mail)...
#PBS -M joe.doe@bioquant.uni-heidelberg.de
# ...and specify that an e-mail should be sent to the user when the job begins (b), ends (e) or aborts (a)
#PBS -m bea
# Define a file where stderr will be redirected to
#PBS -e my_pbs_job.err
# Define a file where stdout will be redirected to
#PBS -o my_pbs_job.log
# finally we let our job run as usual
./my_job
# or if it is an MPI job mpirun
./my_mpi_job

Some PBS-specific environment variables:
PBS_JOBID: The job ID of the job
PBS_JOBNAME: The name of the job
PBS_NODEFILE: Path to the file which contains the list with the reserved nodes
PBS_O_WORKDIR The current working directory
PBS_O_QUEUE: The name of the original queue to which the job was submitted

More details to the qsub command

Available Queues

General Queues:
(Note: You cannot submit into those queues directly; jobs are routed into those queues
according to the cput requested; defaults to batch, on which a limitation of max_user_run applies.)

short: {default|max}.cput=01:00:00
research: {default|max}.cput=05:00:00
batch: default.cput=168:00:00 default.walltime=336:00:00 (max. 30 Running Jobs/User)
quick: {default|max}.cput=00:15:00

funnel: routing queue for the above queues
cuda: special queue for jobs that run on the Tesla machines; use -q cuda in the qsub command line to submit to this queue.

Available Resources and Node Features

PBS Integrated Resources

  • ppn: # processors per node
  • mem: # total amount of memory
  • cput: # Maximum amount of CPU time used by all processes in the job (seconds,or [[HH:]MM:]SS)
  • walltime: # Maximum amount of real time during which the job can be in the running state

More details to PBS integrated resources

BioQuant HPC Specific Resources

The following resources are defined within our cluster. Most of them are self-explained.

  • matlab # a node with a Matlab installation
  • math # a node with a Mathematica installation
  • maple # a node with a Maple installation
  • xeon # a node with an Intel Xeon Processor
  • opteron # a node with an AMD Opteron Processor

Multithreaded jobs

If your job supports multi-threaded execution (e.g. Matlab executables with enabled multi-threading), you will need more than one processor (core) to execute it. This should be no problem as long as you know how many cores your job can use at most. If not, then you will presumably want to reserve a whole node for your job regardless of how many jobs you will start or how many cores you will be using. There are two ways to do that. The first is something like the followin

 qsub -I -l nodes=clnxxx-int:ppn=n 

where n is the total amount of processor cores of node clnxxx-int. You can find n for clnxxx-int with

 pbsnodes clnxxx-int  

n is the np value in the output of the command. The other way is to simply tell PBS explicitly that you need the whole node for yourself:

 qsub -I -W x=NACCESSPOLICY:SINGLEJOB 

or from a PBS script:

 #PBS -W x=NACCESSPOLICY:SINGLEJOB 

This node allocating manner apart from multi-threaded jobs is also apt for jobs that tend to consume most of the node's memory or network bandwidth. Please consider these possibilities before you submit a job and allocate nodes accordingly.

 

Monitoring

 # Show the complete status of the job
qstat -f <JOB_ID>
# Show all jobs
qstat -a
# Show queue status
qstat -q
# Show the nodes my job is running on
# without JOB_ID shows all jobs
qstat -n <JOB_ID>

More details to the qstat command  


Real-time performance monitoring Ganglia (accessible within Bioquant)

 

Cluster Users Meeting 2010

 

 

Contact: E-Mail (Last update: 13/10/2016)