Linux Cluster Info

• Configuration
• Access
• Known problems
• Compilers
• MPICH
• Batch system
• Temporary disk space
• Transition to 64 bits

• 2 master + 34 compute nodes
• Dual Opteron 250 (2400 MHz)
• 4 GB RAM
• MPI over Infiniband
• SuSE Linux 10.0

• Master: frontend.physik.hu-berlin.de
• Accounts will be created on demand.

• Disabled nodes
  broken:	node03, node17
  unstable:	node34
  reserved:	-

   
• MPICH with gfortran (GCC-4) does not compile.
• MPICH with PathScale C++ compiler is missing.

GCC

• Version 4.0.2 (20050901), 64 bit
• Optimisation is off by default. Try -O2 or -O3 for production runs.
• Compiles in 64 bit by default. Use switch -m32 to compile in 32 bit mode.
• Includes new GNU Fortran 95 compiler (instead of g77), which supports Fortran 90ff (to some extend).
  Invocation: gfortran ...

Intel

• Version 9.0 (20050430)
• 32 bit and 64 bit versions installed in parallel. Actual version is selected via environment variables. Standard user setup defaults to 64 bit, use shell commands intel32 and intel64 to switch the current setup.
• Invocation:
  icc ... for C and C++
  ifort ... for Fortran 77, Fortran 90.
• icc help:
  man icc
  icc -help | less
  /software/intel/cc/9.0/doc/*   # documents for 32 bit
  /software/intel/cce/9.0/doc/*  # documents for 64 bit
• ifort help:
  man ifort
  etc.

Portland

• Version 6.2-3
• 32 bit and 64 bit versions installed in parallel. Default setup selects 64 bit version.
• Invocation:
  pgcc ... for C
  pgCC ... for C++
  pgf77 ... for Fortran 77
  pgf90 ... for Fortran 90/95
• Help:
  man pages
  /software/pgi/linux86-64/6.2/doc/index.htm

PathScale

• Version 2.3
• Compiles in 64 bit by default. Use switch -m32 to compile in 32 bit mode.
• Invocation:
  pathcc ... for C
  pathCC ... for C++
  pathf90 ... for Fortran 77, 90, 95
• help:
  man pages
  /software/pathscale/share/doc/pathscale-compilers-2.3/*

Parallel jobs should be submitted via the batch system.

top

Queues and their (tentative) limits

• The default queue is single.
• For parallel jobs, the queuing system allocates the processors and povides a machine file to be passed to mpirun, see sample script below.
• Note that PBS allocates nodes, not CPUs in the first place. You may ask for 8 CPUs with
  -l nodes=4:ppn=2 (8 CPUs on 4 nodes, using both processors) or with
  -l nodes=8 (8 CPUs on 8 nodes).
  In the latter case, the spare processors may be allocated to other jobs. Therefore, the first version is preferred if you want to have full control over the resources (memory, network traffic).
• Queue setup and limits are preliminary - please let me know about your plans to start parallel jobs for production.

Node info

      pbsnodes -a             # show status of all nodes
      pbsnodes -a nodeNN      # show status of specified node
      pbsnodes -l             # list inactive nodes
      pbsnodelist             # list status of all nodes (one per line)

Queue info

      qstat -Q                      # show all queues
      qstat -Q <queue>              # show status of specified queue
      qstat -f -Q <queue>           # show full info for specified queue
      qstat -q                      # show all queues (alternative format)
      qstat -q <queue>              # show status of specified queue (alt.)

Job submission and monitoring

      qsub <jobscript>              # submit to default queue
      qsub -q <queue> <jobscript>   # submit to specified queue
      qsub -l nodes=4:ppn=2 <jobscript>   # request 4x2 processors
      qsub -l nodes=nodeNN  <jobscript>   # run on specified node

      qsub -l cput=HH:MM:SS <jobscript>       # limit on CPU time (serial job)
      qsub -l walltime=HH:MM:SS <jobscript>   # limit on wallclock time (parallel job)

      qdel <job_no>                 # delete job (with <job_no> from qstat)

      qstat -a                      # show all jobs
      qstat -a <queue>              # show all jobs in specified queue        
      qstat -f <job_no>             # show full info for specified job
      qstat -n                      # show all jobs and the nodes they occupy

Jobscript

Note that the batch system does not perform an interactive login, but only starts the job script with a remote shell command. This means that the job does not "see" your full interactive environment. You may have to extend the PATH explicitly or specify commands with full pathname. As an illustration, run a job with the command "set" only - it writes the environment (as seen by the job script) to the output file.

top

• /data/cluster (500 GB, globally available)
• /scratch (58 GB, local to each compute node)
• /tmp (local to frontend and each compute node)

top

See also the following article.

Fortran90 programs don't seem to pose problems:

• integer remains at 32 bits,
• the correspondence real(8) <-> double precision still holds.