Personal tools
You are here: Home User Services Documentation Ristra Reference Card
Document Actions

Ristra Reference Card

Instructions for how to do many things on Ristra, including compiling, running jobs, etc

This page includes:

For assistance, please contact the Help Desk at or (505)  277-8348 (Monday through Friday, 9 a.m. to 5 p.m. Mountain Time)


Login using secure-shell (ssh):


Transfer files to the Linux cluster from your home machine  using secure copy protocol (scp):

scp <file> <user><dest>

Your Account

Change your password on the Linux cluster:


Change your login shell on the Linux cluster:



Please forward your mail to your local system. To forward mail,  create a .forward file in your Linux cluster home directory and put your email address in it:



Compilers from GNU and IBM are installed.


gcc –o exec [options] filename(s).c


g++ –o exec [options] filename(s).C

GNU Fortran77

g77 –o exec [options] filename(s).f

Programming Tools

Source code debuggers:


Selecting an MPI Library

There are several builds of the MPICH MPI library on each Linux cluster.  Some use the special GM interface to the Myrinet high-speed network hardware and some use TCP/IP. Select  an MPICH build by setting the MPIHOME environment variable in your .cshrc file (if your login shell is tcsh). You can set this  variable by editing your .cshrc file and removing the comment sentinel (#) from the beginning of the  line specifying the MPICH library build that you want to use. Be sure all other lines setting a value for MPIHOME start with the comment sentinel (#).  Refer to your .cshrc file for examples. Log out and log back in to load the new environment.  Use the "which mpirun" command to verify the MPICH library build you have chosen.

For example:

which mpirun


shows that you are using MPICH with the GM interface, and the corresponding MPICH compiling scripts.

Compiling with  MPI


mpicc –o exec [options] filename(s).c



mpiCC –o exec [options] filename(s).C



mpif77 –o exec [options] filename(s).f



mpif90 -o exec [options] filename(s).f90

Linking Serial Libraries


-L/usr/local/lib -lcblas -lf77blas -latlas


-L/usr/local/lib -llapack -lcblas -lf77blas -latlas

FFTW release 2, complex input

-L/usr/local/lib -lfftw

FFTW release 2, real input

-L/usr/local/lib -lrfftw

FFTW release 3, real or complex input

-L/usr/local/lib -lfftw3

Linking Parallel Libraries

MPICH P4 Compatible Libraries:

FFTW release 2, complex input

-L/usr/parallel/lib/p4 -lfftw_mpi -lfftw

FFTW release 2, real input

-L/usr/parallel/lib/p4 -lrfftw_mpi -lrfftw


-L/usr/parallel/lib/p4 -lscalapack -lblacsf77init -lblacs -lblacsf77init -L/usr/local/lib -lf77blas -latlas

MPICH GM Compatible Libraries:

FFTW release 2, complex input

-L/usr/parallel/lib/gm -lfftw_mpi -lfftw

FFTW release 2, real input

-L/usr/parallel/lib/gm -lrfftw_mpi -lrfftw


-L/usr/parallel/lib/gm -lscalapack -lblacsf77init -lblacs -lblacsf77init -L/usr/local/lib -lf77blas -latlas

Submitting PBS Jobs

PBS supports two modes for running jobs – batch  and interactive. With batch mode, a user submits a job, and the job is queued, scheduled and run without any further interaction.  With interactive mode, a user submits a request for an interactive session on a set of compute nodes assigned by the PBS job manager.

The following examples show simple scripts for batch mode jobs:

Portable Batch System (PBS)

PBS Batch Mode Sample Scripts:

Request 16 nodes and start one mpi process on each node:

#PBS -l nodes=16:ppn=2
mpirun -machinefile $PBS_NODEFILE -np 16 myprog.exe

Request 8 nodes and start 2 mpi processes on each node:

#PBS -l nodes=8:ppn=2
mpirun -machinefile $PBS_NODEFILE -np 16 myprog.exe

Standard output and error files will be written to your working directory.  To submit the script to PBS, use the following command:

qsub <pbs_script_file>

PBS Interactive Mode Sample Session:
Request an interactive session with 8 nodes: qsub –I –l nodes=8

Run your program as follows:

Start one mpi process on each node: 

% mpirun -machinefile $PBS_NODEFILE -np 8 myprog.exe

Start two mpi processes on each node:

% mpirun -machinefile $PBS_NODEFILE -np 16 myprog.exe

Release your nodes when finished by typing: \

% exit

Display contents and status of queue:

% qstat -a

Remove job from queue:

% qdel <job_id>

Powered by Plone CMS, the Open Source Content Management System

This site conforms to the following standards: