OpenMPI

From UF HPC Wiki

*NOTE* Executables built against OpenMPI can run over either IB or GigE. They prefer and will default to InfiniBand if the interfaces are detected and up. In the absence of an IB interconnect, OpenMPI apps will use GigE. When that happens you will see a message saying something like the following

--------------------------------------------------------------------------
[0,1,1]: MVAPI on host r6b-s5.local was unable to find any HCAs.
Another transport will be used instead, although this may result in
lower performance.
--------------------------------------------------------------------------

Unfortunately, recent OpenMPI releases have a bug that causes this fail over to not work.

To work around this, please make your MPIRUN command this:

mpirun --mca btl ^mvapi --mca btl_tcp_if_include eth0

This will cause OpenMPI to ignore IB, and use TCP over ethernet.

Contents 1 Versions Available 2 Compiling Programs 2.1 Fortran 2.2 C

Versions Available

Version	ABI	Compiler	Location	Notes
1.2.2	64 bit	Intel	/usr/mpi/intel/openmpi-1.2.2-1	For Torque/Maui
1.2.4	64 bit	Intel	/usr/mpi/intel/openmpi-1.2.4-1	For Torque/Maui
1.2.2	64 bit	GNU	/usr/mpi/gcc/openmpi-1.2.2-1	For Torque/Maui
1.2.2	64 bit	GNU	/usr/mpi/gcc4/openmpi-1.2.2-1	For Torque/Maui (compiled with GCC4)
1.2.2	32 bit	GNU	/usr/mpi/gcc-32/ompi-1.2.2	For Torque/Maui

Compiling Programs

Fortran

• Be sure that the fortran compiler you are using to compile your code using OpenMPI is coming from the right directory. For instance, if you wanted to use the Intel compiler for version 1.1.2 of OpenMPI, you would want to use the program mpif90 that resides in /opt/intel/ompi/1.1.2/bin.

• Sometimes it may help to add the following (or some variation thereof) to your makefiles:

F90LIBS       = -lifport -lifcoremt -lm -lcxa -lunwind
F90DIR        = /opt/intel/fce/9.1.037
CXXLINKFLAGS  = -L/opt/intel/ict/2.0.1/ita/6.0.1/lib -lstdc++ -L$(F90DIR)/lib/ $(F90LIBS)

• If you are using the METIS libraries, be sure to reference them:

LIBMET = -L/apps/metis/4.0/icc-em64t/lib -lmetis

C

• Make sure that you source the Intel build environment setup from /opt/intel/cce/9.1.044/bin/iccvars.sh.

• Be sure that the C compiler you are using to compile your code using OpenMPI is coming from the right directory. For instance, if you wanted to use the Intel compiler for version 1.2.0 of OpenMPI, you would want to use the program mpicc that resides in /opt/intel/ompi/1.2.0/bin.