VASP

From UF HPC Wiki

1 Introduction
2 Installation
3 VASP with MVAPICH (10/17/2008)
4 VASP 5.2.0 (08/24/2009)
5 Sample PBS Submission Script

Introduction

VAMP/VASP is a package for performing ab-initio quantum-mechanical molecular dynamics (MD) using pseudopotentials and a plane wave basis set. The approach implemented in VAMP/VASP is based on a finite-temperature local-density approximation (with the free energy as variational quantity) and an exact evaluation of the instantaneous electronic ground state at each MD-step using efficient matrix diagonalization schemes and an efficient Pulay mixing. These techniques avoid all problems occurring in the original Car-Parrinello method which is based on the simultaneous integration of electronic and ionic equations of motion. The interaction between ions and electrons is described using ultrasoft Vanderbilt pseudopotentials (US-PP) or the projector augmented wave method (PAW). Both techniques allow a considerable reduction of the necessary number of plane-waves per atom for transition metals and first row elements. Forces and stress can be easily calculated with VAMP/VASP and used to relax atoms into their instantaneous groundstate.

Installation

VASP version 4.6.31 executables have been built and installed in /apps/vasp/4.6.31/intel/bin. Please note the following:

These executables were built using
1. Intel compiler suite
2. MKL libraries
3. OpenMPI version 1.2.2 MPI libraries
4. torque TM API library
These executables include the code for the NEB, dimer, Lanczos, and dynamical matrix methods. See VTS Tools
As in the past there is a nebvasp-cd executable and a nebvasp-gp executable indicating charge density reduced in Z-direction and gamma-point only reduction in the Z direction, respectively.
A sample submission script is included below. Also, if your job will use more than eight (8) cpus, you should probably run it only on infiniband-enabled nodes. You can request IB-enabled nodes by the infiniband feature in your resource specification i.e.

#PBS -l nodes=8:ppn=4:infiniband

VASP with MVAPICH (10/17/2008)

There are two new VASP (4.6.34) executables that have been built and tested. They are....

/apps/vasp/4.6.34/intel9/mvapich/bin/nebvasp-cd /apps/vasp/4.6.34/intel9/mvapich/bin/nebvasp-gp

As many of you are aware, we have had difficulty for some time with VASP jobs. This has been particularly true of jobs using 32 or more processors. See https://bugzilla.hpc.ufl.edu/show_bug.cgi?id=1744 , for example. We have recently found that the problem seems to be with OpenMPI and is not exclusive to VASP. This problem has spanned multiple versions of OpenMPI including the latest production release. For this reason, these new executables are built with MVAPICH rather than OpenMPI. We have yet to observe the problem under MVAPICH.

Additionally, these new executables have been built using scaLAPACK. If you don't want to use scaLAPACK, you can disable it via the "LSCALAPACK = .F." in the INCAR file.

To use the new executables, you *must* change your MPI execution environment. The easiest way to do this is to

run "mpi-selector --set mvapich_intel-0.9.9".
log out
log back in
verify the change with "which mpirun" which should look like the following...

$ which mpirun
/usr/mpi/intel/mvapich-0.9.9/bin/mpirun

You will also have to modify your submission scripts to use the new executable. The relevant portions of a modified submission script are below.

set EXE = /apps/vasp/4.6.34/intel/mvapich/bin/nebvasp-cd
set CPUS = `cat $PBS_NODEFILE  | wc -l`

set IbEnabled = `/usr/local/sbin/IbEnabled`
if ( $IbEnabled ) then
     echo "Running on IB-enabled node set"
     set MPIRUN = "mpiexec"
else
     echo "MVAPICH is not supported over GigE"
     exit
endif

cd $PBS_O_WORKDIR
$MPIRUN -np $CPUS $EXE

Finally note that the OpenMPI-based executables will be removed. If you build your own VASP executables, you should rebuild them using mvapich. In most cases, this will just mean recompiling under the new MPI environment.

Again, the problem forcing this change is not unique to VASP and has been observed with AMBER and other OpenMPI-based executables. It is especially prevalent on jobs using 32 or more processors but is also seen on as few as 8 (rarely).

VASP 5.2.0 (08/24/2009)

Notes

VASP 5.2 is only available to properly licensed users
You must be a member of the "vasp5" group to access VASP 5.
Your MPI environment should be set (via mpi-selector) to "mvapich_intel10-0.9.9".
The executable path is /apps/vasp/5.2.0/intel/mvapich/vasp-cd

Libraries

FFT: MKL w/ FFTW 3.X wrappers
BLAS: goto_penrynp-r1.26
MKL 10.0.2.018: mkl_intel_lp64 mkl_sequential mkl_core
SCALAPACK: Built from source
BLACS: Built from source
MPI: MVAPICH 0.99

Makefile

.SUFFIXES: .inc .f .f90 .F
SUFFIX=.f90

FC=mpif90
FCL=$(FC)

FFLAGS =  -free -lowercase -assume byterecl 

OFLAG=-O3
OFLAG_HIGH = $(OFLAG)
OBJ_HIGH = 
OBJ_NOOPT = 
DEBUG  = -free -O0
INLINE = $(OFLAG)
#-----------------------------------------------------------------------
# additional options for CPP in parallel version (see also above):
# NGZhalf               charge density   reduced in Z direction
# wNGZhalf              gamma point only reduced in Z direction
# scaLAPACK             use scaLAPACK (usually slower on 100 Mbit Net)
#-----------------------------------------------------------------------
# Define these when using MKL BLAS?
#    -DRPROMU_DGEMV  -DRACCMU_DGEMV
#-----------------------------------------------------------------------
CPP_ = ./preprocess <$*.F | /usr/bin/cpp -P -C -traditional >$*$(SUFFIX)

CPP  = $(CPP_) -DHOST=\"LinuxUFHPC\" \
-DMPI \
-DIFC \
-DPGF90 \
-Dkind8 \
-DNGZhalf \
-Davoidalloc \
-DMPI_BLOCK=8000  \
-DCACHE_SIZE=4000 \
-DscaLAPACK
#-----------------------------------------------------------------------
# MKL 10.X
# set -DRPROMU_DGEMV  -DRACCMU_DGEMV in the CPP lines
#
MKLDIR  = -L/opt/intel/mkl/10.0.2.018/lib/em64t
MKLCORE = -lmkl_intel_lp64 -lmkl_sequential -lmkl_core
MKL     = $(MKLDIR) $(MKLCORE)
#-----------------------------------------------------------------------

#-----------------------------------------------------------------------
# GOTO BLAS Library (Slightly faster than MKL)
# http://www.cs.utexas.edu/users/kgoto/signup_first.html
#
BLAS = -L/opt/lib/goto/1.26 lgoto_penrynp-r1.26
#-----------------------------------------------------------------------

#-----------------------------------------------------------------------
# ScaLaPack + BLACS
#
SCA    = -L/home/taylor/scalapack/lib -lscalapack
BLACS  = -L/home/taylor/blacs/lib -lBlacsF77Init -lBlacsCInit -lBlacs

SCALAPACK = $(SCA) $(BLACS)
#-----------------------------------------------------------------------

#-----------------------------------------------------------------------
# MKL FFTs w/ FFTW wrappers
#
FFTLIB  = -lfftw3xf_intel

# Bonifide FFTW 3.1.2
#FFTLIB  = -L/opt/intel/fftw/3.1.2/lib -lfftw3
#-----------------------------------------------------------------------

#-----------------------------------------------------------------------
# Miscellaneous VASP routines

MISC = -L../vasp.5.lib -ldmy
#-----------------------------------------------------------------------

LIBS = $(MISC) $(FFTLIB) $(SCALAPACK) $(BLAS) $(MKL)

LINK   = -i_dynamic -Wl,-rpath -Wl,$(MKLDIR)
#-----------------------------------------------------------------------
# general rules and compile lines
#-----------------------------------------------------------------------
LPD_OBJS = ../vasp.5.lib/linpack_double.o
FFT_OBJS = fftmpi.o fftmpi_map.o fftw3d.o fft3dlib.o  

BASIC =  symmetry.o symlib.o   lattlib.o  random.o   

SOURCE = base.o     mpi.o      smart_allocate.o      xml.o  \
         constant.o jacobi.o   main_mpi.o  scala.o   \
         asa.o      lattice.o  poscar.o   ini.o       xclib.o     xclib_grad.o \
         radial.o   pseudo.o   mgrid.o    gridq.o     ebs.o  \
         mkpoints.o wave.o     wave_mpi.o  wave_high.o  \
         $(BASIC)   nonl.o     nonlr.o    nonl_high.o dfast.o    choleski2.o \
         mix.o      hamil.o    xcgrad.o   xcspin.o    potex1.o   potex2.o  \
         metagga.o constrmag.o cl_shift.o relativistic.o LDApU.o \
         paw_base.o egrad.o    pawsym.o   pawfock.o  pawlhf.o    paw.o   \
         mkpoints_full.o       charge.o   dipol.o    pot.o  \
         dos.o      elf.o      tet.o      tetweight.o hamil_rot.o \
         steep.o    chain.o    dyna.o     sphpro.o    us.o  core_rel.o \
         aedens.o   wavpre.o   wavpre_noio.o broyden.o \
         dynbr.o    rmm-diis.o reader.o   writer.o   tutor.o xml_writer.o \
         brent.o    stufak.o   fileio.o   opergrid.o stepver.o  \
         chgloc.o   fast_aug.o fock.o     mkpoints_change.o sym_grad.o \
         mymath.o   internals.o dimer_heyden.o dvvtrajectory.o vdwforcefield.o \
         hamil_high.o nmr.o    force.o \
         pead.o     subrot.o   subrot_scf.o pwlhf.o  gw_model.o optreal.o   davidson.o \
         electron.o rot.o  electron_all.o shm.o    pardens.o  paircorrection.o \
         optics.o   constr_cell_relax.o   stm.o    finite_diff.o elpol.o    \
         hamil_lr.o rmm-diis_lr.o  subrot_cluster.o subrot_lr.o \
         lr_helper.o hamil_lrf.o   elinear_response.o ilinear_response.o \
         linear_optics.o linear_response.o   \
         setlocalpp.o  wannier.o electron_OEP.o electron_lhf.o twoelectron4o.o \
         ratpol.o screened_2e.o wave_cacher.o chi_base.o wpot.o local_field.o \
         ump2.o bse.o acfdt.o chi.o sydmat.o 

INC =

vasp: $(SOURCE) $(FFT_OBJS) $(INC) main.o 
        rm -f vasp
        $(FCL) -o vasp main.o  $(SOURCE) $(LPD_OBJS) $(FFT_OBJS) $(LIBS) $(LINK)
makeparam: $(SOURCE) $(FFT_OBJS) makeparam.o main.F $(INC)
        $(FCL) -o makeparam  $(LINK) makeparam.o $(SOURCE) $(FFT_OBJS) $(LIBS)
zgemmtest: zgemmtest.o base.o random.o $(INC)
        $(FCL) -o zgemmtest $(LINK) zgemmtest.o random.o base.o $(LIBS)
dgemmtest: dgemmtest.o base.o random.o $(INC)
        $(FCL) -o dgemmtest $(LINK) dgemmtest.o random.o base.o $(LIBS) 
ffttest: base.o smart_allocate.o mpi.o mgrid.o random.o ffttest.o $(FFT_OBJS) $(INC)
        $(FCL) -o ffttest $(LINK) ffttest.o mpi.o mgrid.o random.o smart_allocate.o base.o $(FFT_OBJS) $(LIBS)
kpoints: $(SOURCE) $(FFT_OBJS) makekpoints.o main.F $(INC)
        $(FCL) -o kpoints $(LINK) makekpoints.o $(SOURCE) $(FFT_OBJS) $(LIBS)

clean:
        -rm -f *.g *.f *.f90 *.o *.L *.mod ; touch *.F

main.o: main$(SUFFIX)
        $(FC) $(FFLAGS)$(DEBUG)  $(INCS) -c main$(SUFFIX)
xcgrad.o: xcgrad$(SUFFIX)
        $(FC) $(FFLAGS) $(INLINE)  $(INCS) -c xcgrad$(SUFFIX)
xcspin.o: xcspin$(SUFFIX)
        $(FC) $(FFLAGS) $(INLINE)  $(INCS) -c xcspin$(SUFFIX)

makeparam.o: makeparam$(SUFFIX)
        $(FC) $(FFLAGS)$(DEBUG)  $(INCS) -c makeparam$(SUFFIX)

makeparam$(SUFFIX): makeparam.F main.F 
#
base.o: base.inc base.F
mgrid.o: mgrid.inc mgrid.F
constant.o: constant.inc constant.F
lattice.o: lattice.inc lattice.F
setex.o: setexm.inc setex.F
pseudo.o: pseudo.inc pseudo.F
poscar.o: poscar.inc poscar.F
mkpoints.o: mkpoints.inc mkpoints.F
wave.o: wave.inc wave.F
nonl.o: nonl.inc nonl.F
nonlr.o: nonlr.inc nonlr.F

$(OBJ_HIGH):
        $(CPP)
        $(FC) $(FFLAGS) $(OFLAG_HIGH) $(INCS) -c $*$(SUFFIX)
$(OBJ_NOOPT):
        $(CPP)
        $(FC) $(FFLAGS) $(INCS) -c $*$(SUFFIX)

fft3dlib_f77.o: fft3dlib_f77.F
        $(CPP)
        $(F77) $(FFLAGS_F77) -c $*$(SUFFIX)

.F.o:
        $(CPP)
        $(FC) $(FFLAGS) $(OFLAG) $(INCS) -c $*$(SUFFIX)
.F$(SUFFIX):
        $(CPP)
$(SUFFIX).o:
        $(FC) $(FFLAGS) $(OFLAG) $(INCS) -c $*$(SUFFIX)

# special rules
#-----------------------------------------------------------------------

fft3dlib.o : fft3dlib.F
        $(CPP)
        $(FC) -free -lowercase -O2 -c $*$(SUFFIX)
fft3dfurth.o : fft3dfurth.F
        $(CPP)
        $(FC) -free -lowercase -O1 -c $*$(SUFFIX)

radial.o : radial.F
        $(CPP)
        $(FC) -free -lowercase -O1 -c $*$(SUFFIX)

symlib.o : symlib.F
        $(CPP)
        $(FC) -free -lowercase -O1 -c $*$(SUFFIX)

symmetry.o : symmetry.F
        $(CPP)
        $(FC) -free -lowercase -O1 -c $*$(SUFFIX)

wave_mpi.o : wave_mpi.F
        $(CPP)
        $(FC) -free -lowercase -O1 -c $*$(SUFFIX)

wave.o : wave.F
        $(CPP)
        $(FC) -free -lowercase -O1 -c $*$(SUFFIX)

dynbr.o : dynbr.F
        $(CPP)
        $(FC) -free -lowercase -O1 -c $*$(SUFFIX)

asa.o : asa.F
        $(CPP)
        $(FC) -free -lowercase -O1 -c $*$(SUFFIX)

broyden.o : broyden.F
        $(CPP)
        $(FC) -free -lowercase -O2 -c $*$(SUFFIX)

us.o : us.F
        $(CPP)
        $(FC) -free -lowercase -O1 -c $*$(SUFFIX)

LDApU.o : LDApU.F
        $(CPP)
        $(FC) -free -lowercase -O2 -c $*$(SUFFIX)

Sample PBS Submission Script

The sample PBS script can now be found on the PBS Sample Job Scripts page.

Retrieved from "http://wiki.hpc.ufl.edu/index.php/VASP"

VASP

From UF HPC Wiki

Contents

Introduction

Installation

VASP with MVAPICH (10/17/2008)

VASP 5.2.0 (08/24/2009)

Notes

Libraries

Makefile

Sample PBS Submission Script

Views

Personal tools

Navigation

Search

Toolbox