compiling with HP MPI & Open MPI

[Axel]

On Jul 7, 10:04 am, "C. Va'rnai" <cv... at cam.ac.uk> wrote:
> Dear All,

dear csilla,

> I am trying to compile CP2K on a supercomputer, and this is my first
> experience with supercomputers, I have used clusters so far.
> (http://www.rz.uni-karlsruhe.de/ssck/hpxc)
>
> The default mpif90 compiler is HP MPI. When I compile CP2K with HP MPI,
> after a nice compilation without error message (I know, it doesn't mean
> that it should run fine), it gives segmentation fault. I don't really know
> much about HP MPI, and after fiddling around with the compiler options, I
> decided to use Open MPI I use on a cluster machine with the same
> Linux-x86-64-intel architecture. Running CP2K in interactive mode was

please note, that the machine is _not_ an x86_64 machine but an
itanium2 (aka. IA64) machine.

using OpenMPI needs a lot of care, since you have to make sure that
you have working support for the quadrics network.
building on such an unusual machine can be very tricky, thus
i strongly suggest to get into contact with the user support
staff and have them give you explicit instructions on how to
use/link scalapack/blacs/mpi etc.

please also make sure that you do _not_ compile for OpenMP
and link with the multithreaded MKL unless you are 100% certain
you know what you are doing. both is explained in the intel
compiler and MKL manuals (it may need a little digging).

cheers,
   axel.

> successful. However, I couldn't get CP2K run properly using the batch
> system. For the command
>
> job_submit -t 10 -T 60 -m 500 -p 4/1 -c d
> "/home/ssck/wkhu00f/wkhu23e/openmpi/bin/mpirun -np 4 cp2k.popt H2O.inp"
>
> I got the following error message:
> cannot allocate memory for thread-local data: ABORT
> cannot allocate memory for thread-local data: ABORT
> cannot allocate memory for thread-local data: ABORT
> cannot allocate memory for thread-local data: ABORT
>
> When I decreased the requested memory from 500 to 300 MegaBytes, running
>
> job_submit -t 10 -T 60 -m 300 -p 4/1 -c d
> "/home/ssck/wkhu00f/wkhu23e/openmpi/bin/mpirun -np 4 cp2k.popt H2O.inp"
>
> it got stuck when opening the BASIS_MOLOPT file:
>
>  GENERATE| Preliminary Number of Bonds generated: 0
>  GENERATE|  Achieved consistency in connectivity generation.
>  GENERATE| Number of Bonds generated: 0
>  GENERATE| Preliminary Number of Bends generated: 0
>  GENERATE| Number of Bends generated: 0
>  GENERATE| Number of UB generated: 0
>  GENERATE| Preliminary Number of Torsions generated: 0
>  GENERATE| Number of Torsions generated: 0
>  GENERATE| Number of Impropers generated: 0
>  GENERATE| Number of 1-4 interactions generated: 0
>  CP2K: An error occurred opening the file <BASIS_MOLOPT> with the unit
> number
>     1 (IOSTAT =     41)
>
> I would be glad if you had any idea what is wrong with the HP MPI
> compilation or what I am doing wrong with the OPen MPI compiled version.
> Thanks,
>
> Csilla
>
> The test file is the H2O.inp in the tests/QS/ directory.
> Here is my arch file for the Open MPI run:
>
> INTEL_INC=/opt/intel/mkl/10.0.011/include
> INTEL_LIB=/opt/intel/mkl/10.0.011/lib/em64t
>
> FFTW3_INC=/software/all/fftw/include
> FFTW3_LIB=/software/all/fftw/lib
>
> ACML_INC=/software/all/acml/acml4.0/ifort64_mp/include
> ACML_LIB=/software/all/acml/acml4.0/ifort64_mp/lib
>
> CC = icc CPP = FC = mpif90 LD = mpif90 AR = ar -r DFLAGS = -D__INTEL
> -D__FFTSG -D__parallel -D__BLACS -D__SCALAPACK -D__FFTW3 -D__FFTACML
> CPPFLAGS = -C -traditional $(DFLAGS) -I$(INTEL_INC) #-I$(FFTW3_INC)
> -I$(ACML_INC) FCFLAGS = $(DFLAGS) -I$(INTEL_INC) -I$(FFTW3_INC)
> -I$(ACML_INC) -O0 -openmp -xW -heap-arrays 64 -fpp -free LDFLAGS =
> $(FCFLAGS) LIBS = -L$(INTEL_LIB) -L$(FFTW3_LIB) -L$(ACML_LIB)
> -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -lmkl_intel_lp64
> -lmkl_sequential -lmkl_core -lfftw3 -lacml_mp -lpthread
> #$(ACML_LIB)/libacml_mp.a
>
> OBJECTS_ARCHITECTURE = machine_intel.o
>
> The one for HP MPI differs only in linking -lmkl_blacs_lp64 instead of the
> openmpi version.