Trying to compile cp2k+cuda+intel_mpi in parallel-smp mode

[Alexey Nesternko]

Good morning!

Compiling cp2k on the supercomputer I've formed the best arch-file that I 
was able to do:

MKL_ROOT  = /opt/intel/composer_xe_2013.2.146/mkl
MKL_LIB = $(MKL_ROOT)/lib/intel64
MKL_INCLUDE = $(MKL_ROOT)/include/fftw
XC_LIB = $(HOME)/local/libxc-202i/lib
XC_INC = $(HOME)/local/libxc-202i/include
LI_LIB = $(HOME)/local/libint-114i/lib
LI_INC = $(HOME)/local/libint-114i/include
CU_LIB = /opt/cuda-5.0/lib64
NVCC     = /opt/cuda-5.0/bin/nvcc
CC       = mpiicc -mt_mpi -static_mpi -openmp
CPP      = 
FC       = mpiifort -mt_mpi -FR 
LD       = mpiifort -mt_mpi -i-static -static_mpi -openmp
AR       = xiar -r
DFLAGS   = -D__INTEL -D__FFTSG -D__FFTW3 -D__FFTMKL -D__LIBXC2 -D__parallel 
-D__PW_CUDA -D__DBCSR_CUDA -D__HAS_NO_CUDA_SM30 -D__BLACS -D__SCALAPACK
NVFLAGS  = $(DFLAGS) -arch=sm_20
CPPFLAGS = -C -traditional $(DFLAGS) -I$(MKL_INCLUDE)
FCFLAGS  = $(DFLAGS) -I$(MKL_INCLUDE) -O3 -xHost -heap-arrays 64 
-funroll-loops -fpp -free -openmp
FCFLAGS2 = $(DFLAGS) -O1 -xHost -heap-arrays 64 -fpp -free -openmp
LDFLAGS  = $(FCFLAGS) -L$(CU_LIB) -L$(LI_LIB) -L$(XC_LIB)
LIBS     = -Wl,--start-group \
$(MKL_LIB)/libmkl_scalapack_lp64.a \
$(MKL_LIB)/libmkl_blacs_intelmpi_lp64.a \
$(MKL_LIB)/libmkl_intel_lp64.a \
$(MKL_LIB)/libfftw3xf_intel.a \
$(MKL_LIB)/libmkl_sequential.a \
$(MKL_LIB)/libmkl_core.a \
-Wl,--end-group \
$(XC_LIB)/libxc.a \
$(LI_LIB)/libint.a \
$(LI_LIB)/libderiv.a \
$(CU_LIB)/libcudart.so \
$(CU_LIB)/libcufft.so \
$(CU_LIB)/libcublas.so \
-lrt -lpthread 
OBJECTS_ARCHITECTURE = machine_intel.o
graphcon.o: graphcon.F
    $(FC) -c $(FCFLAGS2) $<


But when I start the test calculation with 

export I_MPI_PIN_DOMAIN=omp
export OMP_NUM_THREADS=4
export MKL_SERIAL=yes

and   'PREFERRED_FFT_LIBRARY FFTSG' set in the input file ... than I always 
obtain SEGFAULT with glibc found 'double memory free or corruption'. For 
instance:

*** glibc detected *** 
/mnt/data/users/dm4/vol12/irinairbis_251473/_scratch/bin/cp2k-240.impi410.cuda.fftw.mt: 
double free or corruption (fasttop): 0x0000000004ffdc20 ***
*** glibc detected *** 
/mnt/data/users/dm4/vol12/irinairbis_251473/_scratch/bin/cp2k-240.impi410.cuda.fftw.mt: 
double free or corruption (fasttop): 0x0000000004ff8750 ***

 And this mistake is only in OPENMP mode, when compiled with flag -mt_mpi. 
When compiled without -mt_mpi, than all is OK, but OMP_NUM_THREADS doesn't 
work. When I recompile it with fftw (instead of mkl fftw wrapper) the 
mistake doesn't disapear....

In the another machine I've built cp2k with openmpi+openmp+mkl+fftw+cuda 
and everything was allright. But thereis no openmpi supporting openmp on 
the machine and so I had to solve intel-mkl problems..  

Does anybody have any ideas about it? 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.cp2k.org/archives/cp2k-user/attachments/20131104/02ca5b23/attachment.htm>