[CP2K-user] Are there any clear instructions for building cuda-enabled cp2k with MPI that work?

[David Gunter]

I've scoured the forum and I've found many old, outdated hints for building 
a cuda-enabled CP2k with MPI enabled I've also found several web links with 
instructions that also fail.

So I've taken the most recent advice to some of my posts and here is what I 
have been able to do. It results in a cp2k.pdbg or a cp2k.psmp executable 
that segfaults upon launch. I'm at a loss for how to proceed.

1) I start by building the third party tools and libs with the toolchain 
script. I'm using the Intel 18.0.5 compiler suite, MKL 2018.0.4, and 
OpenMPI 3.1.2 built with the Intel compiler.

./install_cp2k_toolchain.sh \
  --with-cmake=system \
  --with-openblas=install \
  --mpi-mode=openmpi \
  --with-reflapack=no \
  --with-sirius=no \
  --enable-cuda \
  --gpu-ver=P100 \
  --enable-omp

This builds all the packages, generates the setup file and the arch files.

2) Next, I copy the local_cuda.psmp arch file to cp2k/arch/intel_cuda.psmp.
    I then copy "setup" to cp2k/experimental_intel_cuda_setup

I edit that arch file to add in specifics left out by toolchain. Here is 
how that file looks when I'm done.

(dog at ko-fe1 87%) vi experimental_intel_cuda.psmp 

CC          = icc
CXX        = icpc
CPP        =
AR          = ar -r
FC          = mpif90
LD          = mpif90
#
DFLAGS      = -D__ACC -D__DBCSR_ACC -D__PW_CUDA  -D__LIBXSMM  -D__parallel  
-D__FFTW3  -D__LIBINT -D__LIBINT_MAX_AM=6 -D__LIBDERIV_MAX_AM1=5 -D__LIBXC 
-D__SCALAPACK -D__ELPA=201811 -D__GSL -D__SPGLIB -D__HDF5 -D__LIBVDWXC 
-D__JSON
#
WFLAGS      =
#
FCDEBFLAGS  =
CFLAGS      =    $(PROFOPT) -fopenmp 
-I/usr/projects/hpcsoft/toss3/kodiak/openmpi/3.1.2-intel-18.0.5/include 
-pthread  
-I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/openblas-0.3.6/include' 
-I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/fftw-3.3.8/include' 
-I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libint-1.1.6/include' 
-I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libxc-4.3.4/include' 
-I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libxsmm-1.10.0/include' 
-I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/elpa-2018.11.001/include/elpa_openmp-2018.11.001/modules' 
-I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/elpa-2018.11.001/include/elpa_openmp-2018.11.001/elpa' 
-I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/gsl-2.5/include' 
-I/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/spglib-1.12.2/include 
-I/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/hdf5-1.10.4/include 
-I/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libvdwxc-0.4.0/include 
-I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/json_fortran-7.0.0/include' 
$(DFLAGS)
FCFLAGS     =    $(PROFOPT) -fopenmp 
-I/usr/projects/hpcsoft/toss3/kodiak/openmpi/3.1.2-intel-18.0.5/include 
-pthread  
-I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/openblas-0.3.6/include' 
-I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/fftw-3.3.8/include' 
-I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libint-1.1.6/include' 
-I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libxc-4.3.4/include' 
-I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libxsmm-1.10.0/include' 
-I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/elpa-2018.11.001/include/elpa_openmp-2018.11.001/modules' 
-I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/elpa-2018.11.001/include/elpa_openmp-2018.11.001/elpa' 
-I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/gsl-2.5/include' 
-I/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/spglib-1.12.2/include 
-I/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/hdf5-1.10.4/include 
-I/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libvdwxc-0.4.0/include 
-I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/json_fortran-7.0.0/include' 
$(FCDEBFLAGS) $(WFLAGS) $(DFLAGS)
LDFLAGS     = $(FCFLAGS) -Wl,--enable-new-dtags -pthread -Wl,-rpath 
-Wl,/usr/projects/hpcsoft/toss3/kodiak/openmpi/3.1.2-intel-18.0.5/lib 
-L/usr/projects/hpcsoft/toss3/kodiak/openmpi/3.1.2-intel-18.0.5/lib  
 -L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/openblas-0.3.6/lib' 
-Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/openblas-0.3.6/lib' 
-L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/fftw-3.3.8/lib' 
-Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/fftw-3.3.8/lib' 
-L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libint-1.1.6/lib' 
-Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libint-1.1.6/lib' 
-L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libxc-4.3.4/lib' 
-Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libxc-4.3.4/lib' 
-L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libxsmm-1.10.0/lib' 
-Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libxsmm-1.10.0/lib' 
-L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/scalapack-2.0.2/lib' 
-Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/scalapack-2.0.2/lib' 
-L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/elpa-2018.11.001/lib' 
-Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/elpa-2018.11.001/lib' 
-L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/gsl-2.5/lib' 
-Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/gsl-2.5/lib' 
-L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/spglib-1.12.2/lib' 
-Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/spglib-1.12.2/lib' 
-L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/hdf5-1.10.4/lib' 
-Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/hdf5-1.10.4/lib' 
-L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libvdwxc-0.4.0/lib' 
-Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libvdwxc-0.4.0/lib' 
-L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/json_fortran-7.0.0/lib' 
-Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/json_fortran-7.0.0/lib' 
-L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/openblas-0.3.6/lib' 
-Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/openblas-0.3.6/lib' 
-L'/var/lib/perceus/vnfs/fe/rootfs/opt/cudatoolkit/10.0/targets/x86_64-linux/lib' 
-Wl,-rpath='/var/lib/perceus/vnfs/fe/rootfs/opt/cudatoolkit/10.0/targets/x86_64-linux/lib' 
-L'/usr/lib64' -Wl,-rpath='/usr/lib64'
LDFLAGS_C = $(FCFLAGS) -static-intel -nofor_main
LIBS        = -ljsonfortran -lvdwxc -lhdf5 -lhdf5_hl -lsymspg -lgsl 
-lgslcblas -lelpa_openmp -lscalapack -lxsmmf -lxsmm -ldl -lpthread -lxcf03 
-lxc -Wl,-Bstatic -lderiv -lint -Wl,-Bdynamic -lfftw3_mpi -lfftw3 
-lfftw3_omp   -lmpi  -lopenblas_omp -lstdc++ -lcudart -lnvrtc -lcuda 
-lcufft -lcublas -lrt
#
CXX         = 
/usr/projects/hpcsoft/toss3/common/intel-clusterstudio/2018.4.057/compilers_and_libraries_2018/linux/bin/intel64/icc
CXXFLAGS    = -O2 -ftree-vectorize -g -fno-omit-frame-pointer -march=native 
-I${CUDA_PATH}/include -std=c++11
GPUVER      = P100
NVCC        = nvcc -D__GNUC__=4 -D__GNUC_MINOR__=9
#NVFLAGS     = -arch sm_60 -O3 -Xcompiler='-fopenmp' --std=c++11 $(DFLAGS)
NVFLAGS     = -arch sm_60 -O3 -g -w --std=c++11 $(DFLAGS)
#
FYPPFLAGS   = -n --line-marker-format=gfortran5

3) Finally, I build the cp2k binary. It all goes without error.

$ 
$ make -j 8 ARCH="experimental_intel_cuda" VERSION="psmp"

4) I attempt to run the executable by copying it to a runtime directory 
containing H2)-32.inp
    $ cd /luster/dog/runs
    $ . ~/cp2k/experimental_intel_cuda_setup
    $ mpirun -n 4 ./cp2k.psmp -i H2O-32.inp

App launch reported: 2 (out of 2) daemons - 0 (out of 4) procs
 HINT in environment.F:804 The compiler target flags used to build this 
binary a
 re insufficiently exploiting the extensions which are available for this 
CPU mo
 del.
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source  
           
cp2k.psmp          0000000004A2596E  for__signal_handl     Unknown  Unknown
libc-2.17.so       00002B0B8D0D4280  Unknown               Unknown  Unknown
cp2k.psmp          0000000002FEE2EB  Unknown               Unknown  Unknown
cp2k.psmp          0000000002848FD8  Unknown               Unknown  Unknown
cp2k.psmp          000000000299DB9C  Unknown               Unknown  Unknown
cp2k.psmp          000000000089B34B  Unknown               Unknown  Unknown
cp2k.psmp          0000000000895E21  Unknown               Unknown  Unknown
cp2k.psmp          00000000013E72C8  Unknown               Unknown  Unknown
cp2k.psmp          00000000013DDD94  Unknown               Unknown  Unknown
cp2k.psmp          0000000001121BCD  Unknown               Unknown  Unknown
cp2k.psmp          0000000000DA9518  Unknown               Unknown  Unknown
cp2k.psmp          0000000000DA7961  Unknown               Unknown  Unknown
cp2k.psmp          00000000007B4FA5  Unknown               Unknown  Unknown
cp2k.psmp          0000000000513751  Unknown               Unknown  Unknown
cp2k.psmp          0000000000512C14  Unknown               Unknown  Unknown
cp2k.psmp          00000000004286DD  Unknown               Unknown  Unknown
cp2k.psmp          000000000041EF28  Unknown               Unknown  Unknown
cp2k.psmp          000000000041DB7E  Unknown               Unknown  Unknown
libc-2.17.so       00002B0B8D0C03D5  __libc_start_main     Unknown  Unknown
cp2k.psmp          000000000041DA72  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source  
           
cp2k.psmp          0000000004A2596E  for__signal_handl     Unknown  Unknown
libc-2.17.so       00002B4EEB651280  Unknown               Unknown  Unknown
cp2k.psmp          0000000002848FE3  Unknown               Unknown  Unknown
cp2k.psmp          000000000299DB9C  Unknown               Unknown  Unknown
cp2k.psmp          000000000089B34B  Unknown               Unknown  Unknown
cp2k.psmp          0000000000895E21  Unknown               Unknown  Unknown
cp2k.psmp          00000000013E72C8  Unknown               Unknown  Unknown
cp2k.psmp          00000000013DDD94  Unknown               Unknown  Unknown
cp2k.psmp          0000000001121BCD  Unknown               Unknown  Unknown
cp2k.psmp          0000000000DA9518  Unknown               Unknown  Unknown
cp2k.psmp          0000000000DA7961  Unknown               Unknown  Unknown
cp2k.psmp          00000000007B4FA5  Unknown               Unknown  Unknown
cp2k.psmp          0000000000513751  Unknown               Unknown  Unknown
cp2k.psmp          0000000000512C14  Unknown               Unknown  Unknown
cp2k.psmp          00000000004286DD  Unknown               Unknown  Unknown
cp2k.psmp          000000000041EF28  Unknown               Unknown  Unknown
cp2k.psmp          000000000041DB7E  Unknown               Unknown  Unknown
libc-2.17.so       00002B4EEB63D3D5  __libc_start_main     Unknown  Unknown
cp2k.psmp          000000000041DA72  Unknown               Unknown  Unknown
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source  
           
cp2k.psmp          0000000004A2596E  for__signal_handl     Unknown  Unknown
libc-2.17.so       00002B443B3B3280  Unknown               Unknown  Unknown
cp2k.psmp          0000000002FEE2A2  Unknown               Unknown  Unknown
cp2k.psmp          000000000299C076  Unknown               Unknown  Unknown
cp2k.psmp          0000000002848FD8  Unknown               Unknown  Unknown
cp2k.psmp          000000000299DB9C  Unknown               Unknown  Unknown
cp2k.psmp          000000000089B34B  Unknown               Unknown  Unknown
cp2k.psmp          0000000000895E21  Unknown               Unknown  Unknown
cp2k.psmp          00000000013E72C8  Unknown               Unknown  Unknown
cp2k.psmp          00000000013DDD94  Unknown               Unknown  Unknown
cp2k.psmp          0000000001121BCD  Unknown               Unknown  Unknown
cp2k.psmp          0000000000DA9518  Unknown               Unknown  Unknown
cp2k.psmp          0000000000DA7961  Unknown               Unknown  Unknown
cp2k.psmp          00000000007B4FA5  Unknown               Unknown  Unknown
cp2k.psmp          0000000000513751  Unknown               Unknown  Unknown
cp2k.psmp          0000000000512C14  Unknown               Unknown  Unknown
cp2k.psmp          00000000004286DD  Unknown               Unknown  Unknown
cp2k.psmp          000000000041EF28  Unknown               Unknown  Unknown
cp2k.psmp          000000000041DB7E  Unknown               Unknown  Unknown
libc-2.17.so       00002B443B39F3D5  __libc_start_main     Unknown  Unknown
cp2k.psmp          000000000041DA72  Unknown               Unknown  Unknown


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.cp2k.org/archives/cp2k-user/attachments/20190517/8f622558/attachment.htm>