<div dir="ltr">I've scoured the forum and I've found many old, outdated hints for building a cuda-enabled CP2k with MPI enabled I've also found several web links with instructions that also fail.<div><br></div><div>So I've taken the most recent advice to some of my posts and here is what I have been able to do. It results in a cp2k.pdbg or a cp2k.psmp executable that segfaults upon launch. I'm at a loss for how to proceed.</div><div><br></div><div>1) I start by building the third party tools and libs with the toolchain script. I'm using the Intel 18.0.5 compiler suite, MKL 2018.0.4, and OpenMPI 3.1.2 built with the Intel compiler.</div><div><br></div><div><div>./install_cp2k_toolchain.sh \</div><div>  --with-cmake=system \</div><div>  --with-openblas=install \</div><div>  --mpi-mode=openmpi \</div><div>  --with-reflapack=no \</div><div>  --with-sirius=no \</div><div>  --enable-cuda \</div><div>  --gpu-ver=P100 \</div><div>  --enable-omp</div></div><div><br></div><div>This builds all the packages, generates the setup file and the arch files.</div><div><br></div><div>2) Next, I copy the local_cuda.psmp arch file to cp2k/arch/intel_cuda.psmp.</div><div>    I then copy "setup" to cp2k/experimental_intel_cuda_setup</div><div><br></div><div>I edit that arch file to add in specifics left out by toolchain. Here is how that file looks when I'm done.</div><div><br></div><div><div>(dog@ko-fe1 87%) vi experimental_intel_cuda.psmp </div><div><br></div><div>CC          = icc</div><div>CXX        = icpc</div><div>CPP        =</div><div>AR          = ar -r</div><div>FC          = mpif90</div><div>LD          = mpif90</div><div>#</div><div>DFLAGS      = -D__ACC -D__DBCSR_ACC -D__PW_CUDA  -D__LIBXSMM  -D__parallel  -D__FFTW3  -D__LIBINT -D__LIBINT_MAX_AM=6 -D__LIBDERIV_MAX_AM1=5 -D__LIBXC -D__SCALAPACK -D__ELPA=201811 -D__GSL -D__SPGLIB -D__HDF5 -D__LIBVDWXC -D__JSON</div><div>#</div><div>WFLAGS      =</div><div>#</div><div>FCDEBFLAGS  =</div><div>CFLAGS      =    $(PROFOPT) -fopenmp -I/usr/projects/hpcsoft/toss3/kodiak/openmpi/3.1.2-intel-18.0.5/include -pthread  -I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/openblas-0.3.6/include' -I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/fftw-3.3.8/include' -I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libint-1.1.6/include' -I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libxc-4.3.4/include' -I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libxsmm-1.10.0/include' -I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/elpa-2018.11.001/include/elpa_openmp-2018.11.001/modules' -I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/elpa-2018.11.001/include/elpa_openmp-2018.11.001/elpa' -I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/gsl-2.5/include' -I/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/spglib-1.12.2/include -I/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/hdf5-1.10.4/include -I/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libvdwxc-0.4.0/include -I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/json_fortran-7.0.0/include' $(DFLAGS)</div><div>FCFLAGS     =    $(PROFOPT) -fopenmp -I/usr/projects/hpcsoft/toss3/kodiak/openmpi/3.1.2-intel-18.0.5/include -pthread  -I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/openblas-0.3.6/include' -I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/fftw-3.3.8/include' -I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libint-1.1.6/include' -I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libxc-4.3.4/include' -I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libxsmm-1.10.0/include' -I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/elpa-2018.11.001/include/elpa_openmp-2018.11.001/modules' -I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/elpa-2018.11.001/include/elpa_openmp-2018.11.001/elpa' -I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/gsl-2.5/include' -I/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/spglib-1.12.2/include -I/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/hdf5-1.10.4/include -I/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libvdwxc-0.4.0/include -I'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/json_fortran-7.0.0/include' $(FCDEBFLAGS) $(WFLAGS) $(DFLAGS)<br><div>LDFLAGS     = $(FCFLAGS) -Wl,--enable-new-dtags -pthread -Wl,-rpath -Wl,/usr/projects/hpcsoft/toss3/kodiak/openmpi/3.1.2-intel-18.0.5/lib -L/usr/projects/hpcsoft/toss3/kodiak/openmpi/3.1.2-intel-18.0.5/lib   -L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/openblas-0.3.6/lib' -Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/openblas-0.3.6/lib' -L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/fftw-3.3.8/lib' -Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/fftw-3.3.8/lib' -L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libint-1.1.6/lib' -Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libint-1.1.6/lib' -L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libxc-4.3.4/lib' -Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libxc-4.3.4/lib' -L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libxsmm-1.10.0/lib' -Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libxsmm-1.10.0/lib' -L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/scalapack-2.0.2/lib' -Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/scalapack-2.0.2/lib' -L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/elpa-2018.11.001/lib' -Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/elpa-2018.11.001/lib' -L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/gsl-2.5/lib' -Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/gsl-2.5/lib' -L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/spglib-1.12.2/lib' -Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/spglib-1.12.2/lib' -L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/hdf5-1.10.4/lib' -Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/hdf5-1.10.4/lib' -L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libvdwxc-0.4.0/lib' -Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/libvdwxc-0.4.0/lib' -L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/json_fortran-7.0.0/lib' -Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/json_fortran-7.0.0/lib' -L'/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/openblas-0.3.6/lib' -Wl,-rpath='/usr/projects/icapt/dog/benchmarking/cp2k/tools/toolchain/install/openblas-0.3.6/lib' -L'/var/lib/perceus/vnfs/fe/rootfs/opt/cudatoolkit/10.0/targets/x86_64-linux/lib' -Wl,-rpath='/var/lib/perceus/vnfs/fe/rootfs/opt/cudatoolkit/10.0/targets/x86_64-linux/lib' -L'/usr/lib64' -Wl,-rpath='/usr/lib64'</div><div>LDFLAGS_C = $(FCFLAGS) -static-intel -nofor_main</div><div>LIBS        = -ljsonfortran -lvdwxc -lhdf5 -lhdf5_hl -lsymspg -lgsl -lgslcblas -lelpa_openmp -lscalapack -lxsmmf -lxsmm -ldl -lpthread -lxcf03 -lxc -Wl,-Bstatic -lderiv -lint -Wl,-Bdynamic -lfftw3_mpi -lfftw3 -lfftw3_omp   -lmpi  -lopenblas_omp -lstdc++ -lcudart -lnvrtc -lcuda -lcufft -lcublas -lrt</div><div>#</div><div>CXX         = /usr/projects/hpcsoft/toss3/common/intel-clusterstudio/2018.4.057/compilers_and_libraries_2018/linux/bin/intel64/icc</div><div>CXXFLAGS    = -O2 -ftree-vectorize -g -fno-omit-frame-pointer -march=native -I${CUDA_PATH}/include -std=c++11</div><div>GPUVER      = P100</div><div>NVCC        = nvcc -D__GNUC__=4 -D__GNUC_MINOR__=9</div><div>#NVFLAGS     = -arch sm_60 -O3 -Xcompiler='-fopenmp' --std=c++11 $(DFLAGS)</div><div>NVFLAGS     = -arch sm_60 -O3 -g -w --std=c++11 $(DFLAGS)</div><div>#</div><div>FYPPFLAGS   = -n --line-marker-format=gfortran5</div></div></div><div><br></div><div>3) Finally, I build the cp2k binary. It all goes without error.</div><div><br></div><div>$ </div><div>$ make -j 8 ARCH="experimental_intel_cuda" VERSION="psmp"</div><div><br></div><div>4) I attempt to run the executable by copying it to a runtime directory containing H2)-32.inp</div><div>    $ cd /luster/dog/runs</div><div>    $ . ~/cp2k/experimental_intel_cuda_setup</div><div>    $ mpirun -n 4 ./cp2k.psmp -i H2O-32.inp</div><div><br></div><div><div>App launch reported: 2 (out of 2) daemons - 0 (out of 4) procs</div><div> HINT in environment.F:804 The compiler target flags used to build this binary a</div><div> re insufficiently exploiting the extensions which are available for this CPU mo</div><div> del.</div><div>--------------------------------------------------------------------------</div><div>MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD</div><div>with errorcode 1.</div><div><br></div><div>NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.</div><div>You may or may not see output from other processes, depending on</div><div>exactly when Open MPI kills them.</div><div>--------------------------------------------------------------------------</div><div>forrtl: error (78): process killed (SIGTERM)</div><div>Image              PC                Routine            Line        Source             </div><div>cp2k.psmp          0000000004A2596E  for__signal_handl     Unknown  Unknown</div><div>libc-2.17.so       00002B0B8D0D4280  Unknown               Unknown  Unknown</div><div>cp2k.psmp          0000000002FEE2EB  Unknown               Unknown  Unknown</div><div>cp2k.psmp          0000000002848FD8  Unknown               Unknown  Unknown</div><div>cp2k.psmp          000000000299DB9C  Unknown               Unknown  Unknown</div><div>cp2k.psmp          000000000089B34B  Unknown               Unknown  Unknown</div><div>cp2k.psmp          0000000000895E21  Unknown               Unknown  Unknown</div><div>cp2k.psmp          00000000013E72C8  Unknown               Unknown  Unknown</div><div>cp2k.psmp          00000000013DDD94  Unknown               Unknown  Unknown</div><div>cp2k.psmp          0000000001121BCD  Unknown               Unknown  Unknown</div><div>cp2k.psmp          0000000000DA9518  Unknown               Unknown  Unknown</div><div>cp2k.psmp          0000000000DA7961  Unknown               Unknown  Unknown</div><div>cp2k.psmp          00000000007B4FA5  Unknown               Unknown  Unknown</div><div>cp2k.psmp          0000000000513751  Unknown               Unknown  Unknown</div><div>cp2k.psmp          0000000000512C14  Unknown               Unknown  Unknown</div><div>cp2k.psmp          00000000004286DD  Unknown               Unknown  Unknown</div><div>cp2k.psmp          000000000041EF28  Unknown               Unknown  Unknown</div><div>cp2k.psmp          000000000041DB7E  Unknown               Unknown  Unknown</div><div>libc-2.17.so       00002B0B8D0C03D5  __libc_start_main     Unknown  Unknown</div><div>cp2k.psmp          000000000041DA72  Unknown               Unknown  Unknown</div><div>forrtl: error (78): process killed (SIGTERM)</div><div>Image              PC                Routine            Line        Source             </div><div>cp2k.psmp          0000000004A2596E  for__signal_handl     Unknown  Unknown</div><div>libc-2.17.so       00002B4EEB651280  Unknown               Unknown  Unknown</div><div>cp2k.psmp          0000000002848FE3  Unknown               Unknown  Unknown</div><div>cp2k.psmp          000000000299DB9C  Unknown               Unknown  Unknown</div><div>cp2k.psmp          000000000089B34B  Unknown               Unknown  Unknown</div><div>cp2k.psmp          0000000000895E21  Unknown               Unknown  Unknown</div><div>cp2k.psmp          00000000013E72C8  Unknown               Unknown  Unknown</div><div>cp2k.psmp          00000000013DDD94  Unknown               Unknown  Unknown</div><div>cp2k.psmp          0000000001121BCD  Unknown               Unknown  Unknown</div><div>cp2k.psmp          0000000000DA9518  Unknown               Unknown  Unknown</div><div>cp2k.psmp          0000000000DA7961  Unknown               Unknown  Unknown</div><div>cp2k.psmp          00000000007B4FA5  Unknown               Unknown  Unknown</div><div>cp2k.psmp          0000000000513751  Unknown               Unknown  Unknown</div><div>cp2k.psmp          0000000000512C14  Unknown               Unknown  Unknown</div><div>cp2k.psmp          00000000004286DD  Unknown               Unknown  Unknown</div><div>cp2k.psmp          000000000041EF28  Unknown               Unknown  Unknown</div><div>cp2k.psmp          000000000041DB7E  Unknown               Unknown  Unknown</div><div>libc-2.17.so       00002B4EEB63D3D5  __libc_start_main     Unknown  Unknown</div><div>cp2k.psmp          000000000041DA72  Unknown               Unknown  Unknown</div><div>forrtl: error (78): process killed (SIGTERM)</div><div>Image              PC                Routine            Line        Source             </div><div>cp2k.psmp          0000000004A2596E  for__signal_handl     Unknown  Unknown</div><div>libc-2.17.so       00002B443B3B3280  Unknown               Unknown  Unknown</div><div>cp2k.psmp          0000000002FEE2A2  Unknown               Unknown  Unknown</div><div>cp2k.psmp          000000000299C076  Unknown               Unknown  Unknown</div><div>cp2k.psmp          0000000002848FD8  Unknown               Unknown  Unknown</div><div>cp2k.psmp          000000000299DB9C  Unknown               Unknown  Unknown</div><div>cp2k.psmp          000000000089B34B  Unknown               Unknown  Unknown</div><div>cp2k.psmp          0000000000895E21  Unknown               Unknown  Unknown</div><div>cp2k.psmp          00000000013E72C8  Unknown               Unknown  Unknown</div><div>cp2k.psmp          00000000013DDD94  Unknown               Unknown  Unknown</div><div>cp2k.psmp          0000000001121BCD  Unknown               Unknown  Unknown</div><div>cp2k.psmp          0000000000DA9518  Unknown               Unknown  Unknown</div><div>cp2k.psmp          0000000000DA7961  Unknown               Unknown  Unknown</div><div>cp2k.psmp          00000000007B4FA5  Unknown               Unknown  Unknown</div><div>cp2k.psmp          0000000000513751  Unknown               Unknown  Unknown</div><div>cp2k.psmp          0000000000512C14  Unknown               Unknown  Unknown</div><div>cp2k.psmp          00000000004286DD  Unknown               Unknown  Unknown</div><div>cp2k.psmp          000000000041EF28  Unknown               Unknown  Unknown</div><div>cp2k.psmp          000000000041DB7E  Unknown               Unknown  Unknown</div><div>libc-2.17.so       00002B443B39F3D5  __libc_start_main     Unknown  Unknown</div><div>cp2k.psmp          000000000041DA72  Unknown               Unknown  Unknown</div></div><div><br></div><div><br></div></div>