[CP2K-user] [CP2K:18588] Install issues with IBM Power9 processors with Nvidia V100 GPU

[Nathan Keilbart]

Hello everyone,

I've been working on installing CP2K on a system with IBM Power9 processors 
and Nvidia V100 GPUs. I'm using the toolchain with these options:

./install_cp2k_toolchain.sh -j --with-cmake=system --mpi-mode=openmpi 
--enable-cuda --gpu-ver=V100

It installs all the dependencies without any errors so that I copy over the 
files to the arch folder and then source the setup file followed by

make -j ARCH=local_cuda VERSION=psmp

The following is some of the last lines of output

/usr/bin/env python3 
/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/exts/dbcsr/tools/build_utils/fypp/bin/fypp 
-n --line-marker-format=gfortran5 
/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/exts/dbcsr/src/tensors/dbcsr_tensor_test.F 
dbcsr_tensor_test.F90
c -fno-omit-frame-pointer -fopenmp -g -mtune=native  -O3 -funroll-loops   
 -I'/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/openblas-0.3.21/include' 
-I'/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/fftw-3.3.10/include' 
-I'/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/libint-v2.6.0-cp2k-lmax-5/include' 
-I'/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/libxc-6.0.0/include' 
-I'/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/COSMA-2.6.2/include' 
-I'/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/elpa-2022.11.001/nvidia/include/elpa_openmp-2022.11.001/modules' 
-I'/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/elpa-2022.11.001/nvidia/include/elpa_openmp-2022.11.001/elpa' 
-I'/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/gsl-2.7/include' 
-I/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/hdf5-1.12.0/include 
-I/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/libvdwxc-0.4.0/include 
-I/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/spglib-1.16.2/include 
-I'/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/SpFFT-1.0.6/include' 
-I'/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/SpLA-1.5.4/include/spla' 
-I/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/sirius-7.3.2/include/cuda 
-fbacktrace -ffree-form -fimplicit-none -std=f2008  -Werror=aliasing 
-Werror=ampersand -Werror=c-binding-type -Werror=intrinsic-shadow 
-Werror=intrinsics-std -Werror=line-truncation -Werror=tabs 
-Werror=target-lifetime -Werror=underflow -Werror=unused-but-set-variable 
-Werror=unused-variable -Werror=unused-dummy-argument -Werror=conversion 
-Werror=zerotrip -Wno-maybe-uninitialized -Wuninitialized 
-Wuse-without-only  -D__OFFLOAD_CUDA -D__DBCSR_ACC   -D__FFTW3  -D__LIBINT 
-D__LIBXC -D__SCALAPACK -D__COSMA -D__ELPA -D__ELPA_NVIDIA_GPU -D__GSL 
-D__HDF5 -D__LIBVDWXC -D__SPGLIB -D__LIBVORI -D__SPFFT  -D__OFFLOAD_GEMM 
 -D__SPLA -D__SIRIUS    -D__CUDA -D__SHORT_FILE__="\"dbcsr_tensor_test.F\"" 
-I'/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/exts/dbcsr/src/tensors/' 
-I'/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/exts/dbcsr/src' 
dbcsr_tensor_test.F90 
/bin/sh: c: command not found
make[4]: 
[/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/exts/build_dbcsr//Makefile:258: 
dbcsr_tensor_test.o] Error 127 (ignored)
/usr/bin/env python3 
/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/exts/dbcsr/tools/build_utils/fypp/bin/fypp 
-n --line-marker-format=gfortran5 
/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/exts/dbcsr/src/tensors/dbcsr_tensor_api.F 
dbcsr_tensor_api.F90
c -fno-omit-frame-pointer -fopenmp -g -mtune=native  -O3 -funroll-loops   
 -I'/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/openblas-0.3.21/include' 
-I'/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/fftw-3.3.10/include' 
-I'/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/libint-v2.6.0-cp2k-lmax-5/include' 
-I'/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/libxc-6.0.0/include' 
-I'/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/COSMA-2.6.2/include' 
-I'/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/elpa-2022.11.001/nvidia/include/elpa_openmp-2022.11.001/modules' 
-I'/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/elpa-2022.11.001/nvidia/include/elpa_openmp-2022.11.001/elpa' 
-I'/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/gsl-2.7/include' 
-I/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/hdf5-1.12.0/include 
-I/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/libvdwxc-0.4.0/include 
-I/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/spglib-1.16.2/include 
-I'/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/SpFFT-1.0.6/include' 
-I'/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/SpLA-1.5.4/include/spla' 
-I/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/tools/toolchain/install/sirius-7.3.2/include/cuda 
-fbacktrace -ffree-form -fimplicit-none -std=f2008  -Werror=aliasing 
-Werror=ampersand -Werror=c-binding-type -Werror=intrinsic-shadow 
-Werror=intrinsics-std -Werror=line-truncation -Werror=tabs 
-Werror=target-lifetime -Werror=underflow -Werror=unused-but-set-variable 
-Werror=unused-variable -Werror=unused-dummy-argument -Werror=conversion 
-Werror=zerotrip -Wno-maybe-uninitialized -Wuninitialized 
-Wuse-without-only  -D__OFFLOAD_CUDA -D__DBCSR_ACC   -D__FFTW3  -D__LIBINT 
-D__LIBXC -D__SCALAPACK -D__COSMA -D__ELPA -D__ELPA_NVIDIA_GPU -D__GSL 
-D__HDF5 -D__LIBVDWXC -D__SPGLIB -D__LIBVORI -D__SPFFT  -D__OFFLOAD_GEMM 
 -D__SPLA -D__SIRIUS    -D__CUDA -D__SHORT_FILE__="\"dbcsr_tensor_api.F\"" 
-I'/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/exts/dbcsr/src/tensors/' 
-I'/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/exts/dbcsr/src' 
dbcsr_tensor_api.F90 
/bin/sh: c: command not found
make[4]: 
[/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/exts/build_dbcsr//Makefile:258: 
dbcsr_tensor_api.o] Error 127 (ignored)
Updating archive 
/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/lib/local_cuda/psmp/exts/dbcsr/libdbcsr.a
ar: creating 
/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/lib/local_cuda/psmp/exts/dbcsr/libdbcsr.a
ar: dbcsr_cuda_profiling.o: No such file or directory
make[4]: *** 
[/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/exts/build_dbcsr//Makefile:330: 
/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/lib/local_cuda/psmp/exts/dbcsr/libdbcsr.a] 
Error 1
make[3]: *** 
[/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/exts/build_dbcsr/Makefile:179: 
libdbcsr] Error 2
make[2]: *** 
[/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/exts/Makefile.inc:38: dbcsr] 
Error 2
make[1]: *** [/usr/gapps/qsg/codes/cp2k/lassen/v2023.1/Makefile:128: psmp] 
Error 2
make: *** [Makefile:123: all] Error 2

It seems that it is having issues with the DBCSR module. I initially had an 
issue with this because I seemed to have left off the --recursive option 
and after making sure my git clone had that it at least let me build most 
of the serial version. It at least gave me the cp2k.sopt binary and it 
seems to at least take inputs. I didn't have a chance to test it too much 
yet. When I got this binary I had done 

make -j ARCH=local_cuda VERSION="ssmp sdbg psmp pdbg"

as suggested.

Also, I've attempted to install with spack by using

spack install 
cp2k at 2023.1+cosma+cuda+elpa+libint+libxc+mpi+openmp+pexsi+plumed+sirius+spglib 
smm=blas cuda_arch=70

These are some of the last lines of output

 >> 4028    collect2: error: ld returned 1 exit status
  >> 4029    collect2: error: ld returned 1 exit status
  >> 4030    make[3]: *** 
[/tmp/keilbart/spack-stage/spack-stage-cp2k-2023.1-24dhoyt24tbnn4d423glgoeqqquibmb6/spack-src/obj/linux-rhel7-power9le-gcc/psmp/
             all.dep:178: 
/tmp/keilbart/spack-stage/spack-stage-cp2k-2023.1-24dhoyt24tbnn4d423glgoeqqquibmb6/spack-src/exe/linux-rhel7-power9le-gcc/cp2k.p
             smp] Error 1
     4031    make[3]: *** Waiting for unfinished jobs....
  >> 4032    make[3]: *** 
[/tmp/keilbart/spack-stage/spack-stage-cp2k-2023.1-24dhoyt24tbnn4d423glgoeqqquibmb6/spack-src/obj/linux-rhel7-power9le-gcc/psmp/
             all.dep:194: 
/tmp/keilbart/spack-stage/spack-stage-cp2k-2023.1-24dhoyt24tbnn4d423glgoeqqquibmb6/spack-src/exe/linux-rhel7-power9le-gcc/libcp2
             k_unittest.psmp] Error 1
  >> 4033    make[2]: *** 
[/tmp/keilbart/spack-stage/spack-stage-cp2k-2023.1-24dhoyt24tbnn4d423glgoeqqquibmb6/spack-src/Makefile:146: 
all] Error 2
  >> 4034    make[1]: *** 
[/tmp/keilbart/spack-stage/spack-stage-cp2k-2023.1-24dhoyt24tbnn4d423glgoeqqquibmb6/spack-src/Makefile:128: 
psmp] Error 2
  >> 4035    make: *** [Makefile:123: all] Error 2

Finally, I also have some intel machines that I'm attempting to build on 
and having issues as well but we can start with the IBM machine as we're 
hoping to accelerate the simulations with the GPU.

Please let me know what other information I can provide. Thank you.

Nathan

-- 
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+unsubscribe at googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/0ba3ac06-1cf6-44ef-a626-bc12491acc86n%40googlegroups.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.cp2k.org/archives/cp2k-user/attachments/20230329/2bfd43bf/attachment.htm>