Gromacs requires many other shell variables (which are Gromacs specific) for specifically running the code on the GPUs. On the other hand, NAMD (which is another MD code) just requires that you load the required bindings and use the binary to run the executable. The first case that I had reported is similar to the case of NAMD (which I had demonstrated that it does not work). I checked the GPU usage using "nvidia-smi" and observed that the GPU utilization was 0% and the process was not registered on the GPU cores, even though the program was running. <div><br /></div><div>Best,</div><div>Hemanth<br /><br /></div><div class="gmail_quote"><div dir="auto" class="gmail_attr">On Tuesday, June 11, 2024 at 9:35:24 AM UTC-6 Johann Pototschnig wrote:<br/></div><blockquote class="gmail_quote" style="margin: 0 0 0 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div>It doesn't have to be cp2k. Any program that uses GPUs in order to figure out the script you need. <br></div><div><div><br></div><div>srun / mpirun might need additional options:</div><div><a href="https://slurm.schedmd.com/srun.html#OPT_gpu-bind" target="_blank" rel="nofollow" data-saferedirecturl="https://www.google.com/url?hl=en&q=https://slurm.schedmd.com/srun.html%23OPT_gpu-bind&source=gmail&ust=1718206843145000&usg=AOvVaw0VcaWvqTbPFESRNi5pTBxQ">https://slurm.schedmd.com/srun.html#OPT_gpu-bind</a></div></div><div><br></div><div>Otherwise there are some options to get additional information:<br></div><div>- Since you have OpenMPI you can get more information if you put <span>"ompi_info"<span> in your script. <br></span></span></div><div><br></div><div>-You can also put <span>"nvidia-smi" in the script to get GPU information:</span></div><div> <a href="https://developer.nvidia.com/system-management-interface" target="_blank" rel="nofollow" data-saferedirecturl="https://www.google.com/url?hl=en&q=https://developer.nvidia.com/system-management-interface&source=gmail&ust=1718206843145000&usg=AOvVaw2z_SANCOyAunxeNevQ6Di2">https://developer.nvidia.com/system-management-interface</a></div><div><br></div><div>- "echo $<span>CUDA_VISIBLE_DEVICES" should show the GPUs that are visible<br></span></div><div><br></div><div>Also you can check: <a href="https://docs.open-mpi.org/en/v5.0.x/tuning-apps/networking/cuda.html" target="_blank" rel="nofollow" data-saferedirecturl="https://www.google.com/url?hl=en&q=https://docs.open-mpi.org/en/v5.0.x/tuning-apps/networking/cuda.html&source=gmail&ust=1718206843145000&usg=AOvVaw15oa2OD8_IZ_WUpFYKAdsh">https://docs.open-mpi.org/en/v5.0.x/tuning-apps/networking/cuda.html</a></div><br><div class="gmail_quote"><div dir="auto" class="gmail_attr">On Tuesday, June 11, 2024 at 4:37:29 PM UTC+2 Hemanth Haridas wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">The script that I used for submitting is reproduced below<div><br></div><div></div><div><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(50,244,241)"><span style="font-variant-ligatures:no-common-ligatures">#!/bin/bash</span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(50,244,241)"><span style="font-variant-ligatures:no-common-ligatures">#SBATCH --job-name=LiCl ### Job Name</span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(50,244,241)"><span style="font-variant-ligatures:no-common-ligatures">#SBATCH --output=cp2k.out ### File in which to store job output</span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(50,244,241)"><span style="font-variant-ligatures:no-common-ligatures">#SBATCH --error=cp2k.err ### File in which to store job error messages</span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(50,244,241)"><span style="font-variant-ligatures:no-common-ligatures">#SBATCH --time=3-00:00:00 ### Wall clock time limit in Days-HH:MM:SS </span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(50,244,241)"><span style="font-variant-ligatures:no-common-ligatures">#SBATCH --ntasks=64</span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(50,244,241)"><span style="font-variant-ligatures:no-common-ligatures">#SBATCH --gres=gpu:1</span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(50,244,241)"><span style="font-variant-ligatures:no-common-ligatures">#SBATCH --cpus-per-task=1</span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0);min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">module load gcc cuda/</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(251,30,255)">11</span><span style="font-variant-ligatures:no-common-ligatures">.</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(251,30,255)">8</span><span style="font-variant-ligatures:no-common-ligatures">.</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(251,30,255)">0</span><span style="font-variant-ligatures:no-common-ligatures"> openmpi/</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(251,30,255)">4</span><span style="font-variant-ligatures:no-common-ligatures">.</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(251,30,255)">1</span><span style="font-variant-ligatures:no-common-ligatures">.6-gpu intel-oneapi-mkl/</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(251,30,255)">2022</span><span style="font-variant-ligatures:no-common-ligatures">.</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(251,30,255)">0</span><span style="font-variant-ligatures:no-common-ligatures">.</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(251,30,255)">2</span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0);min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(235,239,24)">source</span><span style="font-variant-ligatures:no-common-ligatures"> /cp2k_plumed_gpu/cp2k-2024.</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(251,30,255)">1</span><span style="font-variant-ligatures:no-common-ligatures">/tools/toolchain/install/setup</span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0);min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(50,244,241)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(235,239,24)">export</span><span style="font-variant-ligatures:no-common-ligatures"><b> OMP_NUM_THREADS=</b></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(99,226,255)">$SLURM_NTASKS</span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0);min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p></div><div><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">mpirun -np 1 /cp2k_plumed_gpu/cp2k-2024.</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(251,30,255)">1</span><span style="font-variant-ligatures:no-common-ligatures">/exe/local_cuda/cp2k.psmp </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(254,215,216)">-i</span><span style="font-variant-ligatures:no-common-ligatures"> colvars.inp </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(254,215,216)">-o</span><span style="font-variant-ligatures:no-common-ligatures"> colvars.out</span></p><br></div><div><br></div><div>We do not have a version of cp2k installed cluster-wide meaning that I do not have a script that I can compare with.</div><div class="gmail_quote"><div dir="auto" class="gmail_attr">On Tuesday, June 11, 2024 at 5:07:07 AM UTC-6 Johann Pototschnig wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Which GPU bindings did you end up using?</div><div><br></div><div>Did you compare to submission scripts for your cluster for which the GPU is used?<br></div><div><br></div><br><div class="gmail_quote"><div dir="auto" class="gmail_attr">On Monday, June 10, 2024 at 6:31:45 PM UTC+2 Hemanth Haridas wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I tried running cp2k as described in the previous email, but the code still does not run on the gpu, and the GPU usage is still zero.<div><br></div><div>Sincererly,</div><div>Hemanth</div><div><br></div><div class="gmail_quote"><div dir="auto" class="gmail_attr">On Friday, June 7, 2024 at 11:02:42 AM UTC-6 Johann Pototschnig wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>For cuda it is rather:</div><div><a href="https://github.com/tom-papatheodore/hello_jobstep" rel="nofollow" target="_blank" data-saferedirecturl="https://www.google.com/url?hl=en&q=https://github.com/tom-papatheodore/hello_jobstep&source=gmail&ust=1718206843146000&usg=AOvVaw2ModD1cGTEEwYGCN4LR78p">https://github.com/tom-papatheodore/hello_jobstep</a></div><div><br></div><div class="gmail_quote"><div dir="auto" class="gmail_attr">On Friday, June 7, 2024 at 6:52:33 PM UTC+2 Johann Pototschnig wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>It links to cuda, there should be no problem, but you are missing the mpirun / srun:</div><div><br></div><div>mpirun -n 1 -x OMP_NUM_THREADS=$... <span style="font-variant-ligatures:no-common-ligatures">/cp2k_plumed_gpu/cp2k-2024.</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(251,30,255)">1</span><span style="font-variant-ligatures:no-common-ligatures">/exe/local_cuda/cp2k.psmp </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(254,215,216)">-i</span><span style="font-variant-ligatures:no-common-ligatures"> colvars.inp </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(254,215,216)">-o</span><span style="font-variant-ligatures:no-common-ligatures"> colvars.out</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div>depending on your system there might be additional options for mpirun/srun necessary. <br></div><div><br></div><div>The following program can help to figure out bindings:<br></div><div><a href="https://code.ornl.gov/olcf/hello_jobstep" rel="nofollow" target="_blank" data-saferedirecturl="https://www.google.com/url?hl=en&q=https://code.ornl.gov/olcf/hello_jobstep&source=gmail&ust=1718206843146000&usg=AOvVaw0nMdo6RN_gPGltuUmQPBF-">https://code.ornl.gov/olcf/hello_jobstep</a></div><div><br></div><div class="gmail_quote"><div dir="auto" class="gmail_attr">On Friday, June 7, 2024 at 5:18:47 PM UTC+2 Hemanth Haridas wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">CC = /gcc-8.5.0/openmpi-4.1.6-chhfokbbf3fbb2t4uo7ns4ukaripskzj/bin/mpicc</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">CXX = /gcc-8.5.0/openmpi-4.1.6-chhfokbbf3fbb2t4uo7ns4ukaripskzj/bin/mpic++</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">AR = ar -r</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">FC = /gcc-8.5.0/openmpi-4.1.6-chhfokbbf3fbb2t4uo7ns4ukaripskzj/bin/mpifort</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">LD = /gcc-8.5.0/openmpi-4.1.6-chhfokbbf3fbb2t4uo7ns4ukaripskzj/bin/mpifort</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">DFLAGS = -D__OFFLOAD_CUDA -D__DBCSR_ACC -D__LIBXSMM -D__parallel -D__MKL -D__FFTW3 -D__SCALAPACK -D__LIBINT -D__LIBXC -D__LIBGRPP -D__GSL -D__PLUMED2 -D__SPGLIB -D__OFFLOAD_GEMM -D__SPLA</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">WFLAGS = -Werror=aliasing -Werror=ampersand -Werror=c-binding-type -Werror=intrinsic-shadow -Werror=intrinsics-std -Werror=line-truncation -Werror=tabs -Werror=target-lifetime -Werror=underflow -Werror=unused-but-set-variable -Werror=unused-variable -Werror=unused-dummy-argument -Werror=unused-parameter -Werror=unused-label -Werror=conversion -Werror=zerotrip -Wno-maybe-uninitialized -Wuninitialized -Wuse-without-only</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">FCDEBFLAGS = -fbacktrace -ffree-form -fimplicit-none -std=f2008</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">CFLAGS = -fno-omit-frame-pointer -fopenmp -g -mtune=native -O3 -funroll-loops $(PROFOPT) -I/gcc-8.5.0/openmpi-4.1.6-chhfokbbf3fbb2t4uo7ns4ukaripskzj/include -pthread -m64 -I/gcc-8.5.0/intel-oneapi-mkl-2023.2.0-etwucm5d3s2qu7eiuaaxastbiukj2ori/mkl/2023.2.0/include -I/gcc-8.5.0/intel-oneapi-mkl-2023.2.0-etwucm5d3s2qu7eiuaaxastbiukj2ori/mkl/2023.2.0/include/fftw -I'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/libint-v2.6.0-cp2k-lmax-5/include' -I'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/libxc-6.2.2/include' -I'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/libgrpp-main-20231215/include' -I'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/libxsmm-1.17/include' -I'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/gsl-2.7/include' -I/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/spglib-1.16.2/include -I'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/SpLA-1.5.5/include/spla' -std=c11 -Wall -Wextra -Werror -Wno-vla-parameter -Wno-deprecated-declarations $(DFLAGS) -I/gcc-8.5.0/cuda-11.8.0-3wlxktsbgw2ui4wvdnsy7w7xyxlkkwju/include</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">FCFLAGS = -fno-omit-frame-pointer -fopenmp -g -mtune=native -O3 -funroll-loops $(PROFOPT) -I/gcc-8.5.0/openmpi-4.1.6-chhfokbbf3fbb2t4uo7ns4ukaripskzj/include -pthread -m64 -I/gcc-8.5.0/intel-oneapi-mkl-2023.2.0-etwucm5d3s2qu7eiuaaxastbiukj2ori/mkl/2023.2.0/include -I/gcc-8.5.0/intel-oneapi-mkl-2023.2.0-etwucm5d3s2qu7eiuaaxastbiukj2ori/mkl/2023.2.0/include/fftw -I'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/libint-v2.6.0-cp2k-lmax-5/include' -I'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/libxc-6.2.2/include' -I'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/libgrpp-main-20231215/include' -I'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/libxsmm-1.17/include' -I'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/gsl-2.7/include' -I/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/spglib-1.16.2/include -I'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/SpLA-1.5.5/include/spla' $(FCDEBFLAGS) $(WFLAGS) $(DFLAGS)</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">CXXFLAGS = -O2 -fPIC -fno-omit-frame-pointer -fopenmp -g -march=native -mtune=native --std=c++14 $(DFLAGS) -Wno-deprecated-declarations -I/gcc-8.5.0/cuda-11.8.0-3wlxktsbgw2ui4wvdnsy7w7xyxlkkwju/include</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">LDFLAGS = $(FCFLAGS) -Wl,--enable-new-dtags -pthread -L/gcc-8.5.0/openmpi-4.1.6-chhfokbbf3fbb2t4uo7ns4ukaripskzj/lib -L/gcc-8.5.0/ucx-1.14.0-cupo7hrn2exqqwfyatdigeuiiqijaulw/lib -L/uufs/<a href="http://chpc.utah.edu/sys/spack/v019/linux-rocky8-x86_64/gcc-8.5.0/zlib-1.2.13-dcpzngybj4fisn6ojapnels3yfwcxqgk/lib" rel="nofollow" target="_blank" data-saferedirecturl="https://www.google.com/url?hl=en&q=http://chpc.utah.edu/sys/spack/v019/linux-rocky8-x86_64/gcc-8.5.0/zlib-1.2.13-dcpzngybj4fisn6ojapnels3yfwcxqgk/lib&source=gmail&ust=1718206843146000&usg=AOvVaw1VlEwt8rmI_C4KrO3yPMaf">chpc.utah.edu/sys/spack/v019/linux-rocky8-x86_64/gcc-8.5.0/zlib-1.2.13-dcpzngybj4fisn6ojapnels3yfwcxqgk/lib</a> -Wl,-rpath -Wl,/gcc-8.5.0/openmpi-4.1.6-chhfokbbf3fbb2t4uo7ns4ukaripskzj/lib -Wl,-rpath -Wl,/gcc-8.5.0/ucx-1.14.0-cupo7hrn2exqqwfyatdigeuiiqijaulw/lib -Wl,-rpath -Wl,/gcc-8.5.0/zlib-1.2.13-dcpzngybj4fisn6ojapnels3yfwcxqgk/lib -L'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/libint-v2.6.0-cp2k-lmax-5/lib' -L'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/libxc-6.2.2/lib' -Wl,-rpath,'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/libxc-6.2.2/lib' -L'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/libgrpp-main-20231215/lib' -Wl,-rpath,'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/libgrpp-main-20231215/lib' -L'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/libxsmm-1.17/lib' -Wl,-rpath,'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/libxsmm-1.17/lib' -L'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/gsl-2.7/lib' -Wl,-rpath,'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/gsl-2.7/lib' -L'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/plumed-2.9.0/lib' -Wl,-rpath,'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/plumed-2.9.0/lib' -L'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/spglib-1.16.2/lib' -Wl,-rpath,'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/spglib-1.16.2/lib' -L'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/SpLA-1.5.5/lib/cuda' -Wl,-rpath,'/cp2k_plumed_gpu/cp2k-2024.1/tools/toolchain/install/SpLA-1.5.5/lib/cuda' -L'/gcc-8.5.0/cuda-11.8.0-3wlxktsbgw2ui4wvdnsy7w7xyxlkkwju/targets/x86_64-linux/lib' -Wl,-rpath,'/gcc-8.5.0/cuda-11.8.0-3wlxktsbgw2ui4wvdnsy7w7xyxlkkwju/targets/x86_64-linux/lib' -L'/usr/lib64' -Wl,-rpath,'/usr/lib64'</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">LDFLAGS_C =</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">LIBS = -lspla -lsymspg -l:libplumed.a -ldl -lstdc++ -lz -ldl -lgsl -lxsmmf -lxsmm -ldl -lpthread -llibgrpp -lxcf03 -lxc -lint2 -lmpi_cxx -lmpi -L/gcc-8.5.0/intel-oneapi-mkl-2023.2.0-etwucm5d3s2qu7eiuaaxastbiukj2ori/mkl/2023.2.0/lib/intel64 -Wl,-rpath,/gcc-8.5.0/intel-oneapi-mkl-2023.2.0-etwucm5d3s2qu7eiuaaxastbiukj2ori/mkl/2023.2.0/lib/intel64 -lmkl_scalapack_lp64 -Wl,--start-group -lmkl_gf_lp64 -lmkl_sequential -lmkl_core -lmkl_blacs_openmpi_lp64 -Wl,--end-group -lpthread -lm -ldl -lstdc++ -lcudart -lnvrtc -lcuda -lcufft -lcublas -lrt</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">GPUVER = A100</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">OFFLOAD_CC = nvcc</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">OFFLOAD_FLAGS = -g -arch sm_80 -O3 -allow-unsupported-compiler -Xcompiler='-fopenmp -Wall -Wextra -Werror' --std=c++11 $(DFLAGS)</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">OFFLOAD_TARGET = cuda</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">FYPPFLAGS = -n --line-marker-format=gfortran5</span></p><div class="gmail_quote"><div dir="auto" class="gmail_attr">On Friday, June 7, 2024 at 1:51:34 AM UTC-6 Johann Pototschnig wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Can you provide the local_cuda.psmp file which you find in the arch folder?<br><br><div class="gmail_quote"><div dir="auto" class="gmail_attr">On Thursday, June 6, 2024 at 9:10:38 PM UTC+2 Hemanth Haridas wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><font face="Sans Serif" size="3">I am trying to run CP2K on a linux cluster with GPU support. I have successfully complied the code with CUDA support. But the utilization of GPU is zero, even though the program is running , meaning that the code is running on cpu cores.</font><div><font face="Sans Serif" size="3"><br></font></div><div><font face="Sans Serif" size="3">This is the script that I am using to run cp2k </font></div><div><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(50,244,241)"><span style="font-variant-ligatures:no-common-ligatures">#!/bin/bash</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(50,244,241)"><span style="font-variant-ligatures:no-common-ligatures">#SBATCH --job-name=LiCl ### Job Name</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(50,244,241)"><span style="font-variant-ligatures:no-common-ligatures">#SBATCH --output=cp2k.out ### File in which to store job output</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(50,244,241)"><span style="font-variant-ligatures:no-common-ligatures">#SBATCH --error=cp2k.err ### File in which to store job error messages</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(50,244,241)"><span style="font-variant-ligatures:no-common-ligatures">#SBATCH --time=3-00:00:00 ### Wall clock time limit in Days-HH:MM:SS </span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(50,244,241)"><span style="font-variant-ligatures:no-common-ligatures">#SBATCH --ntasks=64</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(50,244,241)"><span style="font-variant-ligatures:no-common-ligatures">#SBATCH --gres=gpu:1</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(50,244,241)"><span style="font-variant-ligatures:no-common-ligatures">#SBATCH --cpus-per-task=1</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0);min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">module load gcc cuda/</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(251,30,255)">11</span><span style="font-variant-ligatures:no-common-ligatures">.</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(251,30,255)">8</span><span style="font-variant-ligatures:no-common-ligatures">.</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(251,30,255)">0</span><span style="font-variant-ligatures:no-common-ligatures"> openmpi/</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(251,30,255)">4</span><span style="font-variant-ligatures:no-common-ligatures">.</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(251,30,255)">1</span><span style="font-variant-ligatures:no-common-ligatures">.6-gpu intel-oneapi-mkl/</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(251,30,255)">2022</span><span style="font-variant-ligatures:no-common-ligatures">.</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(251,30,255)">0</span><span style="font-variant-ligatures:no-common-ligatures">.</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(251,30,255)">2</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0);min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(235,239,24)">source</span><span style="font-variant-ligatures:no-common-ligatures"> /cp2k_plumed_gpu/cp2k-2024.</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(251,30,255)">1</span><span style="font-variant-ligatures:no-common-ligatures">/tools/toolchain/install/setup</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0);min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(50,244,241)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(235,239,24)">export</span><span style="font-variant-ligatures:no-common-ligatures"><b> OMP_NUM_THREADS=</b></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(99,226,255)">$SLURM_NTASKS</span></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0);min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p> <p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">/cp2k_plumed_gpu/cp2k-2024.</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(251,30,255)">1</span><span style="font-variant-ligatures:no-common-ligatures">/exe/local_cuda/cp2k.psmp </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(254,215,216)">-i</span><span style="font-variant-ligatures:no-common-ligatures"> colvars.inp </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(254,215,216)">-o</span><span style="font-variant-ligatures:no-common-ligatures"> colvars.out</span></p><p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;font-size-adjust:none;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><br></span></p></div><div><font face="Sans Serif" size="3">Are there any additional flags that I need to use to run the code on GPUs?</font></div></blockquote></div></blockquote></div></blockquote></div></blockquote></div></blockquote></div></blockquote></div></blockquote></div></blockquote></div></blockquote></div> <p></p> -- <br /> You received this message because you are subscribed to the Google Groups "cp2k" group.<br /> To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:cp2k+unsubscribe@googlegroups.com">cp2k+unsubscribe@googlegroups.com</a>.<br /> To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/cp2k/6decace8-842a-4cb9-a3aa-fab4dd07f564n%40googlegroups.com?utm_medium=email&utm_source=footer">https://groups.google.com/d/msgid/cp2k/6decace8-842a-4cb9-a3aa-fab4dd07f564n%40googlegroups.com</a>.<br />