[CP2K-user] [CP2K:18385] Re: GPU oversubscription errors with 2023.1
jerryt...@gmail.com
jerrytanoury at gmail.com
Wed Jan 25 11:37:49 UTC 2023
Hi Alfio,
Yes, ELPA was the problem. I removed it from my build and CP2K worked as
expected. Where does ELPA help the most? The majority of my AIMD jobs are
1000 atoms or less. Will ELPA provide a performance advantage over
SCALAPACK for systems of that size?
Thank you,
Jerry
On Monday, January 23, 2023 at 4:09:41 AM UTC-5 Alfio Lazzaro wrote:
> I have no clue what's wrong here, however I see in your log that ELPA is
> giving some warning message. For this reason, I would suggest to avoid
> elpa, i.e. add `--with-elpa=no` during the toolchain installation. Does it
> work on a single GPU, i.e. a single MPI rank?
>
> Il giorno venerdì 20 gennaio 2023 alle 15:33:45 UTC+1 jerryt... at gmail.com
> ha scritto:
>
>> Dear Forum,
>> I successfully compiled v2023.1 (gcc-10.3.0, cuda-11.2, and MKL lib) with
>> the toolchain using:
>>
>> "-j 8 --no-check-certificate --install-all --with-gcc=system
>> --with-openmpi --with-mkl --with-sirius=no --with-spfft=no
>> --with-cmake=system --enable-cuda --gpu-ver=P100 --with-pexsi
>> --with-sirius=no --with-quip=no --with-hdf5=no --with-libvdwxc=no
>> --with-spla=no --with-libtorch=no"
>>
>> However, when I ran a test job, the job crashed and I got GPU
>> oversubscription as shown below:
>>
>> +-----------------------------------------------------------------------------+
>> | NVIDIA-SMI 515.65.01 Driver Version: 515.65.01 CUDA Version: 11.7
>> |
>>
>> |-------------------------------+----------------------+----------------------+
>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr.
>> ECC |
>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
>> Compute M. |
>> | | |
>> MIG M. |
>>
>> |===============================+======================+======================|
>> | 0 Tesla V100-SXM2... Off | 00000000:1A:00.0 Off |
>> 0 |
>> | N/A 42C P0 80W / 300W | 2585MiB / 16384MiB | 53%
>> Default |
>> | | |
>> N/A |
>>
>> +-------------------------------+----------------------+----------------------+
>> | 1 Tesla V100-SXM2... Off | 00000000:1C:00.0 Off |
>> 0 |
>> | N/A 36C P0 75W / 300W | 1664MiB / 16384MiB | 65%
>> Default |
>> | | |
>> N/A |
>>
>> +-------------------------------+----------------------+----------------------+
>> | 2 Tesla V100-SXM2... Off | 00000000:1D:00.0 Off |
>> 0 |
>> | N/A 36C P0 73W / 300W | 1616MiB / 16384MiB | 54%
>> Default |
>> | | |
>> N/A |
>>
>> +-------------------------------+----------------------+----------------------+
>> | 3 Tesla V100-SXM2... Off | 00000000:1E:00.0 Off |
>> 0 |
>> | N/A 40C P0 71W / 300W | 1614MiB / 16384MiB | 55%
>> Default |
>> | | |
>> N/A |
>>
>> +-------------------------------+----------------------+----------------------+
>>
>>
>>
>> +-----------------------------------------------------------------------------+
>> | Processes:
>> |
>> | GPU GI CI PID Type Process name GPU
>> Memory |
>> | ID ID Usage
>> |
>>
>> |=============================================================================|
>> | 0 N/A N/A 159608 C .../exe/local_cuda/cp2k.psmp
>> 1655MiB |
>> | 0 N/A N/A 159609 C .../exe/local_cuda/cp2k.psmp
>> 307MiB |
>> | 0 N/A N/A 159610 C .../exe/local_cuda/cp2k.psmp
>> 307MiB |
>> | 0 N/A N/A 159611 C .../exe/local_cuda/cp2k.psmp
>> 307MiB |
>> | 1 N/A N/A 159609 C .../exe/local_cuda/cp2k.psmp
>> 1659MiB |
>> | 2 N/A N/A 159610 C .../exe/local_cuda/cp2k.psmp
>> 1611MiB |
>> | 3 N/A N/A 159611 C .../exe/local_cuda/cp2k.psmp
>> 1609MiB |
>>
>> +-----------------------------------------------------------------------------+
>>
>> However, using CP2K 2022.2, I ran the job successfully and did not get
>> this oversubscription.
>>
>>
>> +-----------------------------------------------------------------------------+
>> | NVIDIA-SMI 515.65.01 Driver Version: 515.65.01 CUDA Version: 11.7
>> |
>>
>> |-------------------------------+----------------------+----------------------+
>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr.
>> ECC |
>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
>> Compute M. |
>> | | |
>> MIG M. |
>>
>> |===============================+======================+======================|
>> | 0 Tesla V100-SXM2... Off | 00000000:1A:00.0 Off |
>> 0 |
>> | N/A 37C P0 63W / 300W | 1598MiB / 16384MiB | 31%
>> Default |
>> | | |
>> N/A |
>>
>> +-------------------------------+----------------------+----------------------+
>> | 1 Tesla V100-SXM2... Off | 00000000:1C:00.0 Off |
>> 0 |
>> | N/A 33C P0 63W / 300W | 1606MiB / 16384MiB | 29%
>> Default |
>> | | |
>> N/A |
>>
>> +-------------------------------+----------------------+----------------------+
>> | 2 Tesla V100-SXM2... Off | 00000000:1D:00.0 Off |
>> 0 |
>> | N/A 34C P0 63W / 300W | 1562MiB / 16384MiB | 27%
>> Default |
>> | | |
>> N/A |
>>
>> +-------------------------------+----------------------+----------------------+
>> | 3 Tesla V100-SXM2... Off | 00000000:1E:00.0 Off |
>> 0 |
>> | N/A 37C P0 67W / 300W | 1560MiB / 16384MiB | 25%
>> Default |
>> | | |
>> N/A |
>>
>> +-------------------------------+----------------------+----------------------+
>>
>>
>>
>> +-----------------------------------------------------------------------------+
>> | Processes:
>> |
>> | GPU GI CI PID Type Process name GPU
>> Memory |
>> | ID ID Usage
>> |
>>
>> |=============================================================================|
>> | 0 N/A N/A 163862 C .../exe/local_cuda/cp2k.psmp
>> 1599MiB |
>> | 1 N/A N/A 163863 C .../exe/local_cuda/cp2k.psmp
>> 1603MiB |
>> | 2 N/A N/A 163864 C .../exe/local_cuda/cp2k.psmp
>> 1557MiB |
>> | 3 N/A N/A 163865 C .../exe/local_cuda/cp2k.psmp
>> 1555MiB |
>>
>> +-----------------------------------------------------------------------------+
>>
>> Additionally, the system output file shows the following CUDA runtime
>> error:
>>
>> CUDA RUNTIME API error: EventRecord failed with error
>> cudaErrorInvalidResourceHandle
>> (/cluster/home/tanoury/CP2K/cp2k-2023.1_GPU/exts/dbcsr/src/acc/cuda_hip/acc_event.cpp::60)
>>
>> I have also attached the error output.
>>
>> Any help to solve this problem is greatly appreciated.
>>
>> Thank you so much,
>> Jerry
>>
>>
--
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+unsubscribe at googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/345c710b-5323-4f3b-93cf-f34ae7c58300n%40googlegroups.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.cp2k.org/archives/cp2k-user/attachments/20230125/c88cba6e/attachment-0001.htm>
More information about the CP2K-user
mailing list