<div dir="ltr">Thanks, Fabian and Thomas for the clarifications!<div><br></div><div><blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;"><div style="word-wrap:break-word;line-break:after-white-space"><div>to the best of my knowledge all other machines except for Piz Daint, where the "no <br></div><div>GPU" comment is present, are not equipped with GPUs, so everything is consistent. </div></div></blockquote><div>Ok, hearing you say this it may seem obvious, also since only CPU core counts are reported.</div><div>I still feel it would be useful to mention somewhere in the beginning that the tests are run just on CPU cores, since it may not be clear to everyone.</div><div><br></div><div>And as I mentioned, given that there is the CUDA version with significant speedups for hybrids & post-HF, perhaps it would make sense to include in the list a few benchmark results with the GPU as well?</div><div><br></div><div>Best,</div><div>Leo</div><div> </div><blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;"><div style="word-wrap:break-word;line-break:after-white-space"><div><br></div><div>Cheers, </div><div>Thomas</div><div><div><br><blockquote type="cite"><div>Am 04.05.2020 um 18:12 schrieb Leopold Talirz <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="686010KOBQAJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">leo...@gmail.com</a>>:</div><br><div><div dir="ltr">Dear Fabian,<div><br></div><div>thanks a lot for checking and for pinning down the issue.</div><div><br></div><div>Since this is a rather serious issue, my first instinct was to check on the performance page of cp2k to see whether CUDA + OMP was ever used in benchmark studies.</div><div><a href="https://www.cp2k.org/performance" target="_blank" rel="nofollow" onmousedown="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.cp2k.org%2Fperformance\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF_nRoOGPnzhl_CsnTSI65GmqkL-g';return true;" onclick="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.cp2k.org%2Fperformance\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNF_nRoOGPnzhl_CsnTSI65GmqkL-g';return true;">https://www.cp2k.org/<wbr>performance</a></div><div><br></div><div>Unfortunately, it is not clear to me from the page - something I now remember to have run in before:</div><div>E.g. for some systems it says explicitly "no GPU" but for others that can have a GPU (like Cray XC40) it does not say it and it is not clear whether this means the GPU was used or not.</div><div>May I suggest to the maintainer of this page to make this information explicit?</div><div><br></div><div>And if it turns out that there are currently no tests including the CUDA version on the list, perhaps it would make sense to include some?</div><div><br></div><div>Best wishes from Bern,</div><div>Leopold</div><div><br></div><div><br></div><div><br><br>On Monday, 4 May 2020 17:35:08 UTC+2, Fabian Ducry wrote:<blockquote class="gmail_quote" style="margin:0;margin-left:0.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Dear Andres,</div><div><br></div><div>I can confirm and reproduce the issue. Apparently it appears when combining CUDA + OMP in hybrid calculations. In that case the energy becomes a function of #OMP threads per rank. For your input I got (cp2k 8.0, revision 3e7b916, run on Piz Daint)</div><div></div><div> <wbr> no cuda OMP_NUM_THREADS = 1 OMP_NUM_THREADS = 3 OMP_NUM_THREADS = 6<br> Exchange-correlation energy: -433.84964308969535 -433.84964308969302 -435.33426106395467 <wbr> -435.96513615032325<br> Hartree-Fock Exchange energy: -127.87395928499694 -127.87395928499325 -125.97109874333140 <wbr> -125.24809389970088<br> Total energy: <wbr> -1976.39722899739672 -1976.39722899739013 -1975.95046919253809 <wbr> -1975.87080541858177</div><div><br></div><div>Without OMP parallelization the energies agrees with the calculation without CUDA accelleration. Increasing OMP_NUM_THREADS beyond 1 increases the Hartree-Fock Exchange energy.</div><div>Apparently you have to disable OMP to obtain correct results. This is obviously not very satisfying and I hope this gets fixed. I see that you used 1 MPI/12 OMP ranks per node. Try increasing the number of MPI ranks per node. To do so you have to set <br></div><div>export CRAY_CUDA_MPS=1 in the submission script.<br></div><div><br></div><div>I hope this helps.</div><div><br></div><div>Best,</div><div>Fabian<br></div></div></blockquote></div></div><div><br></div>
-- <br>
You received this message because you are subscribed to the Google Groups "cp2k" group.<br>
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="686010KOBQAJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">cp...@googlegroups.<wbr>com</a>.<br>
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/cp2k/048030dd-6532-4fc5-b127-b7a5b017dad7%40googlegroups.com?utm_medium=email&utm_source=footer" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/msgid/cp2k/048030dd-6532-4fc5-b127-b7a5b017dad7%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter';return true;" onclick="this.href='https://groups.google.com/d/msgid/cp2k/048030dd-6532-4fc5-b127-b7a5b017dad7%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter';return true;">https://groups.google.com/d/<wbr>msgid/cp2k/048030dd-6532-4fc5-<wbr>b127-b7a5b017dad7%<wbr>40googlegroups.com</a>.<br>
</div></blockquote></div><br></div></div></blockquote></div></div>