<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Dear Leopold, <div class=""><br class=""></div><div class="">to the best of my knowledge all other machines except for Piz Daint, where the "no </div><div class="">GPU" comment is present, are not equipped with GPUs, so everything is consistent. </div><div class=""><br class=""></div><div class="">Cheers, </div><div class="">Thomas</div><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">Am 04.05.2020 um 18:12 schrieb Leopold Talirz <<a href="mailto:leopol...@gmail.com" class="">leopol...@gmail.com</a>>:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Dear Fabian,<div class=""><br class=""></div><div class="">thanks a lot for checking and for pinning down the issue.</div><div class=""><br class=""></div><div class="">Since this is a rather serious issue, my first instinct was to check on the performance page of cp2k to see whether CUDA + OMP was ever used in benchmark studies.</div><div class=""><a href="https://www.cp2k.org/performance" class="">https://www.cp2k.org/performance</a></div><div class=""><br class=""></div><div class="">Unfortunately, it is not clear to me from the page - something I now remember to have run in before:</div><div class="">E.g. for some systems it says explicitly "no GPU" but for others that can have a GPU (like Cray XC40) it does not say it and it is not clear whether this means the GPU was used or not.</div><div class="">May I suggest to the maintainer of this page to make this information explicit?</div><div class=""><br class=""></div><div class="">And if it turns out that there are currently no tests including the CUDA version on the list, perhaps it would make sense to include some?</div><div class=""><br class=""></div><div class="">Best wishes from Bern,</div><div class="">Leopold</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><br class="">On Monday, 4 May 2020 17:35:08 UTC+2, Fabian Ducry wrote:<blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;"><div dir="ltr" class=""><div class="">Dear Andres,</div><div class=""><br class=""></div><div class="">I can confirm and reproduce the issue. Apparently it appears when combining CUDA + OMP in hybrid calculations. In that case the energy becomes a function of #OMP threads per rank. For your input I got (cp2k 8.0, revision 3e7b916, run on Piz Daint)</div><div class=""></div><div class=""> <wbr class=""> no cuda OMP_NUM_THREADS = 1 OMP_NUM_THREADS = 3 OMP_NUM_THREADS = 6<br class=""> Exchange-correlation energy: -433.84964308969535 -433.84964308969302 -435.33426106395467 <wbr class=""> -435.96513615032325<br class=""> Hartree-Fock Exchange energy: -127.87395928499694 -127.87395928499325 -125.97109874333140 <wbr class=""> -125.24809389970088<br class=""> Total energy: <wbr class=""> -1976.39722899739672 -1976.39722899739013 -1975.95046919253809 <wbr class=""> -1975.87080541858177</div><div class=""><br class=""></div><div class="">Without OMP parallelization the energies agrees with the calculation without CUDA accelleration. Increasing OMP_NUM_THREADS beyond 1 increases the Hartree-Fock Exchange energy.</div><div class="">Apparently you have to disable OMP to obtain correct results. This is obviously not very satisfying and I hope this gets fixed. I see that you used 1 MPI/12 OMP ranks per node. Try increasing the number of MPI ranks per node. To do so you have to set <br class=""></div><div class="">export CRAY_CUDA_MPS=1 in the submission script.<br class=""></div><div class=""><br class=""></div><div class="">I hope this helps.</div><div class=""><br class=""></div><div class="">Best,</div><div class="">Fabian<br class=""></div></div></blockquote></div></div><div class=""><br class="webkit-block-placeholder"></div>
-- <br class="">
You received this message because you are subscribed to the Google Groups "cp2k" group.<br class="">
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:cp...@googlegroups.com" class="">cp...@googlegroups.com</a>.<br class="">
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/cp2k/048030dd-6532-4fc5-b127-b7a5b017dad7%40googlegroups.com?utm_medium=email&utm_source=footer" class="">https://groups.google.com/d/msgid/cp2k/048030dd-6532-4fc5-b127-b7a5b017dad7%40googlegroups.com</a>.<br class="">
</div></blockquote></div><br class=""></div></body></html>