<div>If you are using the Intel MKL library, you may consider changing the version of the library. We have observed (with Qbox, another code) memory leaks when using <br></div><div><div style="font-weight: 400; font-style: normal; text-decoration: none;">(1) intel/19.1.1</div><div style="font-weight: 400; font-style: normal; text-decoration: none;">(2) intelmpi/2019.up7+intel-19.1.1</div><div style="font-weight: 400; font-style: normal; text-decoration: none;">(3) mkl/2020.up1</div><div style="font-weight: 400; font-style: normal; text-decoration: none;"><br></div><div style="font-weight: 400; font-style: normal; text-decoration: none;">and the leaks disappeared when switching to older versions<br></div><div style="font-weight: 400; font-style: normal; text-decoration: none;"><div style="font-weight: 400; font-style: normal; text-decoration: none;">(1) intel/18.0<br>(2) intelmpi/2018.2.199+intel-18.0<br>(3) mkl/2018.up2</div></div><div style="font-weight: 400; font-style: normal; text-decoration: none;"><br></div><div style="font-weight: 400; font-style: normal; text-decoration: none;">There is also a documented risk of memory leak when using the fast matrix multiply in the oneAPI Intel MKL library:</div><div style="font-weight: 400; font-style: normal; text-decoration: none;"><a href="https://www.intel.com/content/www/us/en/develop/documentation/onemkl-linux-developer-guide/top/managing-performance-and-memory/using-memory-functions/avoiding-memory-leaks-in-onemkl.html">https://www.intel.com/content/www/us/en/develop/documentation/onemkl-linux-developer-guide/top/managing-performance-and-memory/using-memory-functions/avoiding-memory-leaks-in-onemkl.html</a></div><div style="font-weight: 400; font-style: normal; text-decoration: none;"><br></div><div style="font-weight: 400; font-style: normal; text-decoration: none;">Francois <br></div></div><br><div class="gmail_quote"><div dir="auto" class="gmail_attr">On Friday, September 23, 2022 at 9:00:17 AM UTC-7 Matthias Krack wrote:<br/></div><blockquote class="gmail_quote" style="margin: 0 0 0 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"> <div lang="en-CH" link="blue" vlink="purple" style="word-wrap:break-word"> <div> <p class="MsoNormal"><span lang="DE-CH">Dear Matthew<u></u><u></u></span></p> <p class="MsoNormal"><span lang="DE-CH"><u></u> <u></u></span></p> <p class="MsoNormal"><span lang="EN-US">The memory growth for v2022.1 looks not too bad and it might be fine to survive longer runs.<u></u><u></u></span></p> <p class="MsoNormal"><span lang="EN-US">I remember issues with memory leaks caused by the MPI implementation. Especially OpenMPI showed such problems in the past and that is why I used only MPICH for years, because leak checking in CP2K was impossible with OpenMPI. Have a look at this <a href="https://github.com/cp2k/cp2k/issues/1830" target="_blank" rel="nofollow" data-saferedirecturl="https://www.google.com/url?hl=en&q=https://github.com/cp2k/cp2k/issues/1830&source=gmail&ust=1664221936965000&usg=AOvVaw0nC4DpdeU4lcu3K7HEYCSI"> issue</a> from Jan this year for instance.<u></u><u></u></span></p> <p class="MsoNormal"><span lang="EN-US">The presence of memory leaks usually does not imply that the results are wrong.<u></u><u></u></span></p> <p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p> <p class="MsoNormal"><span lang="EN-US">Best regards<u></u><u></u></span></p></div></div><div lang="en-CH" link="blue" vlink="purple" style="word-wrap:break-word"><div> <p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p> <p class="MsoNormal"><span lang="EN-US">Matthias<u></u><u></u></span></p> <p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p> <div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm"> <p class="MsoNormal" style="margin-left:36.0pt"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">"<a href data-email-masked rel="nofollow">cp...@googlegroups.com</a>" <<a href data-email-masked rel="nofollow">cp...@googlegroups.com</a>> on behalf of Matthew Emerson <<a href data-email-masked rel="nofollow">mr...@uiowa.edu</a>><br> <b>Reply to: </b>"<a href data-email-masked rel="nofollow">cp...@googlegroups.com</a>" <<a href data-email-masked rel="nofollow">cp...@googlegroups.com</a>><br> <b>Date: </b>Friday, 23 September 2022 at 16:55<br> <b>To: </b>"<a href data-email-masked rel="nofollow">cp...@googlegroups.com</a>" <<a href data-email-masked rel="nofollow">cp...@googlegroups.com</a>><br> <b>Subject: </b>Re: [CP2K:17749] Possible Memory Leak<u></u><u></u></span></p> </div> <div> <p class="MsoNormal" style="margin-left:36.0pt"><u></u> <u></u></p> </div> <p class="MsoNormal" style="margin-right:0cm;margin-bottom:12.0pt;margin-left:36.0pt"> Dear Dr. Matthias, <br> <br> Sorry for the late response, I wanted to test these things carefully. Below is what I have found.<br> <br> All versions of CP2K that I have tested appear to still have a memory leak. The issue is less severe on version 2022.1, but as you can see, in the graph "CP2K-2022.1.png", the program keeps growing with time and our runs are long. For version 8.1, in graph "CP2K-8.1.png", the problem is much more severe (on the same hardware - 56 MPI ranks).<br> <br> My primary concern is with the correctness of the calculations. We can of course restart the program, but how do I know that a code that looks like it is leaking is not corrupting data. If you continue the run you started, I predict it will continue growing in memory usage. I thank you for your time.<br> <br> <br> Sincerely,<br> Matthew S. Emerson<u></u><u></u></p> </div></div><div lang="en-CH" link="blue" vlink="purple" style="word-wrap:break-word"><div><div> <div> <p class="MsoNormal" style="margin-left:36.0pt">On Wednesday, September 21, 2022 at 9:23:41 AM UTC-5 Matthias Krack wrote:<u></u><u></u></p> </div> </div></div></div><div lang="en-CH" link="blue" vlink="purple" style="word-wrap:break-word"><div><div><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm"> <div> <div> <p class="MsoNormal" style="margin-left:36.0pt"> <span lang="EN-US">Hi Matthew</span><u></u><u></u></p> <p class="MsoNormal" style="margin-left:36.0pt"> <span lang="EN-US"> </span><u></u><u></u></p> <p class="MsoNormal" style="margin-left:36.0pt"> <span lang="EN-US">I used <a href="https://github.com/cp2k/cp2k/blob/master/arch/Linux-gnu-x86_64.psmp" target="_blank" rel="nofollow" data-saferedirecturl="https://www.google.com/url?hl=en&q=https://github.com/cp2k/cp2k/blob/master/arch/Linux-gnu-x86_64.psmp&source=gmail&ust=1664221936965000&usg=AOvVaw2N-Icx42mAm1qH3FFqylJX"> this arch file</a> to build the current cp2k release version 2022.1 on our local cluster, basically by running</span><u></u><u></u></p> <p class="MsoNormal" style="margin-left:72.0pt"> <u></u><span style="font-size:10.0pt;font-family:Symbol"><span>·<span style="font:7.0pt "Times New Roman""> </span></span></span><u></u><span lang="FR-CH">source arch/Linux-gnu-x86_64.psmp</span><u></u><u></u></p> <p class="MsoNormal" style="margin-left:36.0pt"> <span lang="EN-US">in the main cp2k folder and then run make as proposed after the cp2k toolchain has been built successfully. This is also done for the continuous regression testing (see first two entries in the <a href="https://dashboard.cp2k.org/index.html" target="_blank" rel="nofollow" data-saferedirecturl="https://www.google.com/url?hl=en&q=https://dashboard.cp2k.org/index.html&source=gmail&ust=1664221936965000&usg=AOvVaw2Z9RiZ7nAw2f5aAKOIYKXd">CP2K dashboard</a>, just click on the “OK” link to see the details).</span><u></u><u></u></p> <p class="MsoNormal" style="margin-left:36.0pt"> <span lang="EN-US"> </span><u></u><u></u></p> <p class="MsoNormal" style="margin-left:36.0pt"> <span lang="FR-CH">HTH</span><u></u><u></u></p> <p class="MsoNormal" style="margin-left:36.0pt"> <span lang="FR-CH"> </span><u></u><u></u></p> <p class="MsoNormal" style="margin-left:36.0pt"> <span lang="FR-CH">Matthias</span><u></u><u></u></p> </div> </div> <div> <div> <p class="MsoNormal" style="margin-left:36.0pt"> <span lang="FR-CH"> </span><u></u><u></u></p> <div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm"> <p class="MsoNormal" style="margin-left:72.0pt"> <b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">"<span><a href data-email-masked rel="nofollow">cp...@googlegroups.com</a></span>" <<span><a href data-email-masked rel="nofollow">cp...@googlegroups.com</a></span>> on behalf of Matthew Emerson <<span><a href data-email-masked rel="nofollow">mr...@uiowa.edu</a></span>><br> <b>Reply to: </b>"<span><a href data-email-masked rel="nofollow">cp...@googlegroups.com</a></span>" <<span><a href data-email-masked rel="nofollow">cp...@googlegroups.com</a></span>><br> <b>Date: </b>Wednesday, 21 September 2022 at 15:38<br> <b>To: </b>"<span><a href data-email-masked rel="nofollow">cp...@googlegroups.com</a></span>" <<span><a href data-email-masked rel="nofollow">cp...@googlegroups.com</a></span>><br> <b>Subject: </b>Re: [CP2K:17727] Possible Memory Leak</span><u></u><u></u></p> </div> <div> <p class="MsoNormal" style="margin-left:72.0pt"> <u></u><u></u></p> </div> <p class="MsoNormal" style="margin-left:72.0pt"> Dear Dr. Matthias, <br> <br> Do you have an ARCH file for this build that you could point me to? I would like to build using the same settings to test at ORNL.<br> <br> Sincerely, <u></u><u></u></p> <div> <p class="MsoNormal" style="margin-bottom:12.0pt;margin-left:72.0pt"> Matthew S. Emerson<u></u><u></u></p> </div> </div> </div> <div> <div> <div> <div> <p class="MsoNormal" style="margin-left:72.0pt"> On Wednesday, September 21, 2022 at 4:00:21 AM UTC-5 Matthias Krack wrote:<u></u><u></u></p> </div> </div> </div> </div> <div> <div> <div> <blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt"> <div> <div> <p class="MsoNormal" style="margin-left:72.0pt"> <span lang="EN-US">Hello Matthew</span><u></u><u></u></p> <p class="MsoNormal" style="margin-left:72.0pt"> <span lang="EN-US"> </span><u></u><u></u></p> <p class="MsoNormal" style="margin-left:72.0pt"> <span lang="EN-US">I have run your case on our local compute cluster with CP2K v2022.1 (gnu 11.2.0, OpenMPI 4.1.3) using 144 CPU cores. I observe only a small increase in memory usage after the usual initial growth during the first MD steps (see attached plot).</span><u></u><u></u></p> <p class="MsoNormal" style="margin-left:72.0pt"> <span lang="EN-US"> </span><u></u><u></u></p> <p class="MsoNormal" style="margin-left:72.0pt"> <span lang="EN-US">Best regards</span><u></u><u></u></p> <p class="MsoNormal" style="margin-left:72.0pt"> <span lang="EN-US"> </span><u></u><u></u></p> <p class="MsoNormal" style="margin-left:72.0pt"> <span lang="EN-US">Matthias</span><u></u><u></u></p> <p class="MsoNormal" style="margin-left:72.0pt"> <span lang="EN-US"> </span><u></u><u></u></p> </div> </div> </blockquote> </div> </div> </div> </blockquote></div></div></div><div lang="en-CH" link="blue" vlink="purple" style="word-wrap:break-word"><div><div><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm"><div> <div> <div> <blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt"> <div> <div> <p class="MsoNormal" style="margin-left:72.0pt"> <b><span lang="EN-GB" style="border:solid windowtext 1.0pt;padding:0cm">Error! Filename not specified.</span></b><u></u><u></u></p> </div> </div> </blockquote> </div> </div> </div></blockquote></div></div></div><div lang="en-CH" link="blue" vlink="purple" style="word-wrap:break-word"><div><div><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm"> <div> <div> <div> <blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt"> <div> <div> <p class="MsoNormal" style="margin-left:72.0pt"> <span lang="EN-US"> </span><u></u><u></u></p> <div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm"> <p class="MsoNormal" style="margin-left:108.0pt"> <b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">"<span><a href data-email-masked rel="nofollow">cp...@googlegroups.com</a></span>" <<span><a href data-email-masked rel="nofollow">cp...@googlegroups.com</a></span>> on behalf of Matthew Emerson <<span><a href data-email-masked rel="nofollow">mr...@uiowa.edu</a></span>><br> <b>Reply to: </b>"<span><a href data-email-masked rel="nofollow">cp...@googlegroups.com</a></span>" <<span><a href data-email-masked rel="nofollow">cp...@googlegroups.com</a></span>><br> <b>Date: </b>Tuesday, 20 September 2022 at 23:21<br> <b>To: </b>"<span><a href data-email-masked rel="nofollow">cp...@googlegroups.com</a></span>" <<span><a href data-email-masked rel="nofollow">cp...@googlegroups.com</a></span>><br> <b>Subject: </b>[CP2K:17721] Possible Memory Leak</span><u></u><u></u></p> </div> <div> <p class="MsoNormal" style="margin-left:108.0pt"> <u></u><u></u></p> </div> <p class="MsoNormal" style="margin-left:108.0pt"> Dear CP2K Developers/Community, <u></u><u></u></p> <div> <p class="MsoNormal" style="margin-left:108.0pt"> <u></u><u></u></p> </div> <div> <p class="MsoNormal" style="margin-left:108.0pt"> I have attached an input file which I believe shows an example of a possible memory leak in CP2K. It is a typical NVT DFT simulation of molten MgCl2 with PBE-D3 dispersion corrections. <u></u><u></u></p> </div> <div> <p style="margin-left:108.0pt">I have tried well-tested CP2K builds on our local cluster (v6.1, v8.1, v2022.1), our university supercomputer (v6.1, v8.1), and even the system-wide installations of CP2K on Cori at Oak Ridge National Lab (v8.1 and v9.1) and memory usage grows linear with time until either the node locks up/dies from insufficient memory usage or the job dies from maximum walltime (ORNL). I've done enough testing that I can almost tell how many MD steps before the job will die for a given machine with X amount of RAM and Y amount of MPI ranks. <br> <br> I normally wouldn’t email about things like this but I’ve tried multiple combinations (w/unit-testing) of GCC, OpenMPI, OpenBLAS/MKL, etc. and nothing seems to work. I am hoping this is simply an input file issue or my own error.<u></u><u></u></p> <p style="margin-left:108.0pt"> <u></u><u></u></p> <p style="margin-left:108.0pt">Any help will be much appreciated.<br> <br> Matthew S. Emerson<br> Margulis Research Group<br> Department of Chemistry<br> The University of Iowa<u></u><u></u></p> </div> </div> </div> <div> <div> <p class="MsoNormal" style="margin-bottom:12.0pt;margin-left:108.0pt"> -- <br> You received this message because you are subscribed to the Google Groups "cp2k" group.<br> To unsubscribe from this group and stop receiving emails from it, send an email to <span><a href data-email-masked rel="nofollow">cp2k+uns...@googlegroups.com</a></span>.<br> To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/cp2k/0c20a8f2-3658-4429-a2e9-7f4aa0edb321n%40googlegroups.com?utm_medium=email&utm_source=footer" target="_blank" rel="nofollow" data-saferedirecturl="https://www.google.com/url?hl=en&q=https://groups.google.com/d/msgid/cp2k/0c20a8f2-3658-4429-a2e9-7f4aa0edb321n%2540googlegroups.com?utm_medium%3Demail%26utm_source%3Dfooter&source=gmail&ust=1664221936966000&usg=AOvVaw3UEq0iP27NCQVgTJdQMy5j"> https://groups.google.com/d/msgid/cp2k/0c20a8f2-3658-4429-a2e9-7f4aa0edb321n%40googlegroups.com</a>.<u></u><u></u></p> </div> </div> </blockquote> </div> </div> </div> <div> <div> <p class="MsoNormal" style="margin-left:72.0pt"> -- <br> You received this message because you are subscribed to the Google Groups "cp2k" group.<br> To unsubscribe from this group and stop receiving emails from it, send an email to <span><a href data-email-masked rel="nofollow">cp2k+uns...@googlegroups.com</a></span>.<u></u><u></u></p> </div> </div> <div> <div> <p class="MsoNormal" style="margin-bottom:12.0pt;margin-left:72.0pt"> To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/cp2k/c69d3c0d-e3d2-489c-8040-19ac7c4e396en%40googlegroups.com?utm_medium=email&utm_source=footer" target="_blank" rel="nofollow" data-saferedirecturl="https://www.google.com/url?hl=en&q=https://groups.google.com/d/msgid/cp2k/c69d3c0d-e3d2-489c-8040-19ac7c4e396en%2540googlegroups.com?utm_medium%3Demail%26utm_source%3Dfooter&source=gmail&ust=1664221936966000&usg=AOvVaw2Rsd6tcCle2XVnpw5KMbjO"> https://groups.google.com/d/msgid/cp2k/c69d3c0d-e3d2-489c-8040-19ac7c4e396en%40googlegroups.com</a>.<u></u><u></u></p> </div> </div> </blockquote></div></div></div><div lang="en-CH" link="blue" vlink="purple" style="word-wrap:break-word"><div><p class="MsoNormal" style="margin-left:36.0pt">-- <br> You received this message because you are subscribed to the Google Groups "cp2k" group.<br> To unsubscribe from this group and stop receiving emails from it, send an email to <a href data-email-masked rel="nofollow">cp2k+uns...@googlegroups.com</a>.<br></p></div></div><div lang="en-CH" link="blue" vlink="purple" style="word-wrap:break-word"><div><p class="MsoNormal" style="margin-left:36.0pt"> To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/cp2k/3bb44147-b5e4-4edf-aa0c-14b6eb9195den%40googlegroups.com?utm_medium=email&utm_source=footer" target="_blank" rel="nofollow" data-saferedirecturl="https://www.google.com/url?hl=en&q=https://groups.google.com/d/msgid/cp2k/3bb44147-b5e4-4edf-aa0c-14b6eb9195den%2540googlegroups.com?utm_medium%3Demail%26utm_source%3Dfooter&source=gmail&ust=1664221936966000&usg=AOvVaw2JLJPi6oOq1y9ff8NOl1Mk"> https://groups.google.com/d/msgid/cp2k/3bb44147-b5e4-4edf-aa0c-14b6eb9195den%40googlegroups.com</a>.<br> <br> <u></u><u></u></p> </div> </div> </blockquote></div> <p></p> -- <br /> You received this message because you are subscribed to the Google Groups "cp2k" group.<br /> To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:cp2k+unsubscribe@googlegroups.com">cp2k+unsubscribe@googlegroups.com</a>.<br /> To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/cp2k/4ba4d6e5-86ab-4122-b8ff-0261e32c271cn%40googlegroups.com?utm_medium=email&utm_source=footer">https://groups.google.com/d/msgid/cp2k/4ba4d6e5-86ab-4122-b8ff-0261e32c271cn%40googlegroups.com</a>.<br />