<div dir="ltr">Dear Fernan,<div>Thanks for the new files.</div><div>OK, let me summarize:</div><div>1) The GNU+OpenMPI version of the code is terribly slow</div><div>2) The Intel (MPI + MKL) version hangs</div><div><br></div><div>Now, let's start with the first problem. I took your input file and executed on my system with 24 MPI ranks and 1 OpenMP thread. I'm using GNU 5.3 + OpenMPI without any special optimizations/libraries. The first thing I noticed is that the initial values are different (w.r.t log.out_openmpi):</div><div><br></div><div>Yours:</div><div><div>TOTAL NUMBERS AND MAXIMUM NUMBERS</div><div><br></div><div> Total number of - Atomic kinds: 4</div><div> - Atoms: 950</div><div> - Shell sets: 1900</div><div> - Shells: 4094</div><div> - Primitive Cartesian functions: 4750</div><div> - Cartesian basis functions: 10348</div><div> - Spherical basis functions: 9726</div><div><br></div><div> Maximum angular momentum of- Orbital basis functions: 2</div><div> - Local part of the GTH pseudopotential: 2</div><div> - Non-local part of the GTH pseudopotential: 2</div></div><div><br></div><div>Mine:<br></div><div><div> TOTAL NUMBERS AND MAXIMUM NUMBERS</div><div><br></div><div> Total number of - Atomic kinds: 3</div><div> - Atoms: 938</div><div> - Shell sets: 1876</div><div> - Shells: 3434</div><div> - Primitive Cartesian functions: 4690</div><div> - Cartesian basis functions: 7480</div><div> - Spherical basis functions: 7170</div><div><br></div><div> Maximum angular momentum of- Orbital basis functions: 2</div><div> - Local part of the GTH pseudopotential: 2</div><div> - Non-local part of the GTH pseudopotential: 0</div></div><div><br></div><div>Checking with your log.out, I see that you are using CP2K 4.1 and the output is different, but I see that some values are equal to mine, for instance:</div><div><br></div><div><br></div><div><div>Number of electrons: 2168</div><div> Number of occupied orbitals: 1084</div><div> Number of molecular orbitals: 1084</div><div><br></div><div> Number of orbital functions: 7170</div><div> Number of independent orbital functions: 7170</div></div><div><br></div><div>These values are very important for the performance. For the GNU+OpenMPI they are bigger, that means that you can expect slower performance there.</div><div><br></div><div>At this point, I strongly suggest to run the regtests to check your installation. Make sure </div><div><font size="2" style="background-color: rgb(255, 255, 255);" color="#000000">Then, I can suggest you run a smaller test (you can take some a test under <span style="font-variant-ligatures: no-common-ligatures; font-family: Monaco;">tests/QS/benchmark/</span><span style="font-variant-ligatures: no-common-ligatures; font-family: Monaco;">H2O-32.inp) and run with a single rank, so that you can do a fast comparison without MPI. If this is reasonable, then you can move to more ranks. </span></font></div><div><font size="2" style="background-color: rgb(255, 255, 255);" color="#000000"><span style="font-variant-ligatures: no-common-ligatures; font-family: Monaco;">Another suggestion is to check how many cores are really involved during the execution (you can use htop).</span></font></div><div><font size="2" style="background-color: rgb(255, 255, 255);" color="#000000"><span style="font-variant-ligatures: no-common-ligatures; font-family: Monaco;"><br></span></font></div><div><font size="2" style="background-color: rgb(255, 255, 255);" color="#000000"><span style="font-variant-ligatures: no-common-ligatures; font-family: Monaco;">Alfio</span></font></div>
<style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Monaco; color: #f4f4f4; background-color: #000000; background-color: rgba(0, 0, 0, 0.85)}
span.s1 {font-variant-ligatures: no-common-ligatures}
</style>
<style type="text/css"> p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Monaco; color: #f4f4f4; background-color: #000000; background-color: rgba(0, 0, 0, 0.85)} span.s1 {font-variant-ligatures: no-common-ligatures} </style>
<div><br></div><div><br></div><div><br></div><div><br>Il giorno mercoledì 21 marzo 2018 14:22:54 UTC+1, Fernan Saiz ha scritto:<blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;"><div dir="ltr"><div><div><div>Dear Alfio,<br></div>2) The log.out file was written using the Intel Compilers (MPI and MKL). Please find attached file a new log.out_openmpi, which was built with libxsmm where I show the times that are ridiculous long. When the
cp2k
version compiled with
Intel runs fine, these times are between 6 and 10 seconds, whereas with openmpi they are between 28 and 76 seconds, even if I increase the number of cores.<br><br></div>I also attached my arch file used for openmpi.<br><br></div>Best regards,<br></div><div><br clear="all"><div><div><div dir="ltr">- Fernan<br></div></div></div>
<br><div class="gmail_quote">On Wed, Mar 21, 2018 at 10:57 AM, Alfio Lazzaro <span dir="ltr"><<a href="javascript:" target="_blank" gdf-obfuscated-mailto="bzVDWHmxAwAJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">alfi...@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Dear Fernan,<div>we are working on making a list of supported compilers for CP2K (see <a href="https://www.cp2k.org/dev:compiler_support" target="_blank" rel="nofollow" onmousedown="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.cp2k.org%2Fdev%3Acompiler_support\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq7WVu4QkGpDyG7P2tzgPChzf4YQ';return true;" onclick="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.cp2k.org%2Fdev%3Acompiler_support\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHq7WVu4QkGpDyG7P2tzgPChzf4YQ';return true;">https://www.cp2k.org/dev:<wbr>compiler_support</a> ).</div><div>Now, I'm confused about your findings:</div><div><br></div><div>1) "CRAY compilers on the UK's ARCHER supercomputer" ==> Do they really compile CP2K with the Cray CCE compiler? This is somehow surprising to me since we found that CCE is broken...</div><div><br></div><div>2) "I have compiled cp2k with gcc and openmpi modules, but my code is much slower than that built with Intel Compilers" ==> This is strange too. I see that you are not using libxsmm, i.e.</div><div><br></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">DBCSR| Multiplication driver <wbr> BLAS</span><br></span></div><div><br></div><div>Libxsmm will give a much better performance. Where is BLAS coming from? MKL for GCC and Intel compilations? Could you elaborate a bit more on this comparison?</div><div><br></div><div>3) Before any production run, it would be a good idea to test the CP2K installation. Have you executed the regtests? (<a href="https://www.cp2k.org/dev:regtesting" target="_blank" rel="nofollow" onmousedown="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.cp2k.org%2Fdev%3Aregtesting\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG1cOJGd_1hhwVDqInibboNBS5zMw';return true;" onclick="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.cp2k.org%2Fdev%3Aregtesting\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG1cOJGd_1hhwVDqInibboNBS5zMw';return true;">https://www.cp2k.<wbr>org/dev:regtesting</a> )</div><div><br></div><div>Best regards,</div><div><br></div><div>Alfio</div><div><div><div><br><br>Il giorno mercoledì 21 marzo 2018 03:31:29 UTC+1, Wei Lai ha scritto:<blockquote class="gmail_quote" style="margin:0;margin-left:0.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Were you using Intel compilers with OpenMPI or MVAPICH? I had the same issue. Switching to Intel MPI solved the problem.<div><br></div><div>Wei<br><br>On Monday, March 19, 2018 at 2:35:35 PM UTC-4, Fernan Saiz wrote:<blockquote class="gmail_quote" style="margin:0;margin-left:0.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Dear All,<br>I have been experiencing a problem with cp2k for large systems of around 1,000 atoms, in which this code prints no data at the beginning of SCF cycle as shown in the attached file log.out. I am using a cp2k version compiled with Intel compilers 2017 at my institution's HPC systems. It is strange to me that sometimes cp2k runs fine with a different set of nuclei positions while keeping untouched the rest of the parameters. However, I need to restart my run (see ape-water.inp) to continue with the NVT simulation for several ps. It also strikes me that I have never faced this problem when using a version built with CRAY compilers on the UK's ARCHER supercomputer. As an alternative, I have compiled cp2k with gcc and openmpi modules (I received your helped in this forum a few days ago for this compilation), but my code is much slower than that built with Intel Compilers, which makes it not suitable for my purposes. I have also played with the OpenMP vs MPI load in the PBS file, but I got no luck. Therefore, I would really appreciate some advice on how to correct this problem, if possible.<br><br>Best regards,<br> - Fernan Saiz, PhD<br>Department of Chemistry<br>Imperial College London<br></div></blockquote></div></div></blockquote></div></div></div></div><div><div>
<p></p>
-- <br>
You received this message because you are subscribed to a topic in the Google Groups "cp2k" group.<br>
To unsubscribe from this topic, visit <a href="https://groups.google.com/d/topic/cp2k/XzwfDfJtW08/unsubscribe" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/topic/cp2k/XzwfDfJtW08/unsubscribe';return true;" onclick="this.href='https://groups.google.com/d/topic/cp2k/XzwfDfJtW08/unsubscribe';return true;">https://groups.google.com/d/<wbr>topic/cp2k/XzwfDfJtW08/<wbr>unsubscribe</a>.<br>
To unsubscribe from this group and all its topics, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="bzVDWHmxAwAJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">cp2k+uns...@googlegroups.<wbr>com</a>.<br>
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="bzVDWHmxAwAJ" rel="nofollow" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">cp...@googlegroups.com</a>.<br>
Visit this group at <a href="https://groups.google.com/group/cp2k" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/group/cp2k';return true;" onclick="this.href='https://groups.google.com/group/cp2k';return true;">https://groups.google.com/<wbr>group/cp2k</a>.<br>
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;">https://groups.google.com/d/<wbr>optout</a>.<br>
</div></div></blockquote></div><br></div>
</blockquote></div></div>