<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Monaco;
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.E-MailFormatvorlage19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.E-MailFormatvorlage20
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DE" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hi Chris<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">You can try to launch the jobs with I_MPI_DEBUG=5 set to get more insight how the processes are distributed and which interface is used. Btw, MKL should include FFTW3 and thus there
is no need for a separate installation of that lib.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Matthias<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><b>Von:</b> cp...@googlegroups.com <...@googlegroups.com> <b>
Im Auftrag von </b>Christmas Mwk<br>
<b>Gesendet:</b> Freitag, 2. August 2019 22:40<br>
<b>An:</b> cp2k <...@googlegroups.com><br>
<b>Betreff:</b> {Spam?} [CP2K:12029] Performance of CP2K for MPT vs IntelMPI<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hi all,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Recently I was trying to run H2O-64 benchmark on CIRRUS. I compiled CP2K versions "popt" and "psmp" with GCC 8.2, FFTW3, Libint, libxsmm and MKL 2019.3. For MPI I used MPT 2.18 and IntelMPI 18.0.5.274 (both available as modules on CIRRUS
so no problems with infiniband) for comparison. In my surprise, while on single node IntelMPI had better performance (59.5s) compared to what is shown in <a href="https://www.cp2k.org/performance:cirrus-h2o-64">cirrus-h2o-64</a>, when I tried to run it across
more than 1 node, eg 4 nodes(144 cores), the runtime wasn't scaling. However in the case of MPT the results were similar to what is published on the web. Observing the output timings, I saw that a significant portion of the time is spent on mp_wait_any and
mp_waitall_1 (around 21s out of 60s) in the case of IntelMPI across 4 nodes, while for MPT only around 6s is spent in these routines with overall runtime around 28s.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Initially I suspected that using IntelMPI might require some manual process pinning so I tried various options such as setting I_MPI_PIN_DOMAIN to compact, core etc. While there was some improvement in performance, these overheads in MPI
routines were still the same. I also considered IntelMPI 2017 but same performance obtained. Additionally, similar results are obtained for both "popt" and "psmp" with OMP threads set to 1. I assume that if there was a load imbalance issue then performance
for both MPT and IntelMPI would have been comparable but still not sure.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Is there anything that I am missing here or is this performance behaviour expected in the case of IntelMPI? If performance should be similar or comparable, could you please suggest how I can launch the executable using mpirun and IntelMPI?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thank you in advance. Any help would be much appreciated. I attach the arch files (popt files are similar) and run time results for the runs on 4 nodes for IntelMPI(compact and core), MPT and also the single node result for IntelMPI. Below,
examples of how the executable is launched are also provided.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Best,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Chris<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:black">Mpt<o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C1651C">export</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#2EAEBB"> OMP_NUM_THREADS=</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#B42419">1</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#2EAEBB"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2">/lustre/sw/cp2k/4.1.17462/cp2k/cp2k/exe/mpt/placement </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#B42419">1</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2">mpiexec_mpt </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-n</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#B42419">144</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-ppn</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#B42419">36</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> dplace </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-p</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> place.txt
/lustre/home/d167/s1887443/scc/cp2k/exe/broadwell-o2-libs-mpt/cp2k.psmp H2O-64.inp<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:black">Compact<o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C1651C">export</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#2EAEBB"> OMP_NUM_THREADS=</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#B42419">1</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#2EAEBB"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2">mpirun </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-n</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#B42419">144</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-ppn</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#B42419">36</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-env</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> I_MPI_PIN_DOMAIN
omp </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-env</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> I_MPI_PIN_ORDER compact </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-print-rank-map</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> /lustre/home/d167/s1887443/scc/cp2k/exe/broadwell-o2-libs/cp2k.psmp </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-i</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> H2O-64.inp </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-o</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> out.txt<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:black">Core<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C1651C">export</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#2EAEBB"> OMP_NUM_THREADS=</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#B42419">1</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#2EAEBB"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C1651C">export</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#2EAEBB"> I_MPI_PIN_DOMAIN=</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2">core</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#2EAEBB"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2">mpirun </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-n</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#B42419">144</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-ppn</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#B42419">36</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-genv</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#2EAEBB">I_MPI_PIN_DOMAIN</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2">=core </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-genv</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#2EAEBB">OMP_NUM_THREADS</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2">=</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#B42419">1</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-print-rank-map</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> /lustre/home/d167/s1887443/scc/cp2k/exe/broadwell-o2-libs/cp2k.psmp </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-i</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> H2O-64.inp </span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-o</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> out.txt<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal">-- <br>
You received this message because you are subscribed to the Google Groups "cp2k" group.<br>
To unsubscribe from this group and stop receiving emails from it, send an email to
<a href="mailto:cp2k+unsu...@googlegroups.com">cp2k+unsu...@googlegroups.com</a>.<br>
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/cp2k/885a737b-0798-468e-b7c5-eb46c4bf5255%40googlegroups.com?utm_medium=email&utm_source=footer">
https://groups.google.com/d/msgid/cp2k/885a737b-0798-468e-b7c5-eb46c4bf5255%40googlegroups.com</a>.<o:p></o:p></p>
</div>
</body>
</html>