<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
        {font-family:"Cambria Math";
        panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
        {font-family:Calibri;
        panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
        {font-family:Monaco;
        panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0cm;
        margin-bottom:.0001pt;
        font-size:12.0pt;
        font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
        {mso-style-priority:99;
        color:blue;
        text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
        {mso-style-priority:99;
        color:purple;
        text-decoration:underline;}
p
        {mso-style-priority:99;
        mso-margin-top-alt:auto;
        margin-right:0cm;
        mso-margin-bottom-alt:auto;
        margin-left:0cm;
        font-size:12.0pt;
        font-family:"Times New Roman",serif;}
p.msonormal0, li.msonormal0, div.msonormal0
        {mso-style-name:msonormal;
        mso-margin-top-alt:auto;
        margin-right:0cm;
        mso-margin-bottom-alt:auto;
        margin-left:0cm;
        font-size:12.0pt;
        font-family:"Times New Roman",serif;}
span.EmailStyle19
        {mso-style-type:personal-reply;
        font-family:"Calibri",sans-serif;
        color:#1F497D;}
.MsoChpDefault
        {mso-style-type:export-only;
        font-family:"Calibri",sans-serif;}
@page WordSection1
        {size:612.0pt 792.0pt;
        margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
        {page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Chris<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The MPI_startup looks also fine to me at least as far as I can assess it. You can show it to one of your admins who will know if that is the suggested
 setting for your box.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I don’t think that the FFTW3 installation will affect the scaling assuming that you have installed the lib properly. My point was that MKL provides
 already a tuned FFTW3 version. Your installation won’t hardly perform better and you spare the local installation of that lib.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Matthias<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> cp...@googlegroups.com <...@googlegroups.com>
<b>On Behalf Of </b>Christmas Mwk<br>
<b>Sent:</b> Samstag, 3. August 2019 17:36<br>
<b>To:</b> cp2k <...@googlegroups.com><br>
<b>Subject:</b> Re: {Spam?} [CP2K:12033] Performance of CP2K for MPT vs IntelMPI<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
Hi,<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="DE"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">You can try to launch the jobs with I_MPI_DEBUG=5</span><span lang="DE"><o:p></o:p></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal">I 've tried that and you can find the debug output attached. In the MPI_startup() section it seems that each process is placed correctly on each core and no oversubscription or anything like that is observed. However I'm not sure what the
 last line means (I_MPI_PIN_MAPPING). Could you please guide me through how to make CP2K with IntelMPI scale across multiple nodes? Or is the performance as expected? I launch the executable by:<o:p></o:p></p>
</div>
<div>
<p style="font-stretch: normal;background-color:rgba(0, 0, 0, 0.85098)"><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C1651C">export</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#2EAEBB"> OMP_NUM_THREADS=</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#B42419">1</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#2EAEBB"><o:p></o:p></span></p>
<p style="font-stretch: normal;background-color:rgba(0, 0, 0, 0.85098)"><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C1651C">export</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#2EAEBB"> I_MPI_PIN_DOMAIN=</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2">core</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#2EAEBB"><o:p></o:p></span></p>
<p style="font-stretch: normal;background-color:rgba(0, 0, 0, 0.85098)"><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C1651C">export</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#2EAEBB"> I_MPI_DEBUG=</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#B42419">5</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#2EAEBB"><o:p></o:p></span></p>
<p style="font-stretch: normal;background-color:rgba(0, 0, 0, 0.85098)"><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2">mpirun
</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-n</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2">
</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#B42419">144</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2">
</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-ppn</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2">
</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#B42419">36</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2">
</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-genv</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2">
</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#2EAEBB">I_MPI_DEBUG</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2">=</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#B42419">5</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2">
</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-genv</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2">
</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#2EAEBB">I_MPI_PIN_DOMAIN</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2">=core
</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-genv</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2">
</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#2EAEBB">OMP_NUM_THREADS</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2">=</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#B42419">1</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2">
</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-print-rank-map</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> /lustre/home/d167/s1887443/scc/cp2k/exe/broadwell-o2-libs/cp2k.psmp
</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-i</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> H2O-64.inp
</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#C814C9">-o</span><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#F2F2F2"> out.txt<o:p></o:p></span></p>
<p style="font-stretch: normal;background-color:rgba(0, 0, 0, 0.85098)"><span style="font-size:7.5pt;font-family:"Monaco",serif;color:#4A00FF">~                                                                                                                 
                                                                                        <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB">MKL should include FFTW3 and thus there is no need for a separate installation of that lib.</span><span lang="DE"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-GB"> </span><span lang="DE"><o:p></o:p></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal">Will the choice of FFTW3 affect the scaling performance though?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Best, <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Chris<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal">-- <br>
You received this message because you are subscribed to the Google Groups "cp2k" group.<br>
To unsubscribe from this group and stop receiving emails from it, send an email to
<a href="mailto:cp2k+unsu...@googlegroups.com">cp2k+unsu...@googlegroups.com</a>.<br>
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/cp2k/4b29d19c-f13e-4010-9405-b5be31ac0e94%40googlegroups.com?utm_medium=email&utm_source=footer">
https://groups.google.com/d/msgid/cp2k/4b29d19c-f13e-4010-9405-b5be31ac0e94%40googlegroups.com</a>.<o:p></o:p></p>
</div>
</body>
</html>