<br><br>On Thursday, January 26, 2012 3:50:00 AM UTC-5, Christiane Pousa Ribeiro wrote:<div> </div><blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>yes, this kind of behavior is what i would have expected.</div><div>this should also help with the internal threading in OpenMPI.</div> </blockquote><div><br>The main goal is to avoid memory allocations and access from different MPIs on remote NUMA nodes.</div></div></blockquote><div><br></div><div>i know. ;)</div><div> </div><blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;"><div class="gmail_quote"><div> But, If you want to pin also threads you can try the Linear strategy, which will pin process and threads. <br></div></div></blockquote><div><br></div><div>unlike with binding MPI tasks to "NUMA units",</div><div>i didn't see a significant difference in performance.</div><div><br></div><blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;"><div class="gmail_quote"><div></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><br></div></div><div>please have a look at the attached file. you'll see that there</div> <div>are some entries that don't look right. particularly the node</div><div>names are all that of MPI rank 0.</div></blockquote><div><br>I did some changes to fix this. Could you try the latest version of CP2K?<br></div></div></blockquote><div><br></div><div>yes. updated, compiled and tested. it gives the output that i expect now.</div><div> </div><blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;"><div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><br></div></div><div>yes. our MPI installation is configured by default to have a 1:1 core to MPI</div> <div>rank mapping (since there is practically nobody yet using MPI+OpenMP)</div><div>with memory affinity for giving people the best MPI-only performance.</div><div><br></div></blockquote><div><br>Ok. So, for threads, even with this installation you can not specify their cores?<br></div></div></blockquote><div><br></div><div>it is not alone a matter of "want". the majority of users that i am working</div><div>with doesn't care (well, they do care if things run faster, but they don't</div><div>care so much, if it looks/sounds/is complicated). with hiding most of</div><div>the complexity in a script and having it not allow unreasonable choices,</div><div>i don't get the maximal flexibility, but all i need to do is to tell people:</div><div>just use this wrapper and it'll work. if every application would be hardware</div><div>topology aware and adjust itself as needed, that is even better and that</div><div>is why i am trying to compile cp2k this way.</div><div><br></div><blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div></div><div>at the end of the attached file i include a copy of the wrapper script,</div> <div>that is OpenMPI specific (since that is the only MPI library installed).</div></blockquote><div><br>thanks for the script. <br> <br></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div><br></div><div>overall, it looks to me like that default settings are giving a desirable </div><div>processor and memory affinity (which is great) that is consistent with</div><div>the best settings i could get using my wrapper script, but the diagnostics</div> <div>seems to be off and may be confusing people, particularly technical</div><div>support in computing centers, that are often too literal and assume </div><div>that any software is always giving 100% correct information. ;-)</div> </blockquote><div><br>Now, it should work :) Let me know if you find new bugs.<br></div></div></blockquote><div><br></div><div>thanks a lot. much appreciated. will let you know,</div><div>if i run across any additional problems.</div><div> </div><blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;"><div class="gmail_quote"><div>Considering your machine, the cores number problem comes from the fact that I was using the number that the OS gives to the cores. Now, I'm using the logical ones. BTW, is your machine intel? <br></div></div></blockquote><div><br></div><div>We have both. Intel and AMD (which is forcing me to use compiler settings,</div><div>that are compatible with a common subset of both). overall the AMD ones</div><div>benefit the most from using processor and memory affinity, but i was surprised</div><div>how much impact it has on the X5677 Intel CPUs (quad-core westmere ep</div><div>with 3.5GHz). just proves that there is always something new to learn...</div><div> </div><div>thanks again,</div><div> axel.</div><div><br></div><blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;"><div class="gmail_quote"><div> <br></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><br></div><div>cheers,</div><div> axel.</div><br clear="all"></blockquote></div> <br>cheers,<br><br>Christiane Pousa Ribeiro<br> <br> </blockquote>