<blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;"><blockquote class="gmail_quote" style="margin:0;margin-left:0.8ex;border-left:1px #ccc solid;padding-left:1ex"><p>I believe we don't (or shouldn't) use such calls. Also, you can<br>explicitly make CP2K call mpi_init_thread() instead of mpi_init()<br>(change .TRUE. to .FALSE. on line 479 of src/message_passing.F)--the<br>current default is a workaround for de facto MPI threading behavior but<br>is technically wrong. Also, I just committed a patch to declare the use<br>of MPI funneled mode instead of general mode.</p></blockquote><div>i tried that, but now cp2k quits right away with a confusing error:</div><div><br></div><div> CP2K| MPI error 0 in Inadequate level of thread support is provided. : MPI_SUCCESS: no errors</div><div> CP2K| Abnormal program termination, stopped by process number 0</div></blockquote><div><br></div><div>ok. i figured that one out by myself. i needed to compile a new MPI </div><div>version that has MPI-threads enabled (which wasn't by default).</div><div><br></div><blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;"><blockquote class="gmail_quote" style="margin:0;margin-left:0.8ex;border-left:1px #ccc solid;padding-left:1ex"><p>> also, can you comment on what these two defines are for?<br>> -D__USE_CP2K_TRACE -D__HAS_smm_dnn </p><p>__USE_CP2K_TRACE uses CP2K's stack trace infrastructure on failed<br>assertions in the DBCSR sparse matrix (sub)library. To use it you must<br>also add the timings_mp.o module to the LIBS variable (used to get<br>around this and some other circular dependencies). While using it or<br>not should not affect behavior, there are OpenMP statements in stack<br>tracing and timing.</p></blockquote><div>i have so far tried without.</div></blockquote><div><br></div><div>i've tried with -D__USE_CP2K_TRACE and without. no difference.</div><blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;"><blockquote class="gmail_quote" style="margin:0;margin-left:0.8ex;border-left:1px #ccc solid;padding-left:1ex"><p>> this one is a bit of a surprise, since already gfortran 4.3.x claims<br>> to be OpenMP 3 compliant.<br>> -D__HAS_NO_OMP_3</p><p>I used gfortran 4.3 until recently (where psmp also worked last time I<br>checked) and didn't update the defines...</p></blockquote></blockquote><div>and it appears that -D__HAS_NO_OMP_3 seems to help with cp2k.ssmp</div><div><br></div><blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;"><div>for good measure, i am linking with the serial version</div><div>now, but that didn't change the behavior.</div></blockquote><div><br></div><div>yep. serial or threaded MKL doesn't make a difference.</div><div><br></div><div>i am now down to two cases:</div><div>cp2k.ssmp works fine with the development code for as long</div><div>as -D__HAS_NO_OMP_3 is set. the same is true for cp2k.psmp</div><div>if as long as i use only 1 MPI task. cp2k.psmp is also working</div><div>fine if i use only 1 OpenMP thread and multiple MPI tasks, but</div><div>using multiple MPI tasks _and_ OpenMP threads gives me random</div><div>numbers. for example the H2O-32.inp benchmark with two MPI and 6 OpenMP gives:</div><div><br></div><div><div> ----------------------------------- OT ---------------------------------------</div><div><br></div><div> Step Update method Time Convergence Total energy Change</div><div> ------------------------------------------------------------------------------</div><div> 1 OT DIIS 0.15E+00 1.0 1942.49460508 -33766612.6904597133 -3.38E+07</div><div> 2 OT DIIS 0.15E+00 1.0 605.50197323 ******************** -2.64E+09</div><div> 3 OT DIIS 0.15E+00 1.1 160.86073279 4380968.2349330271 2.68E+09</div><div> 4 OT DIIS 0.15E+00 1.1 164.34481270 865533.1852639467 -3.52E+06</div><div> 5 OT DIIS 0.15E+00 1.1 428.43636015 -30478863.0777326487 -3.13E+07</div><div> 6 OT DIIS 0.15E+00 1.1 1094.51147521 3880172.3745214189 3.44E+07</div><div> 7 OT DIIS 0.15E+00 1.1 33.84945332 ******************** -1.37E+08</div><div> 8 OT DIIS 0.15E+00 1.1 164.71992666 -2837998.3761878917 1.30E+08</div><div> 9 OT SD 0.15E+00 1.1 2844.75780601 ******************** -1.30E+08</div><div> 10 OT SD 0.15E+00 1.1 3605.58561902 2964610.0015976029 1.36E+08</div><div> 11 OT DIIS 0.15E+00 1.1 209.53192021 796475.0800868375 -2.17E+06</div><div> 12 OT DIIS 0.15E+00 1.1 380.38070104 -26949187.3137775287 -2.77E+07</div></div><div><br></div><div>with 12 MPI and 1 OpenMP and the same binary i get the proper behavior.</div><div> ----------------------------------- OT ---------------------------------------<br></div><div><br></div><div> Step Update method Time Convergence Total energy Change</div><div> ------------------------------------------------------------------------------</div><div> 1 OT DIIS 0.15E+00 0.6 0.01909729 -529.3015728815 -5.29E+02</div><div> 2 OT DIIS 0.15E+00 0.8 0.01238424 -536.3043281413 -7.00E+00</div><div> 3 OT DIIS 0.15E+00 0.8 0.00874280 -540.9240856869 -4.62E+00</div><div> 4 OT DIIS 0.15E+00 0.8 0.00605219 -544.3264898596 -3.40E+00</div><div> 5 OT DIIS 0.15E+00 0.8 0.00458609 -546.1883072280 -1.86E+00</div><div> 6 OT DIIS 0.15E+00 0.8 0.00351827 -547.5582206572 -1.37E+00</div><div> 7 OT DIIS 0.15E+00 0.8 0.00273439 -548.5013690323 -9.43E-01</div><div> 8 OT DIIS 0.15E+00 0.8 0.00208753 -549.1681124465 -6.67E-01</div><div> 9 OT DIIS 0.15E+00 0.8 0.00176820 -549.4777508042 -3.10E-01</div><div> 10 OT DIIS 0.15E+00 0.8 0.00140542 -549.7607744914 -2.83E-01</div><div> 11 OT DIIS 0.15E+00 0.8 0.00119366 -549.9316698866 -1.71E-01</div><div> 12 OT DIIS 0.15E+00 0.8 0.00102681 -550.0679169205 -1.36E-01</div><div><br></div><div><br></div><div>could it be that there is some code path where some initializations that </div><div>are required for "clean" OpenMP behavior are not executed, if the MPI</div><div>rank /= 0?</div><div><br></div><div>this seems to be quite a generic problem and it puzzles me</div><div>why it may work on one installation and not another.</div><div><br></div><div>the only other significant difference that i see to your setup</div><div>is that you are using MPICH2 and i use OpenMPI (1.4.3), but</div><div>if that would make a difference, google will probably request </div><div>that this group will become adult only after my next post.</div><div><br></div><div>thanks for your help,</div><div> axel.</div><div><br></div><blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;"><div>just make sure we're on the same page. the input(s) that</div><div>are giving me the problems are the water benchmark inputs.</div><div>in tests/QS/benchmark/</div><div><br></div><div>the cp2k.?smp binaries work fine for OMP_NUM_THREADS=1,</div><div>but as soon as i enable more than one thread with cp2k.psmp</div><div>and multiple MPI tasks, i get NaNs or random energies.</div><div>cp2k.ssmp appears to be working for both intel 11.1 and gfortran 4.4.4</div><div><br></div><div>thanks,</div><div> axel.</div><div><br></div><blockquote class="gmail_quote" style="margin:0;margin-left:0.8ex;border-left:1px #ccc solid;padding-left:1ex"><p>Cheers,<br>Urban.</p><p>> <br>> in any case, i'll give it a try.<br>> <br>> <br>> thanks,<br>> axel.<br>> <br>> -- <br>> You received this message because you are subscribed to the Google<br>> Groups "cp2k" group.<br>> To post to this group, send email to <a>cp...@googlegroups.com</a>.<br>> To unsubscribe from this group, send email to cp2k<br>> +<a>uns...@googlegroups.com</a>.<br>> For more options, visit this group at<br>> <a href="http://groups.google.com/group/cp2k?hl=en" target="_blank">http://groups.google.com/<wbr>group/cp2k?hl=en</a>.</p><p><br></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p></blockquote></blockquote>