[CP2K:3237] status of hybrid OpenMP+MPI version
Axel
akoh... at gmail.com
Tue May 10 02:27:21 UTC 2011
>
> I believe we don't (or shouldn't) use such calls. Also, you can
>> explicitly make CP2K call mpi_init_thread() instead of mpi_init()
>> (change .TRUE. to .FALSE. on line 479 of src/message_passing.F)--the
>> current default is a workaround for de facto MPI threading behavior but
>> is technically wrong. Also, I just committed a patch to declare the use
>> of MPI funneled mode instead of general mode.
>>
> i tried that, but now cp2k quits right away with a confusing error:
>
> CP2K| MPI error 0 in Inadequate level of thread support is provided. :
> MPI_SUCCESS: no errors
> CP2K| Abnormal program termination, stopped by process number 0
>
ok. i figured that one out by myself. i needed to compile a new MPI
version that has MPI-threads enabled (which wasn't by default).
> also, can you comment on what these two defines are for?
>> > -D__USE_CP2K_TRACE -D__HAS_smm_dnn
>>
>> __USE_CP2K_TRACE uses CP2K's stack trace infrastructure on failed
>> assertions in the DBCSR sparse matrix (sub)library. To use it you must
>> also add the timings_mp.o module to the LIBS variable (used to get
>> around this and some other circular dependencies). While using it or
>> not should not affect behavior, there are OpenMP statements in stack
>> tracing and timing.
>>
> i have so far tried without.
>
i've tried with -D__USE_CP2K_TRACE and without. no difference.
> > this one is a bit of a surprise, since already gfortran 4.3.x claims
>> > to be OpenMP 3 compliant.
>> > -D__HAS_NO_OMP_3
>>
>> I used gfortran 4.3 until recently (where psmp also worked last time I
>> checked) and didn't update the defines...
>>
> and it appears that -D__HAS_NO_OMP_3 seems to help with cp2k.ssmp
for good measure, i am linking with the serial version
> now, but that didn't change the behavior.
>
yep. serial or threaded MKL doesn't make a difference.
i am now down to two cases:
cp2k.ssmp works fine with the development code for as long
as -D__HAS_NO_OMP_3 is set. the same is true for cp2k.psmp
if as long as i use only 1 MPI task. cp2k.psmp is also working
fine if i use only 1 OpenMP thread and multiple MPI tasks, but
using multiple MPI tasks _and_ OpenMP threads gives me random
numbers. for example the H2O-32.inp benchmark with two MPI and 6 OpenMP
gives:
----------------------------------- OT
---------------------------------------
Step Update method Time Convergence Total energy
Change
------------------------------------------------------------------------------
1 OT DIIS 0.15E+00 1.0 1942.49460508 -33766612.6904597133
-3.38E+07
2 OT DIIS 0.15E+00 1.0 605.50197323 ********************
-2.64E+09
3 OT DIIS 0.15E+00 1.1 160.86073279 4380968.2349330271
2.68E+09
4 OT DIIS 0.15E+00 1.1 164.34481270 865533.1852639467
-3.52E+06
5 OT DIIS 0.15E+00 1.1 428.43636015 -30478863.0777326487
-3.13E+07
6 OT DIIS 0.15E+00 1.1 1094.51147521 3880172.3745214189
3.44E+07
7 OT DIIS 0.15E+00 1.1 33.84945332 ********************
-1.37E+08
8 OT DIIS 0.15E+00 1.1 164.71992666 -2837998.3761878917
1.30E+08
9 OT SD 0.15E+00 1.1 2844.75780601 ********************
-1.30E+08
10 OT SD 0.15E+00 1.1 3605.58561902 2964610.0015976029
1.36E+08
11 OT DIIS 0.15E+00 1.1 209.53192021 796475.0800868375
-2.17E+06
12 OT DIIS 0.15E+00 1.1 380.38070104 -26949187.3137775287
-2.77E+07
with 12 MPI and 1 OpenMP and the same binary i get the proper behavior.
----------------------------------- OT
---------------------------------------
Step Update method Time Convergence Total energy
Change
------------------------------------------------------------------------------
1 OT DIIS 0.15E+00 0.6 0.01909729 -529.3015728815
-5.29E+02
2 OT DIIS 0.15E+00 0.8 0.01238424 -536.3043281413
-7.00E+00
3 OT DIIS 0.15E+00 0.8 0.00874280 -540.9240856869
-4.62E+00
4 OT DIIS 0.15E+00 0.8 0.00605219 -544.3264898596
-3.40E+00
5 OT DIIS 0.15E+00 0.8 0.00458609 -546.1883072280
-1.86E+00
6 OT DIIS 0.15E+00 0.8 0.00351827 -547.5582206572
-1.37E+00
7 OT DIIS 0.15E+00 0.8 0.00273439 -548.5013690323
-9.43E-01
8 OT DIIS 0.15E+00 0.8 0.00208753 -549.1681124465
-6.67E-01
9 OT DIIS 0.15E+00 0.8 0.00176820 -549.4777508042
-3.10E-01
10 OT DIIS 0.15E+00 0.8 0.00140542 -549.7607744914
-2.83E-01
11 OT DIIS 0.15E+00 0.8 0.00119366 -549.9316698866
-1.71E-01
12 OT DIIS 0.15E+00 0.8 0.00102681 -550.0679169205
-1.36E-01
could it be that there is some code path where some initializations that
are required for "clean" OpenMP behavior are not executed, if the MPI
rank /= 0?
this seems to be quite a generic problem and it puzzles me
why it may work on one installation and not another.
the only other significant difference that i see to your setup
is that you are using MPICH2 and i use OpenMPI (1.4.3), but
if that would make a difference, google will probably request
that this group will become adult only after my next post.
thanks for your help,
axel.
just make sure we're on the same page. the input(s) that
> are giving me the problems are the water benchmark inputs.
> in tests/QS/benchmark/
>
> the cp2k.?smp binaries work fine for OMP_NUM_THREADS=1,
> but as soon as i enable more than one thread with cp2k.psmp
> and multiple MPI tasks, i get NaNs or random energies.
> cp2k.ssmp appears to be working for both intel 11.1 and gfortran 4.4.4
>
> thanks,
> axel.
>
> Cheers,
>> Urban.
>>
>> >
>> > in any case, i'll give it a try.
>> >
>> >
>> > thanks,
>> > axel.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "cp2k" group.
>> > To post to this group, send email to cp... at googlegroups.com.
>> > To unsubscribe from this group, send email to cp2k
>> > +uns... at googlegroups.com.
>> > For more options, visit this group at
>> > http://groups.google.com/group/cp2k?hl=en.
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.cp2k.org/archives/cp2k-user/attachments/20110509/54d15c23/attachment.htm>
More information about the CP2K-user
mailing list