[CP2K-user] [CP2K:17435] CP2K freeze
SALVATORE LABONIA
salvatore.labonia at gmail.com
Thu Aug 4 13:05:49 UTC 2022
Hello Krack,
we have tried to run cp2k using TRACE keyword and cp2k compute job died ...
Is it possible to identify the issue using attached cp2k log file ?
Can you help us to identify the error in the regression tests ?
Regards
Salvatore
Il 03/08/2022 16:11, Krack Matthias (PSI) ha scritto:
>
> Hi Salvatore
>
> Your CP2K 8.2 build has obviously issues, because many tests completed
> with the status FAILED or WRONG. Thus that cp2k binary cannot be
> considered ready for production.
>
> From the error messages, it is not clear to me what the problem could
> be. Maybe, someone else has an idea.
>
> BR
>
> Matthias
>
> *From: *SALVATORE LABONIA <salvatore.labonia at gmail.com>
> *Date: *Wednesday, 3 August 2022 at 14:40
> *To: *"cp2k at googlegroups.com" <cp2k at googlegroups.com>, "Krack Matthias
> (PSI)" <matthias.krack at psi.ch>
> *Subject: *Re: [CP2K:17431] CP2K freeze
>
> Hello,
>
> cp2k 8.2 was compiled using EasyBuild
>
> I attach some output from regression test
>
> Regards
>
> Salvatore
>
> Il 03/08/2022 13:34, Krack Matthias (PSI) ha scritto:
>
> Hi Salvatore
>
> You can add the keyword TRACE
> <https://manual.cp2k.org/cp2k-9_1-branch/CP2K_INPUT/GLOBAL.html#TRACE>
> (or TRACE_MASTER
> <https://manual.cp2k.org/cp2k-9_1-branch/CP2K_INPUT/GLOBAL.html#list_TRACE_MASTER>
> to trace only the MPI root process) in the &GLOBAL section of the
> CP2K input to get a more detailed output.
>
> Does the run freeze for any kind of CP2K input? How did you
> compile CP2K? Could you run the regression test successfully? It
> is difficult to make any suggestion without further information.
>
> Best
>
> Matthias
>
> *From: *"cp2k at googlegroups.com" <mailto:cp2k at googlegroups.com>
> <cp2k at googlegroups.com> <mailto:cp2k at googlegroups.com> on behalf
> of Salvatore Labonia <salvatore.labonia at gmail.com>
> <mailto:salvatore.labonia at gmail.com>
> *Reply to: *"cp2k at googlegroups.com" <mailto:cp2k at googlegroups.com>
> <cp2k at googlegroups.com> <mailto:cp2k at googlegroups.com>
> *Date: *Wednesday, 3 August 2022 at 12:35
> *To: *"cp2k at googlegroups.com" <mailto:cp2k at googlegroups.com>
> <cp2k at googlegroups.com> <mailto:cp2k at googlegroups.com>
> *Subject: *[CP2K:17431] CP2K freeze
>
> Hello,
>
> we are facing freeze using CP2K on our HPC cluster.
>
> We have totally 94 Dell server but running cp2k v9.1 compiled with
> intel compiler and linked with intel mpi library, customer is
> experiencing running freeze.
>
> No matter the number or the type of involved nodes.
>
> The freeze happens randomly, not at the same interaction number,
> even using the same running command and the same dataset for input.
>
> Looking at processes status on nodes when freeze occurs, they seem
> to be running, using CPU but, if we try to attach to any process
> (and forked children of course), we can see that they all are
> sitting on a wait system call for data coming (orout going) from
> (to) a pipe.
>
> No other systems call are run by processes…
>
> Slurm thinks that job is still running.
>
> Killing one of the stuck processes causes the death of orher
> processes and finally slurm realizes that job has crashed.
>
> Is this behaviour usual in same circumstances (and therefore
> customer has something to do to avoid it) or could it be caused by
> some other reason (cp2k compilation, mpi version, intel compilers
> version)?
>
> Is there any way to have a debugging execution of cp2k/mpi with a
> more or less verbose output in order to understand at which
> point/call does the freeze happen?
>
> Regards
>
> Salvatore
>
> --
> You received this message because you are subscribed to the Google
> Groups "cp2k" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to cp2k+unsubscribe at googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/cp2k/2bffd2de-1afd-4980-b3aa-6438990d81a9n%40googlegroups.com
> <https://groups.google.com/d/msgid/cp2k/2bffd2de-1afd-4980-b3aa-6438990d81a9n%40googlegroups.com?utm_medium=email&utm_source=footer>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "cp2k" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to cp2k+unsubscribe at googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/cp2k/4B492500-D071-47FB-B7F6-9D95EA33A429%40psi.ch
> <https://groups.google.com/d/msgid/cp2k/4B492500-D071-47FB-B7F6-9D95EA33A429%40psi.ch?utm_medium=email&utm_source=footer>.
>
--
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+unsubscribe at googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/502df271-acd9-34d9-5975-a4617300c3df%40gmail.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.cp2k.org/archives/cp2k-user/attachments/20220804/ac2cd42f/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 34888 bytes
Desc: not available
URL: <https://lists.cp2k.org/archives/cp2k-user/attachments/20220804/ac2cd42f/attachment-0001.png>
-------------- next part --------------
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 13 18382 rs_pw_transfer_RS2PW
_50 0.113 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 12 138779 rs_pw_transfer 0.
113 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 12 138780 rs_pw_transfer st
art Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 13 18382 rs_pw_transfer_RS2PW
_20 start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688192 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688192 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688193 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688193 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 13 18382 rs_pw_transfer_RS2PW
_20 0.049 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 12 138780 rs_pw_transfer 0.
049 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 12 138781 rs_pw_transfer st
art Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 13 18382 rs_pw_transfer_RS2PW
_10 start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688194 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688194 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_isendrecv_dv
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_isendrecv_dv
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 ****** mp_waitall_1
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 14 ****** mp_waitall_1
0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 13 18382 rs_pw_transfer_RS2PW
_10 0.037 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 12 138781 rs_pw_transfer 0.
037 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 12 18382 pw_zero start Hos
tmem: 688195 MB GPUmem: 0 MB
000000:000360<< 12 18382 pw_zero 0.005 Hos
tmem: 688195 MB GPUmem: 0 MB
000000:000360>> 12 272351 pw_transfer start
Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 13 253903 fft_wrap_pw1pw2
start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 14 143155 fft_wrap_pw1pw2_4
00 start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 15 253903 fft3d_ps
start Hostmem: 688204 MB GPUmem: 0 MB
000000:000360>> 16 258723 mp_environ_
l start Hostmem: 688204 MB GPUmem: 0 MB
000000:000360<< 16 258723 mp_environ_
l 0.000 Hostmem: 688204 MB GPUmem: 0 MB
000000:000360>> 16 253907 mp_environ_
c start Hostmem: 688204 MB GPUmem: 0 MB
000000:000360<< 16 253907 mp_environ_
c 0.000 Hostmem: 688204 MB GPUmem: 0 MB
000000:000360>> 16 253907 mp_comm_com
pare start Hostmem: 688204 MB GPUmem: 0 MB
000000:000360<< 16 253907 mp_comm_com
pare 0.000 Hostmem: 688204 MB GPUmem: 0 MB
000000:000360>> 16 253903 mp_rank_com
pare start Hostmem: 688204 MB GPUmem: 0 MB
000000:000360<< 16 253903 mp_rank_com
pare 0.000 Hostmem: 688204 MB GPUmem: 0 MB
000000:000360>> 16 253904 get_fft_scr
atch start Hostmem: 688204 MB GPUmem: 0 MB
000000:000360<< 16 253904 get_fft_scr
atch 0.000 Hostmem: 688204 MB GPUmem: 0 MB
000000:000360>> 16 55675 yz_to_x
start Hostmem: 688204 MB GPUmem: 0 MB
000000:000360>> 17 253903 mp_allto
all_z22v start Hostmem: 688204 MB GPUmem: 0 MB
000000:000360<< 17 253903 mp_allto
all_z22v 0.063 Hostmem: 688204 MB GPUmem: 0 MB
000000:000360<< 16 55675 yz_to_x
0.072 Hostmem: 688204 MB GPUmem: 0 MB
000000:000360<< 15 253903 fft3d_ps
0.118 Hostmem: 688204 MB GPUmem: 0 MB
000000:000360>> 15 129199 pw_gather_p
start Hostmem: 688204 MB GPUmem: 0 MB
000000:000360<< 15 129199 pw_gather_p
0.017 Hostmem: 688204 MB GPUmem: 0 MB
000000:000360<< 14 143155 fft_wrap_pw1pw2_4
00 0.181 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 13 253903 fft_wrap_pw1pw2
0.181 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 272351 pw_transfer 0.181
Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 166326 pw_axpy start Hos
tmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 166326 pw_axpy 0.006 Hos
tmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 272352 pw_transfer start
Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 13 253904 fft_wrap_pw1pw2
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 14 27688 fft_wrap_pw1pw2_1
40 start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 15 253904 fft3d_ps
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 258724 mp_environ_
l start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 258724 mp_environ_
l 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 253908 mp_environ_
c start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 253908 mp_environ_
c 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 253908 mp_comm_com
pare start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 253908 mp_comm_com
pare 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 253904 mp_rank_com
pare start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 253904 mp_rank_com
pare 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 253905 get_fft_scr
atch start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 253905 get_fft_scr
atch 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 73525 cube_transp
ose_2 start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 17 253904 mp_allto
all_z22v start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 17 253904 mp_allto
all_z22v 0.002 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 73525 cube_transp
ose_2 0.003 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 73525 xz_to_yz
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 17 110749 mp_allto
all_z11v start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 17 110749 mp_allto
all_z11v 0.010 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 73525 xz_to_yz
0.015 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 15 253904 fft3d_ps
0.025 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 15 129200 pw_gather_p
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 15 129200 pw_gather_p
0.005 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 14 27688 fft_wrap_pw1pw2_1
40 0.033 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 13 253904 fft_wrap_pw1pw2
0.033 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 272352 pw_transfer 0.033
Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 166327 pw_axpy start Hos
tmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 166327 pw_axpy 0.001 Hos
tmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 272353 pw_transfer start
Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 13 253905 fft_wrap_pw1pw2
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 14 27688 fft_wrap_pw1pw2_5
0 start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 15 253905 fft3d_ps
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 258725 mp_environ_
l start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 258725 mp_environ_
l 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 253909 mp_environ_
c start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 253909 mp_environ_
c 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 253909 mp_comm_com
pare start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 253909 mp_comm_com
pare 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 253905 mp_rank_com
pare start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 253905 mp_rank_com
pare 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 253906 get_fft_scr
atch start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 253906 get_fft_scr
atch 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 73526 cube_transp
ose_2 start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 17 253905 mp_allto
all_z22v start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 17 253905 mp_allto
all_z22v 0.001 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 73526 cube_transp
ose_2 0.001 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 73526 xz_to_yz
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 17 110750 mp_allto
all_z11v start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 17 110750 mp_allto
all_z11v 0.002 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 73526 xz_to_yz
0.003 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 15 253905 fft3d_ps
0.005 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 15 129201 pw_gather_p
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 15 129201 pw_gather_p
0.001 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 14 27688 fft_wrap_pw1pw2_5
0 0.007 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 13 253905 fft_wrap_pw1pw2
0.007 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 272353 pw_transfer 0.007
Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 166328 pw_axpy start Hos
tmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 166328 pw_axpy 0.000 Hos
tmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 272354 pw_transfer start
Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 13 253906 fft_wrap_pw1pw2
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 14 27688 fft_wrap_pw1pw2_2
0 start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 15 253906 fft3d_ps
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 258726 mp_environ_
l start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 258726 mp_environ_
l 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 253910 mp_environ_
c start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 253910 mp_environ_
c 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 253910 mp_comm_com
pare start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 253910 mp_comm_com
pare 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 253906 mp_rank_com
pare start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 253906 mp_rank_com
pare 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 253907 get_fft_scr
atch start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 253907 get_fft_scr
atch 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 73527 cube_transp
ose_2 start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 17 253906 mp_allto
all_z22v start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 17 253906 mp_allto
all_z22v 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 73527 cube_transp
ose_2 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 73527 xz_to_yz
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 17 110751 mp_allto
all_z11v start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 17 110751 mp_allto
all_z11v 0.001 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 73527 xz_to_yz
0.001 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 15 253906 fft3d_ps
0.002 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 15 129202 pw_gather_p
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 15 129202 pw_gather_p
0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 14 27688 fft_wrap_pw1pw2_2
0 0.002 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 13 253906 fft_wrap_pw1pw2
0.002 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 272354 pw_transfer 0.002
Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 166329 pw_axpy start Hos
tmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 166329 pw_axpy 0.000 Hos
tmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 272355 pw_transfer start
Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 13 253907 fft_wrap_pw1pw2
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 14 27688 fft_wrap_pw1pw2_1
0 start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 15 253907 fft3d_ps
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 258727 mp_environ_
l start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 258727 mp_environ_
l 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 253911 mp_environ_
c start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 253911 mp_environ_
c 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 253911 mp_comm_com
pare start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 253911 mp_comm_com
pare 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 253907 mp_rank_com
pare start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 253907 mp_rank_com
pare 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 253908 get_fft_scr
atch start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 253908 get_fft_scr
atch 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 73528 cube_transp
ose_2 start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 17 253907 mp_allto
all_z22v start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 17 253907 mp_allto
all_z22v 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 73528 cube_transp
ose_2 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 73528 xz_to_yz
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 17 110752 mp_allto
all_z11v start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 17 110752 mp_allto
all_z11v 0.001 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 73528 xz_to_yz
0.001 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 15 253907 fft3d_ps
0.001 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 15 129203 pw_gather_p
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 15 129203 pw_gather_p
0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 14 27688 fft_wrap_pw1pw2_1
0 0.002 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 13 253907 fft_wrap_pw1pw2
0.002 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 272355 pw_transfer 0.002
Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 166330 pw_axpy start Hos
tmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 166330 pw_axpy 0.000 Hos
tmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 272356 pw_transfer start
Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 13 253908 fft_wrap_pw1pw2
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 14 143156 fft_wrap_pw1pw2_4
00 start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 15 124705 pw_scatter_p
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 15 124705 pw_scatter_p
0.017 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 15 253908 fft3d_ps
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 258728 mp_environ_
l start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 258728 mp_environ_
l 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 253912 mp_environ_
c start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 253912 mp_environ_
c 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 253912 mp_comm_com
pare start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 253912 mp_comm_com
pare 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 253908 mp_rank_com
pare start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 253908 mp_rank_com
pare 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 253909 get_fft_scr
atch start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 253909 get_fft_scr
atch 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 87481 x_to_yz
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 17 253908 mp_allto
all_z22v start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 17 253908 mp_allto
all_z22v 0.050 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 87481 x_to_yz
0.069 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 15 253908 fft3d_ps
0.132 Hostmem: 688204 MB GPUmem: 0 MB
000000:000360<< 14 143156 fft_wrap_pw1pw2_4
00 0.163 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 13 253908 fft_wrap_pw1pw2
0.163 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 272356 pw_transfer 0.163
Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 494396 pw_pool_give_back_pw
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 494396 pw_pool_give_back_pw
0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 494397 pw_pool_give_back_pw
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 494397 pw_pool_give_back_pw
0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 494398 pw_pool_give_back_pw
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 494398 pw_pool_give_back_pw
0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 494399 pw_pool_give_back_pw
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 494399 pw_pool_give_back_pw
0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 494400 pw_pool_give_back_pw
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 494400 pw_pool_give_back_pw
0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 494401 pw_pool_give_back_pw
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 494401 pw_pool_give_back_pw
0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 494402 pw_pool_give_back_pw
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 494402 pw_pool_give_back_pw
0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 494403 pw_pool_give_back_pw
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 494403 pw_pool_give_back_pw
0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 494404 pw_pool_give_back_pw
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 494404 pw_pool_give_back_pw
0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 494405 pw_pool_give_back_pw
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 494405 pw_pool_give_back_pw
0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 11 18382 density_rs2pw 0.973
Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 11 257892 mp_sum_d start Hostm
em: 688188 MB GPUmem: 0 MB
000000:000360<< 11 257892 mp_sum_d 0.003 Hostm
em: 688188 MB GPUmem: 0 MB
000000:000360<< 10 18382 calculate_rho_elec 1.29
4 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 9 9191 qs_rho_update_rho 2.597 Ho
stmem: 688188 MB GPUmem: 0 MB
000000:000360>> 9 9260 qs_ks_did_change start Hos
tmem: 688188 MB GPUmem: 0 MB
000000:000360<< 9 9260 qs_ks_did_change 0.000 Hos
tmem: 688188 MB GPUmem: 0 MB
000000:000360<< 8 9122 scf_env_do_scf_inner_loop 5.1
68 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 8 9123 scf_env_do_scf_inner_loop sta
rt Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 9 9192 qs_ks_update_qs_env start
Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 10 9191 evaluate_core_matrix_traces
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 11 18381 calculate_ptrace_kp
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 73443 dbcsr_dot_sd star
t Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 13 257893 mp_sum_d start
Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 13 257893 mp_sum_d 0.000
Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 73443 dbcsr_dot_sd 0.00
0 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 73444 dbcsr_dot_sd star
t Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 13 257894 mp_sum_d start
Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 13 257894 mp_sum_d 0.000
Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 73444 dbcsr_dot_sd 0.00
0 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 11 18381 calculate_ptrace_kp
0.001 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 11 18382 calculate_ptrace_kp
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 73445 dbcsr_dot_sd star
t Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 13 257895 mp_sum_d start
Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 13 257895 mp_sum_d 0.000
Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 73445 dbcsr_dot_sd 0.00
0 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 73446 dbcsr_dot_sd star
t Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 13 257896 mp_sum_d start
Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 13 257896 mp_sum_d 0.000
Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 73446 dbcsr_dot_sd 0.00
0 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 11 18382 calculate_ptrace_kp
0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 10 9191 evaluate_core_matrix_traces
0.002 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 10 9191 rebuild_ks_matrix start
Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 11 9191 qs_ks_build_kohn_sham_matr
ix start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 448223 pw_pool_create_pw
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 448223 pw_pool_create_pw
0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 448224 pw_pool_create_pw
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 448224 pw_pool_create_pw
0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 147757 pw_copy start Hos
tmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 147757 pw_copy 0.009 Hos
tmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 166331 pw_axpy start Hos
tmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 166331 pw_axpy 0.008 Hos
tmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 166332 pw_axpy start Hos
tmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 166332 pw_axpy 0.007 Hos
tmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 257897 mp_sum_d start Ho
stmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 257897 mp_sum_d 0.000 Ho
stmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 9191 pw_poisson_solve
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 13 9191 pw_poisson_rebuild
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 13 9191 pw_poisson_rebuild
0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 13 448225 pw_pool_create_pw
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 13 448225 pw_pool_create_pw
0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 13 272357 pw_transfer st
art Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 14 147758 pw_copy sta
rt Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 14 147758 pw_copy 0.0
09 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 13 272357 pw_transfer 0.
009 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 13 272358 pw_transfer st
art Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 14 147759 pw_copy sta
rt Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 14 147759 pw_copy 0.0
08 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 13 272358 pw_transfer 0.
008 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 13 9191 pw_integral_ab
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 14 257898 mp_sum_d st
art Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 14 257898 mp_sum_d 0.
000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 13 9191 pw_integral_ab
0.014 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 13 494406 pw_pool_give_back_pw
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 13 494406 pw_pool_give_back_pw
0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 12 9191 pw_poisson_solve
0.041 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 12 272359 pw_transfer start
Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 13 253909 fft_wrap_pw1pw2
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 14 143157 fft_wrap_pw1pw2_4
00 start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 15 124706 pw_scatter_p
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 15 124706 pw_scatter_p
0.017 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 15 253909 fft3d_ps
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 258729 mp_environ_
l start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 258729 mp_environ_
l 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 253913 mp_environ_
c start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 253913 mp_environ_
c 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 253913 mp_comm_com
pare start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 253913 mp_comm_com
pare 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 253909 mp_rank_com
pare start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 253909 mp_rank_com
pare 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 253910 get_fft_scr
atch start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 253910 get_fft_scr
atch 0.000 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 16 87482 x_to_yz
start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360>> 17 253909 mp_allto
all_z22v start Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 17 253909 mp_allto
all_z22v 0.053 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 16 87482 x_to_yz
0.072 Hostmem: 688188 MB GPUmem: 0 MB
000000:000360<< 15 253909 fft3d_ps
0.148 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 143157 fft_wrap_pw1pw2_4
00 0.181 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 13 253909 fft_wrap_pw1pw2
0.181 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 12 272359 pw_transfer 0.182
Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 12 4654 pw_scale start Ho
stmem: 688205 MB GPUmem: 0 MB
000000:000360<< 12 4654 pw_scale 0.003 Ho
stmem: 688205 MB GPUmem: 0 MB
000000:000360>> 12 494407 pw_pool_give_back_pw
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 12 494407 pw_pool_give_back_pw
0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 12 494408 pw_pool_give_back_pw
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 12 494408 pw_pool_give_back_pw
0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 12 9191 qs_vxc_create sta
rt Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 13 4654 xc_vxc_pw_create
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 9191 xc_rho_set_and_ds
et_create start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 448226 pw_pool_create
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 448226 pw_pool_create
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 147760 pw_copy
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 147760 pw_copy
0.003 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 448227 pw_pool_create
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 448227 pw_pool_create
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 448228 pw_pool_create
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 448228 pw_pool_create
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 448229 pw_pool_create
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 448229 pw_pool_create
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 448230 pw_pool_create
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 448230 pw_pool_create
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 147761 pw_copy
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 147761 pw_copy
0.004 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 83059 pw_derive
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 83059 pw_derive
0.010 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 272360 pw_transfer
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 16 253910 fft_wrap_pw
1pw2 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 17 143158 fft_wrap
_pw1pw2_400 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 124707 pw_sc
atter_p start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 124707 pw_sc
atter_p 0.012 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253910 fft3d
_ps start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 258730 mp
_environ_l start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 258730 mp
_environ_l 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253914 mp
_environ_c start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253914 mp
_environ_c 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253914 mp
_comm_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253914 mp
_comm_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253910 mp
_rank_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253910 mp
_rank_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253911 ge
t_fft_scratch start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253911 ge
t_fft_scratch 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 87483 x_
to_yz start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 20 253910
mp_alltoall_z22v start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 20 253910
mp_alltoall_z22v 0.080 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 87483 x_
to_yz 0.099 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253910 fft3d
_ps 0.153 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 17 143158 fft_wrap
_pw1pw2_400 0.176 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 16 253910 fft_wrap_pw
1pw2 0.176 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 272360 pw_transfer
0.176 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 147762 pw_copy
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 147762 pw_copy
0.004 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 83060 pw_derive
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 83060 pw_derive
0.008 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 272361 pw_transfer
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 16 253911 fft_wrap_pw
1pw2 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 17 143159 fft_wrap
_pw1pw2_400 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 124708 pw_sc
atter_p start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 124708 pw_sc
atter_p 0.010 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253911 fft3d
_ps start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 258731 mp
_environ_l start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 258731 mp
_environ_l 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253915 mp
_environ_c start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253915 mp
_environ_c 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253915 mp
_comm_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253915 mp
_comm_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253911 mp
_rank_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253911 mp
_rank_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253912 ge
t_fft_scratch start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253912 ge
t_fft_scratch 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 87484 x_
to_yz start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 20 253911
mp_alltoall_z22v start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 20 253911
mp_alltoall_z22v 0.074 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 87484 x_
to_yz 0.093 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253911 fft3d
_ps 0.147 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 17 143159 fft_wrap
_pw1pw2_400 0.170 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 16 253911 fft_wrap_pw
1pw2 0.171 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 272361 pw_transfer
0.171 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 147763 pw_copy
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 147763 pw_copy
0.004 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 83061 pw_derive
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 83061 pw_derive
0.008 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 272362 pw_transfer
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 16 253912 fft_wrap_pw
1pw2 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 17 143160 fft_wrap
_pw1pw2_400 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 124709 pw_sc
atter_p start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 124709 pw_sc
atter_p 0.010 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253912 fft3d
_ps start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 258732 mp
_environ_l start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 258732 mp
_environ_l 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253916 mp
_environ_c start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253916 mp
_environ_c 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253916 mp
_comm_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253916 mp
_comm_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253912 mp
_rank_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253912 mp
_rank_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253913 ge
t_fft_scratch start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253913 ge
t_fft_scratch 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 87485 x_
to_yz start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 20 253912
mp_alltoall_z22v start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 20 253912
mp_alltoall_z22v 0.073 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 87485 x_
to_yz 0.092 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253912 fft3d
_ps 0.147 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 17 143160 fft_wrap
_pw1pw2_400 0.169 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 16 253912 fft_wrap_pw
1pw2 0.169 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 272362 pw_transfer
0.169 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 494409 pw_pool_give_b
ack_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 494409 pw_pool_give_b
ack_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 448231 pw_pool_create
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 448231 pw_pool_create
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 147764 pw_copy
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 147764 pw_copy
0.004 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 448232 pw_pool_create
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 448232 pw_pool_create
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 448233 pw_pool_create
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 448233 pw_pool_create
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 448234 pw_pool_create
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 448234 pw_pool_create
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 448235 pw_pool_create
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 448235 pw_pool_create
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 147765 pw_copy
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 147765 pw_copy
0.004 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 83062 pw_derive
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 83062 pw_derive
0.008 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 272363 pw_transfer
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 16 253913 fft_wrap_pw
1pw2 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 17 143161 fft_wrap
_pw1pw2_400 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 124710 pw_sc
atter_p start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 124710 pw_sc
atter_p 0.010 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253913 fft3d
_ps start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 258733 mp
_environ_l start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 258733 mp
_environ_l 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253917 mp
_environ_c start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253917 mp
_environ_c 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253917 mp
_comm_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253917 mp
_comm_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253913 mp
_rank_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253913 mp
_rank_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253914 ge
t_fft_scratch start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253914 ge
t_fft_scratch 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 87486 x_
to_yz start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 20 253913
mp_alltoall_z22v start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 20 253913
mp_alltoall_z22v 0.078 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 87486 x_
to_yz 0.097 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253913 fft3d
_ps 0.152 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 17 143161 fft_wrap
_pw1pw2_400 0.175 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 16 253913 fft_wrap_pw
1pw2 0.175 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 272363 pw_transfer
0.175 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 147766 pw_copy
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 147766 pw_copy
0.004 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 83063 pw_derive
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 83063 pw_derive
0.008 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 272364 pw_transfer
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 16 253914 fft_wrap_pw
1pw2 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 17 143162 fft_wrap
_pw1pw2_400 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 124711 pw_sc
atter_p start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 124711 pw_sc
atter_p 0.010 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253914 fft3d
_ps start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 258734 mp
_environ_l start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 258734 mp
_environ_l 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253918 mp
_environ_c start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253918 mp
_environ_c 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253918 mp
_comm_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253918 mp
_comm_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253914 mp
_rank_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253914 mp
_rank_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253915 ge
t_fft_scratch start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253915 ge
t_fft_scratch 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 87487 x_
to_yz start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 20 253914
mp_alltoall_z22v start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 20 253914
mp_alltoall_z22v 0.073 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 87487 x_
to_yz 0.091 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253914 fft3d
_ps 0.146 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 17 143162 fft_wrap
_pw1pw2_400 0.169 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 16 253914 fft_wrap_pw
1pw2 0.169 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 272364 pw_transfer
0.169 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 147767 pw_copy
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 147767 pw_copy
0.004 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 83064 pw_derive
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 83064 pw_derive
0.008 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 272365 pw_transfer
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 16 253915 fft_wrap_pw
1pw2 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 17 143163 fft_wrap
_pw1pw2_400 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 124712 pw_sc
atter_p start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 124712 pw_sc
atter_p 0.010 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253915 fft3d
_ps start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 258735 mp
_environ_l start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 258735 mp
_environ_l 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253919 mp
_environ_c start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253919 mp
_environ_c 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253919 mp
_comm_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253919 mp
_comm_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253915 mp
_rank_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253915 mp
_rank_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253916 ge
t_fft_scratch start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253916 ge
t_fft_scratch 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 87488 x_
to_yz start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 20 253915
mp_alltoall_z22v start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 20 253915
mp_alltoall_z22v 0.071 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 87488 x_
to_yz 0.090 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253915 fft3d
_ps 0.137 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 17 143163 fft_wrap
_pw1pw2_400 0.158 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 16 253915 fft_wrap_pw
1pw2 0.159 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 272365 pw_transfer
0.159 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 494410 pw_pool_give_b
ack_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 494410 pw_pool_give_b
ack_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 494411 pw_pool_give_b
ack_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 494411 pw_pool_give_b
ack_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 494412 pw_pool_give_b
ack_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 494412 pw_pool_give_b
ack_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 494413 pw_pool_give_b
ack_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 494413 pw_pool_give_b
ack_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 494414 pw_pool_give_b
ack_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 494414 pw_pool_give_b
ack_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 494415 pw_pool_give_b
ack_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 494415 pw_pool_give_b
ack_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 494416 pw_pool_give_b
ack_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 494416 pw_pool_give_b
ack_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 494417 pw_pool_give_b
ack_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 494417 pw_pool_give_b
ack_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 494418 pw_pool_give_b
ack_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 494418 pw_pool_give_b
ack_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 9191 xc_functional_
eval start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 16 9191 pbe_lsd_eva
l start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 16 9191 pbe_lsd_eva
l 0.191 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 9191 xc_functional_
eval 0.192 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 9191 xc_rho_set_and_ds
et_create 1.326 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 51553 pw_create s
tart Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 51553 pw_create 0
.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 51554 pw_create s
tart Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 51554 pw_create 0
.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 448236 pw_pool_create_pw
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 448236 pw_pool_create_pw
0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 448237 pw_pool_create_pw
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 448237 pw_pool_create_pw
0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 448238 pw_pool_create_pw
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 448238 pw_pool_create_pw
0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 51555 pw_create s
tart Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 51555 pw_create 0
.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 51556 pw_create s
tart Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 51556 pw_create 0
.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 51557 pw_create s
tart Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 51557 pw_create 0
.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 51558 pw_create s
tart Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 51558 pw_create 0
.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 494419 pw_pool_give_back
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 494419 pw_pool_give_back
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 166333 pw_axpy sta
rt Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 166333 pw_axpy 0.0
02 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 166334 pw_axpy sta
rt Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 166334 pw_axpy 0.0
02 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 166335 pw_axpy sta
rt Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 166335 pw_axpy 0.0
02 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 448239 pw_pool_create_pw
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 448239 pw_pool_create_pw
0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 272366 pw_transfer
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 253916 fft_wrap_pw1pw
2 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 16 143164 fft_wrap_pw
1pw2_400 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 17 253916 fft3d_ps
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 258736 mp_en
viron_l start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 258736 mp_en
viron_l 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253920 mp_en
viron_c start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253920 mp_en
viron_c 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253920 mp_co
mm_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253920 mp_co
mm_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253916 mp_ra
nk_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253916 mp_ra
nk_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253917 get_f
ft_scratch start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253917 get_f
ft_scratch 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 55676 yz_to
_x start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253916 mp
_alltoall_z22v start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253916 mp
_alltoall_z22v 0.105 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 55676 yz_to
_x 0.112 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 17 253916 fft3d_ps
0.169 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 17 129204 pw_gathe
r_p start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 17 129204 pw_gathe
r_p 0.035 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 16 143164 fft_wrap_pw
1pw2_400 0.210 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 253916 fft_wrap_pw1pw
2 0.210 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 272366 pw_transfer
0.210 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 83065 pw_derive s
tart Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 83065 pw_derive 0
.006 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 494420 pw_pool_give_back
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 494420 pw_pool_give_back
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 448240 pw_pool_create_pw
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 448240 pw_pool_create_pw
0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 272367 pw_transfer
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 253917 fft_wrap_pw1pw
2 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 16 143165 fft_wrap_pw
1pw2_400 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 17 253917 fft3d_ps
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 258737 mp_en
viron_l start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 258737 mp_en
viron_l 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253921 mp_en
viron_c start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253921 mp_en
viron_c 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253921 mp_co
mm_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253921 mp_co
mm_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253917 mp_ra
nk_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253917 mp_ra
nk_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253918 get_f
ft_scratch start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253918 get_f
ft_scratch 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 55677 yz_to
_x start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253917 mp
_alltoall_z22v start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253917 mp
_alltoall_z22v 0.063 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 55677 yz_to
_x 0.071 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 17 253917 fft3d_ps
0.113 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 17 129205 pw_gathe
r_p start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 17 129205 pw_gathe
r_p 0.016 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 16 143165 fft_wrap_pw
1pw2_400 0.137 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 253917 fft_wrap_pw1pw
2 0.137 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 272367 pw_transfer
0.138 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 83066 pw_derive s
tart Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 83066 pw_derive 0
.015 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 166336 pw_axpy sta
rt Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 166336 pw_axpy 0.0
08 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 494421 pw_pool_give_back
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 494421 pw_pool_give_back
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 494422 pw_pool_give_back
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 494422 pw_pool_give_back
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 448241 pw_pool_create_pw
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 448241 pw_pool_create_pw
0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 272368 pw_transfer
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 253918 fft_wrap_pw1pw
2 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 16 143166 fft_wrap_pw
1pw2_400 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 17 253918 fft3d_ps
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 258738 mp_en
viron_l start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 258738 mp_en
viron_l 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253922 mp_en
viron_c start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253922 mp_en
viron_c 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253922 mp_co
mm_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253922 mp_co
mm_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253918 mp_ra
nk_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253918 mp_ra
nk_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253919 get_f
ft_scratch start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253919 get_f
ft_scratch 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 55678 yz_to
_x start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253918 mp
_alltoall_z22v start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253918 mp
_alltoall_z22v 0.068 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 55678 yz_to
_x 0.078 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 17 253918 fft3d_ps
0.131 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 17 129206 pw_gathe
r_p start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 17 129206 pw_gathe
r_p 0.016 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 16 143166 fft_wrap_pw
1pw2_400 0.162 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 253918 fft_wrap_pw1pw
2 0.162 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 272368 pw_transfer
0.162 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 83067 pw_derive s
tart Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 83067 pw_derive 0
.015 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 166337 pw_axpy sta
rt Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 166337 pw_axpy 0.0
08 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 494423 pw_pool_give_back
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 494423 pw_pool_give_back
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 494424 pw_pool_give_back
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 494424 pw_pool_give_back
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 448242 pw_pool_create_pw
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 448242 pw_pool_create_pw
0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 272369 pw_transfer
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 253919 fft_wrap_pw1pw
2 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 16 143167 fft_wrap_pw
1pw2_400 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 17 124713 pw_scatt
er_p start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 17 124713 pw_scatt
er_p 0.017 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 17 253919 fft3d_ps
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 258739 mp_en
viron_l start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 258739 mp_en
viron_l 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253923 mp_en
viron_c start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253923 mp_en
viron_c 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253923 mp_co
mm_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253923 mp_co
mm_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253919 mp_ra
nk_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253919 mp_ra
nk_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253920 get_f
ft_scratch start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253920 get_f
ft_scratch 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 87489 x_to_
yz start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253919 mp
_alltoall_z22v start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253919 mp
_alltoall_z22v 0.060 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 87489 x_to_
yz 0.079 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 17 253919 fft3d_ps
0.133 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 16 143167 fft_wrap_pw
1pw2_400 0.163 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 253919 fft_wrap_pw1pw
2 0.163 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 272369 pw_transfer
0.163 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 166338 pw_axpy sta
rt Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 166338 pw_axpy 0.0
03 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 494425 pw_pool_give_back
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 494425 pw_pool_give_back
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 494426 pw_pool_give_back
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 494426 pw_pool_give_back
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 51559 pw_create s
tart Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 51559 pw_create 0
.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 51560 pw_create s
tart Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 51560 pw_create 0
.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 51561 pw_create s
tart Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 51561 pw_create 0
.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 51562 pw_create s
tart Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 51562 pw_create 0
.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 494427 pw_pool_give_back
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 494427 pw_pool_give_back
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 166339 pw_axpy sta
rt Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 166339 pw_axpy 0.0
03 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 494428 pw_pool_give_back
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 494428 pw_pool_give_back
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 166340 pw_axpy sta
rt Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 166340 pw_axpy 0.0
03 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 494429 pw_pool_give_back
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 494429 pw_pool_give_back
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 166341 pw_axpy sta
rt Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 166341 pw_axpy 0.0
03 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 494430 pw_pool_give_back
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 494430 pw_pool_give_back
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 448243 pw_pool_create_pw
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 448243 pw_pool_create_pw
0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 272370 pw_transfer
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 253920 fft_wrap_pw1pw
2 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 16 143168 fft_wrap_pw
1pw2_400 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 17 253920 fft3d_ps
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 258740 mp_en
viron_l start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 258740 mp_en
viron_l 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253924 mp_en
viron_c start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253924 mp_en
viron_c 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253924 mp_co
mm_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253924 mp_co
mm_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253920 mp_ra
nk_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253920 mp_ra
nk_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253921 get_f
ft_scratch start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253921 get_f
ft_scratch 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 55679 yz_to
_x start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253920 mp
_alltoall_z22v start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253920 mp
_alltoall_z22v 0.078 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 55679 yz_to
_x 0.087 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 17 253920 fft3d_ps
0.132 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 17 129207 pw_gathe
r_p start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 17 129207 pw_gathe
r_p 0.016 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 16 143168 fft_wrap_pw
1pw2_400 0.157 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 253920 fft_wrap_pw1pw
2 0.157 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 272370 pw_transfer
0.157 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 83068 pw_derive s
tart Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 83068 pw_derive 0
.012 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 494431 pw_pool_give_back
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 494431 pw_pool_give_back
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 448244 pw_pool_create_pw
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 448244 pw_pool_create_pw
0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 272371 pw_transfer
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 253921 fft_wrap_pw1pw
2 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 16 143169 fft_wrap_pw
1pw2_400 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 17 253921 fft3d_ps
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 258741 mp_en
viron_l start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 258741 mp_en
viron_l 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253925 mp_en
viron_c start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253925 mp_en
viron_c 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253925 mp_co
mm_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253925 mp_co
mm_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253921 mp_ra
nk_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253921 mp_ra
nk_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253922 get_f
ft_scratch start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253922 get_f
ft_scratch 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 55680 yz_to
_x start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253921 mp
_alltoall_z22v start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253921 mp
_alltoall_z22v 0.074 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 55680 yz_to
_x 0.084 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 17 253921 fft3d_ps
0.139 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 17 129208 pw_gathe
r_p start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 17 129208 pw_gathe
r_p 0.015 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 16 143169 fft_wrap_pw
1pw2_400 0.169 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 253921 fft_wrap_pw1pw
2 0.169 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 272371 pw_transfer
0.169 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 83069 pw_derive s
tart Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 83069 pw_derive 0
.015 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 166342 pw_axpy sta
rt Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 166342 pw_axpy 0.0
08 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 494432 pw_pool_give_back
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 494432 pw_pool_give_back
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 494433 pw_pool_give_back
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 494433 pw_pool_give_back
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 448245 pw_pool_create_pw
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 448245 pw_pool_create_pw
0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 272372 pw_transfer
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 253922 fft_wrap_pw1pw
2 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 16 143170 fft_wrap_pw
1pw2_400 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 17 253922 fft3d_ps
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 258742 mp_en
viron_l start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 258742 mp_en
viron_l 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253926 mp_en
viron_c start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253926 mp_en
viron_c 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253926 mp_co
mm_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253926 mp_co
mm_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253922 mp_ra
nk_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253922 mp_ra
nk_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253923 get_f
ft_scratch start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253923 get_f
ft_scratch 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 55681 yz_to
_x start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253922 mp
_alltoall_z22v start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253922 mp
_alltoall_z22v 0.067 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 55681 yz_to
_x 0.077 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 17 253922 fft3d_ps
0.130 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 17 129209 pw_gathe
r_p start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 17 129209 pw_gathe
r_p 0.016 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 16 143170 fft_wrap_pw
1pw2_400 0.162 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 253922 fft_wrap_pw1pw
2 0.162 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 272372 pw_transfer
0.162 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 83070 pw_derive s
tart Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 83070 pw_derive 0
.015 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 166343 pw_axpy sta
rt Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 166343 pw_axpy 0.0
08 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 494434 pw_pool_give_back
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 494434 pw_pool_give_back
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 494435 pw_pool_give_back
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 494435 pw_pool_give_back
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 448246 pw_pool_create_pw
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 448246 pw_pool_create_pw
0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 272373 pw_transfer
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 15 253923 fft_wrap_pw1pw
2 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 16 143171 fft_wrap_pw
1pw2_400 start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 17 124714 pw_scatt
er_p start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 17 124714 pw_scatt
er_p 0.017 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 17 253923 fft3d_ps
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 258743 mp_en
viron_l start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 258743 mp_en
viron_l 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253927 mp_en
viron_c start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253927 mp_en
viron_c 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253927 mp_co
mm_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253927 mp_co
mm_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253923 mp_ra
nk_compare start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253923 mp_ra
nk_compare 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 253924 get_f
ft_scratch start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 253924 get_f
ft_scratch 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 18 87490 x_to_
yz start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 19 253923 mp
_alltoall_z22v start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 19 253923 mp
_alltoall_z22v 0.056 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 18 87490 x_to_
yz 0.075 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 17 253923 fft3d_ps
0.128 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 16 143171 fft_wrap_pw
1pw2_400 0.157 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 15 253923 fft_wrap_pw1pw
2 0.157 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 272373 pw_transfer
0.157 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 166344 pw_axpy sta
rt Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 166344 pw_axpy 0.0
03 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 494436 pw_pool_give_back
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 494436 pw_pool_give_back
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 494437 pw_pool_give_back
_pw start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 494437 pw_pool_give_back
_pw 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 51563 pw_create s
tart Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 51563 pw_create 0
.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 257899 mp_sum_d st
art Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 257899 mp_sum_d 0.
007 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 13 4654 xc_vxc_pw_create
3.177 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 12 9191 qs_vxc_create 3.1
78 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 12 292885 dbcsr_copy start
Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 13 ****** dbcsr_create_new
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 ****** dbcsr_make_index_
exist start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 ****** dbcsr_make_index_
exist 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 13 ****** dbcsr_create_new
0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 12 292885 dbcsr_copy 0.000
Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 12 292886 dbcsr_copy start
Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 13 ****** dbcsr_create_new
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 14 ****** dbcsr_make_index_
exist start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 14 ****** dbcsr_make_index_
exist 0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 13 ****** dbcsr_create_new
0.000 Hostmem: 688205 MB GPUmem: 0 MB
000000:000360<< 12 292886 dbcsr_copy 0.000
Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 12 4654 sum_up_and_integrate
start Hostmem: 688205 MB GPUmem: 0 MB
000000:000360>> 13 9307 integrate_v_rspace
start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 14 9307 potential_pw2rs
start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 448247 pw_pool_create
_pw start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 448247 pw_pool_create
_pw 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 448248 pw_pool_create
_pw start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 448248 pw_pool_create
_pw 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 448249 pw_pool_create
_pw start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 448249 pw_pool_create
_pw 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 448250 pw_pool_create
_pw start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 448250 pw_pool_create
_pw 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 448251 pw_pool_create
_pw start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 448251 pw_pool_create
_pw 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 448252 pw_pool_create
_pw start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 448252 pw_pool_create
_pw 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 448253 pw_pool_create
_pw start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 448253 pw_pool_create
_pw 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 448254 pw_pool_create
_pw start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 448254 pw_pool_create
_pw 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 448255 pw_pool_create
_pw start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 448255 pw_pool_create
_pw 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 448256 pw_pool_create
_pw start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 448256 pw_pool_create
_pw 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 272374 pw_transfer
start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 16 253924 fft_wrap_pw
1pw2 start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 143172 fft_wrap
_pw1pw2_400 start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 18 253924 fft3d
_ps start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 258744 mp
_environ_l start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 258744 mp
_environ_l 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 253928 mp
_environ_c start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 253928 mp
_environ_c 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 253928 mp
_comm_compare start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 253928 mp
_comm_compare 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 253924 mp
_rank_compare start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 253924 mp
_rank_compare 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 253925 ge
t_fft_scratch start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 253925 ge
t_fft_scratch 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 55682 yz
_to_x start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 20 253924
mp_alltoall_z22v start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 20 253924
mp_alltoall_z22v 0.076 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 55682 yz
_to_x 0.086 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 18 253924 fft3d
_ps 0.137 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 18 129210 pw_ga
ther_p start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 18 129210 pw_ga
ther_p 0.016 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 143172 fft_wrap
_pw1pw2_400 0.167 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 16 253924 fft_wrap_pw
1pw2 0.167 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 272374 pw_transfer
0.167 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 147768 pw_copy
start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 147768 pw_copy
0.004 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 147769 pw_copy
start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 147769 pw_copy
0.002 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 272375 pw_transfer
start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 16 253925 fft_wrap_pw
1pw2 start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 27689 fft_wrap
_pw1pw2_140 start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 18 124715 pw_sc
atter_p start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 18 124715 pw_sc
atter_p 0.003 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 18 253925 fft3d
_ps start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 258745 mp
_environ_l start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 258745 mp
_environ_l 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 253929 mp
_environ_c start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 253929 mp
_environ_c 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 253929 mp
_comm_compare start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 253929 mp
_comm_compare 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 253925 mp
_rank_compare start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 253925 mp
_rank_compare 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 253926 ge
t_fft_scratch start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 253926 ge
t_fft_scratch 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 37225 yz
_to_xz start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 20 110753
mp_alltoall_z11v start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 20 110753
mp_alltoall_z11v 0.011 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 37225 yz
_to_xz 0.019 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 37225 cu
be_transpose_1 start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 20 253925
mp_alltoall_z22v start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 20 253925
mp_alltoall_z22v 0.003 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 37225 cu
be_transpose_1 0.005 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 18 253925 fft3d
_ps 0.031 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 27689 fft_wrap
_pw1pw2_140 0.036 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 16 253925 fft_wrap_pw
1pw2 0.036 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 272375 pw_transfer
0.036 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 147770 pw_copy
start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 147770 pw_copy
0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 272376 pw_transfer
start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 16 253926 fft_wrap_pw
1pw2 start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 27689 fft_wrap
_pw1pw2_50 start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 18 124716 pw_sc
atter_p start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 18 124716 pw_sc
atter_p 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 18 253926 fft3d
_ps start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 258746 mp
_environ_l start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 258746 mp
_environ_l 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 253930 mp
_environ_c start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 253930 mp
_environ_c 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 253930 mp
_comm_compare start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 253930 mp
_comm_compare 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 253926 mp
_rank_compare start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 253926 mp
_rank_compare 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 253927 ge
t_fft_scratch start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 253927 ge
t_fft_scratch 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 37226 yz
_to_xz start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 20 110754
mp_alltoall_z11v start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 20 110754
mp_alltoall_z11v 0.003 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 37226 yz
_to_xz 0.004 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 37226 cu
be_transpose_1 start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 20 253926
mp_alltoall_z22v start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 20 253926
mp_alltoall_z22v 0.001 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 37226 cu
be_transpose_1 0.001 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 18 253926 fft3d
_ps 0.006 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 27689 fft_wrap
_pw1pw2_50 0.007 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 16 253926 fft_wrap_pw
1pw2 0.007 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 272376 pw_transfer
0.007 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 147771 pw_copy
start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 147771 pw_copy
0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 272377 pw_transfer
start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 16 253927 fft_wrap_pw
1pw2 start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 27689 fft_wrap
_pw1pw2_20 start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 18 124717 pw_sc
atter_p start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 18 124717 pw_sc
atter_p 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 18 253927 fft3d
_ps start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 258747 mp
_environ_l start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 258747 mp
_environ_l 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 253931 mp
_environ_c start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 253931 mp
_environ_c 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 253931 mp
_comm_compare start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 253931 mp
_comm_compare 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 253927 mp
_rank_compare start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 253927 mp
_rank_compare 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 253928 ge
t_fft_scratch start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 253928 ge
t_fft_scratch 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 37227 yz
_to_xz start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 20 110755
mp_alltoall_z11v start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 20 110755
mp_alltoall_z11v 0.001 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 37227 yz
_to_xz 0.001 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 37227 cu
be_transpose_1 start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 20 253927
mp_alltoall_z22v start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 20 253927
mp_alltoall_z22v 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 37227 cu
be_transpose_1 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 18 253927 fft3d
_ps 0.002 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 27689 fft_wrap
_pw1pw2_20 0.002 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 16 253927 fft_wrap_pw
1pw2 0.002 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 272377 pw_transfer
0.003 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 147772 pw_copy
start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 147772 pw_copy
0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 272378 pw_transfer
start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 16 253928 fft_wrap_pw
1pw2 start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 27689 fft_wrap
_pw1pw2_10 start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 18 124718 pw_sc
atter_p start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 18 124718 pw_sc
atter_p 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 18 253928 fft3d
_ps start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 258748 mp
_environ_l start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 258748 mp
_environ_l 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 253932 mp
_environ_c start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 253932 mp
_environ_c 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 253932 mp
_comm_compare start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 253932 mp
_comm_compare 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 253928 mp
_rank_compare start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 253928 mp
_rank_compare 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 253929 ge
t_fft_scratch start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 253929 ge
t_fft_scratch 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 37228 yz
_to_xz start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 20 110756
mp_alltoall_z11v start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 20 110756
mp_alltoall_z11v 0.001 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 37228 yz
_to_xz 0.001 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 19 37228 cu
be_transpose_1 start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 20 253928
mp_alltoall_z22v start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 20 253928
mp_alltoall_z22v 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 19 37228 cu
be_transpose_1 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 18 253928 fft3d
_ps 0.001 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 27689 fft_wrap
_pw1pw2_10 0.002 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 16 253928 fft_wrap_pw
1pw2 0.002 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 272378 pw_transfer
0.002 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 494438 pw_pool_give_b
ack_pw start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 494438 pw_pool_give_b
ack_pw 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 494439 pw_pool_give_b
ack_pw start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 494439 pw_pool_give_b
ack_pw 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 494440 pw_pool_give_b
ack_pw start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 494440 pw_pool_give_b
ack_pw 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 494441 pw_pool_give_b
ack_pw start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 494441 pw_pool_give_b
ack_pw 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 494442 pw_pool_give_b
ack_pw start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 15 494442 pw_pool_give_b
ack_pw 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 15 138782 rs_pw_transfer
start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 16 9511 rs_pw_trans
fer_PW2RS_400 start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 18954 rs_grid_
zero start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 18954 rs_grid_
zero 0.122 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_irecv
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_irecv
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_isend
_dv start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_isend
_dv 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.002 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ll_1 start Hostmem: 688206 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ll_1 0.000 Hostmem: 688206 MB GPUmem: 0 MB
000000:000360>> 17 334303 mp_irecv
_dm3 start Hostmem: 688189 MB GPUmem: 0 MB
000000:000360<< 17 334303 mp_irecv
_dm3 0.000 Hostmem: 688189 MB GPUmem: 0 MB
000000:000360>> 17 334303 mp_isend
_dm3 start Hostmem: 688189 MB GPUmem: 0 MB
000000:000360<< 17 334303 mp_isend
_dm3 0.000 Hostmem: 688189 MB GPUmem: 0 MB
000000:000360>> 17 334304 mp_irecv
_dm3 start Hostmem: 688189 MB GPUmem: 0 MB
000000:000360<< 17 334304 mp_irecv
_dm3 0.000 Hostmem: 688189 MB GPUmem: 0 MB
000000:000360>> 17 334304 mp_isend
_dm3 start Hostmem: 688189 MB GPUmem: 0 MB
000000:000360<< 17 334304 mp_isend
_dm3 0.000 Hostmem: 688189 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688189 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.008 Hostmem: 688189 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688189 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.005 Hostmem: 688189 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ll_1 start Hostmem: 688189 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ll_1 0.000 Hostmem: 688189 MB GPUmem: 0 MB
000000:000360>> 17 334305 mp_irecv
_dm3 start Hostmem: 688189 MB GPUmem: 0 MB
000000:000360<< 17 334305 mp_irecv
_dm3 0.000 Hostmem: 688189 MB GPUmem: 0 MB
000000:000360>> 17 334305 mp_isend
_dm3 start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 17 334305 mp_isend
_dm3 0.011 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 17 334306 mp_irecv
_dm3 start Hostmem: 688195 MB GPUmem: 0 MB
000000:000360<< 17 334306 mp_irecv
_dm3 0.000 Hostmem: 688195 MB GPUmem: 0 MB
000000:000360>> 17 334306 mp_isend
_dm3 start Hostmem: 688207 MB GPUmem: 0 MB
000000:000360<< 17 334306 mp_isend
_dm3 0.001 Hostmem: 688207 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688207 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.033 Hostmem: 688219 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688219 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.000 Hostmem: 688219 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ll_1 start Hostmem: 688219 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ll_1 0.000 Hostmem: 688219 MB GPUmem: 0 MB
000000:000360>> 17 334307 mp_irecv
_dm3 start Hostmem: 688213 MB GPUmem: 0 MB
000000:000360<< 17 334307 mp_irecv
_dm3 0.000 Hostmem: 688213 MB GPUmem: 0 MB
000000:000360>> 17 334307 mp_isend
_dm3 start Hostmem: 688221 MB GPUmem: 0 MB
000000:000360<< 17 334307 mp_isend
_dm3 0.000 Hostmem: 688221 MB GPUmem: 0 MB
000000:000360>> 17 334308 mp_irecv
_dm3 start Hostmem: 688209 MB GPUmem: 0 MB
000000:000360<< 17 334308 mp_irecv
_dm3 0.000 Hostmem: 688209 MB GPUmem: 0 MB
000000:000360>> 17 334308 mp_isend
_dm3 start Hostmem: 688216 MB GPUmem: 0 MB
000000:000360<< 17 334308 mp_isend
_dm3 0.000 Hostmem: 688216 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688216 MB GPUmem: 0 MB
000000:000360<< 17 ****** mp_waita
ny 0.018 Hostmem: 688225 MB GPUmem: 0 MB
000000:000360>> 17 ****** mp_waita
ny start Hostmem: 688225 MB GPUmem: 0 MB
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 0 PID 361290 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 1 PID 361291 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 2 PID 361292 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 3 PID 361293 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 4 PID 361294 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 5 PID 361295 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 6 PID 361296 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 7 PID 361297 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 8 PID 361298 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 9 PID 361299 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 10 PID 361300 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 11 PID 361301 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 12 PID 361303 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 13 PID 361304 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 14 PID 361305 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 15 PID 361306 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 16 PID 361307 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 17 PID 361308 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 18 PID 361309 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 19 PID 361310 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 20 PID 361311 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 21 PID 361312 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 22 PID 361313 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 23 PID 361314 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 24 PID 361315 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 25 PID 361316 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 26 PID 361317 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 27 PID 361318 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 28 PID 361319 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 29 PID 361320 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 30 PID 361321 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 31 PID 361322 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 32 PID 361323 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 33 PID 361324 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 34 PID 361325 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 35 PID 361326 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 36 PID 361327 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 37 PID 361328 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 38 PID 361329 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 39 PID 361331 RUNNING AT fn03
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
More information about the CP2K-user
mailing list