<div dir="ltr">Hi,<div><br></div><div>This run is with our internal compiler which is flang based. GNU is working fine. </div><div> </div><div>It is observed that the array fm_matrix%local_data (line number: 96 of cp_dbcsr_cholesky.F) is having different value when number of threads are changed. in GNU the value remains same. Any information about how to root cause the issue through tracing or debugging any point of the code?<br><br>Thanks,</div><div>Shivaram</div><div><br>On Wednesday, April 15, 2020 at 1:37:25 PM UTC+5:30, Alfio Lazzaro wrote:<blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;"><div dir="ltr">Uhm, it seems nothing wrong in the arch file...<div><br></div><div>Some suggestions:</div><div>1. Which version of the GCC compiler are you using (`gfotran --version`)?</div><div>2. Try to reduce the optimization from O3 to O2.</div><div>3. Which version of SCALAPACK/LAPACK/BLAS are you using? Could you use the CP2K toolchain to install these libraries?<br><div>4. CP2K 6.1 is now 2 years old, is it possible for you to upgrade to the 7.1 (at least)?</div><div><br></div><div><br></div><div><br>Il giorno mercoledì 15 aprile 2020 07:03:16 UTC+2, Shivarama Rao ha scritto:<blockquote class="gmail_quote" style="margin:0;margin-left:0.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Alfio,<div><br></div><div>Thanks for looking into this. following is the arch file I am using.</div><div><div><br></div><div>DFLAGS = -D__F2008 -D__FFTW3 -D__LIBINT -D__LIBXC -D__MPI_VERSION=3\</div><div> -D__LIBINT_MAX_AM=5 -D__LIBDERIV_MAX_AM1=4 -D__MAX_CONTR=4\</div><div> -D__parallel -D__SCALAPACK</div><div><br></div><div>CPPFLAGS = -fPIC</div><div>FCFLAGS = $(DFLAGS) -O3 -ffree-form -fPIC\</div><div> -fopenmp -mtune=native \</div><div> -I$(FFTW_INC) -I$(LIBINT_INC) -I$(LIBXC_INC)</div><div>LDFLAGS = $(FCFLAGS) -libverbs -fPIC</div><div><br></div><div>LIBS = $(LIBSCALAPACK_LIB)/<wbr>libscalapack.a\</div><div> $(LIBLAPACK_LIB)/liblapack.a\</div><div> $(LIBBLAS_LIB)/libblas.a\</div><div> $(FFTW_LIB)/libfftw3.a\</div><div> $(FFTW_LIB)/libfftw3_threads.a \</div><div> $(LIBXC_LIB)/libxcf03.a\</div><div> $(LIBXC_LIB)/libxc.a\</div><div> $(LIBINT_LIB)/libderiv.a\</div><div> $(LIBINT_LIB)/libint.a</div><div><br></div><div>let me know if you find any issues with above.</div><div><br></div><div>Thanks,</div><div>Shivaram</div><div><br></div><br>On Tuesday, April 14, 2020 at 12:25:54 PM UTC+5:30, Alfio Lazzaro wrote:<blockquote class="gmail_quote" style="margin:0;margin-left:0.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div>So, from what you are saying the problem is the OpenMP parallelization (it works without).</div><div>However, Cholesky is SCALAPACK, so you should check how you are linking to it. </div><div>In particular, CP2K suggested using Sequential BLAS.</div><div><br></div><div>Could you share you arch file and how you are compiling CP2K? (which libraries)</div><div><br></div><div>Alfio</div><div><br><br>Il giorno lunedì 13 aprile 2020 07:03:42 UTC+2, Shivarama Rao ha scritto:<blockquote class="gmail_quote" style="margin:0;margin-left:0.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>I am trying to debug an issue with cp2k 6.1. The popt run is fine. but there are errors with psmp runs.</div><div><br></div><div> command line used:</div><div><b> mpirun -np 2 -x OMP_NUM_THREADS=4 ../../../exe/Linux-x86-64-<wbr>aocc/cp2k.psmp ./H2O-32.inp<br></b></div><div><br></div><div>psmp executable runs fine with 1 mpi rank and 4 openmp threads. but it fails with 2 mpi rank and 4 openmp threads. </div><div><br></div><div>Following is the error generated with 2 mpi rank and 4 openmp threads. the behavior is same for all other input sets like H2O-64.inp, H20-128.inp H20-256.inp, H20-1024.inp</div><div><br></div><div><br></div><div><div> *****************************<wbr>******************************<wbr>********************</div><div> * ___ *</div><div> * / *</div><div> *[ABORT] *</div><div> * ___/ Cholesky decomposition failed. Matrix ill conditioned ? *</div><div> * | *</div><div> *O/| / *</div><div> *| | / *</div><div> * /home/amd/cp2k_aocc/cp2k-6.1/<wbr>src/cp_dbcsr_cholesky.F:121 *</div><div> *****************************<wbr>******************************<wbr>********************</div><div><br></div><div><br></div><div> ===== Routine Calling Stack =====</div><div><br></div><div> 12 cp_dbcsr_cholesky_decompose</div><div> 11 qs_ot_get_derivative</div><div> 10 ot_mini</div><div> 9 ot_scf_mini</div><div> 8 qs_scf_loop_do_ot</div><div> 7 qs_scf_new_mos</div><div> 6 scf_env_do_scf_inner_loop</div><div> 5 scf_env_do_scf</div><div> 4 qs_energies</div><div> 3 qs_forces</div><div> 2 qs_mol_dyn_low</div><div> 1 CP2K</div></div><div><br></div><div> following are the combinations where the executable work/ dont work</div><div><br></div><div><p class="MsoNormal"> </p>
<table border="0" cellspacing="0" cellpadding="0" width="345" style="width:259.0pt;border-collapse:collapse">
<tbody><tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-top:1pt solid windowtext;border-left:1pt solid windowtext;border-bottom-width:initial;border-bottom-style:none;border-right-width:initial;border-right-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">MPI processes</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-right-width:initial;border-bottom-width:initial;border-left-width:initial;border-style:solid none none;border-top-width:1pt;border-top-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">OPENMP threads</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top:1pt solid windowtext;border-left-width:initial;border-left-style:none;border-bottom-width:initial;border-bottom-style:none;border-right:1pt solid windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal"><span style="color:black">works/not works</span></p>
</td>
</tr>
<tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-width:1pt;border-style:solid;border-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">1</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-top:1pt solid windowtext;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left-width:initial;border-left-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">1</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top:1pt solid windowtext;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left-width:initial;border-left-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">works</span></p>
</td>
</tr>
<tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top-width:initial;border-top-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">1</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">2</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">works</span></p>
</td>
</tr>
<tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top-width:initial;border-top-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">1</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">4</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">works</span></p>
</td>
</tr>
<tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top-width:initial;border-top-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">1</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">8</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">works</span></p>
</td>
</tr>
<tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top-width:initial;border-top-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">1</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">16</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">works</span></p>
</td>
</tr>
<tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top-width:initial;border-top-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">1</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">32</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">works</span></p>
</td>
</tr>
<tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top-width:initial;border-top-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">1</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">64</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">works</span></p>
</td>
</tr>
<tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top-width:initial;border-top-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">2</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">1</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">works</span></p>
</td>
</tr>
<tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top-width:initial;border-top-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">2</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">2</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">works</span></p>
</td>
</tr>
<tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top-width:initial;border-top-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:red">2</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:red">4</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><font color="#ff0000">not works</font></p>
</td>
</tr>
<tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top-width:initial;border-top-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">2</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">8</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><font color="#000000">not works</font></p>
</td>
</tr>
<tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top-width:initial;border-top-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">2</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">16</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><font color="#000000">not works</font></p>
</td>
</tr>
<tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top-width:initial;border-top-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">2</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">32</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><font color="#000000">not works</font></p>
</td>
</tr>
<tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top-width:initial;border-top-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">2</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">64</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><font color="#000000">not works</font></p>
</td>
</tr>
<tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top-width:initial;border-top-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">4</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">1</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">works</span></p>
</td>
</tr>
<tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top-width:initial;border-top-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">4</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">2</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">works</span></p>
</td>
</tr>
<tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top-width:initial;border-top-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:red">4</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:red">4</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><font color="#ff0000">not works</font></p>
</td>
</tr>
<tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top-width:initial;border-top-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">8</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">1</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">works</span></p>
</td>
</tr>
<tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top-width:initial;border-top-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">8</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">2</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">works</span></p>
</td>
</tr>
<tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top-width:initial;border-top-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">8</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">3</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">works</span></p>
</td>
</tr>
<tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top-width:initial;border-top-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:red">8</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:red">4</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><font color="#ff0000">not works</font></p>
</td>
</tr>
<tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top-width:initial;border-top-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">16</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">1</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">works</span></p>
</td>
</tr>
<tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top-width:initial;border-top-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">16</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">2</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:black">works</span></p>
</td>
</tr>
<tr style="height:.2in">
<td width="91" nowrap valign="bottom" style="width:68pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top-width:initial;border-top-style:none;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:red">16</span></p>
</td>
<td width="108" nowrap valign="bottom" style="width:81pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><span style="color:red">3</span></p>
</td>
<td width="147" nowrap valign="bottom" style="width:110pt;border-top-width:initial;border-style:none solid solid none;border-left-width:initial;border-bottom-width:1pt;border-bottom-color:windowtext;border-right-width:1pt;border-right-color:windowtext;padding:0in 5.4pt;height:0.2in">
<p class="MsoNormal" align="center" style="text-align:center"><font color="#ff0000">not works</font></p>
</td>
</tr>
</tbody></table>
<p class="MsoNormal"> </p></div><div><br></div><div>what may be the possible reason for this behavior and what may be the right way to debug this issue?. I tried both with openmpi and mpich and both give similar results. The compiler is in house compiler.</div><div><br></div><div>Thanks for your help,</div><div>Shivarama Rao</div></div></blockquote></div></div></blockquote></div></div></blockquote></div></div></div></blockquote></div></div>