<div dir="ltr">Attached are my benchmark results. Do you think this could be further improved? The machine is <span style="color:rgb(0,0,0);font-family:Arial,Helvetica,Verdana,'Bitstream Vera Sans',sans-serif;font-size:12px;line-height:18px">40 nodes (n001-n040) with 16proc/ node = 640 Intel® Xeon® E5 Cloud Ready 2.4 GHz Compute Processor Cores based on E5-2665 544GB DDR3 1600Mhz ECC REG System Memory 1GB Memory Per Compute Processor Core 4x FDR infiniband Mellanox. Best performance is for </span> <span style="font-family:'Helvetica Neue';font-size:14px">4 threads 16 mpi processes and 4 nodes, speedup of ~32 .</span><div><font color="#000000" face="Arial, Helvetica, Verdana, Bitstream Vera Sans, sans-serif"><span style="font-size:12px;line-height:18px"><br></span></font></div><div><font color="#000000" face="Arial, Helvetica, Verdana, Bitstream Vera Sans, sans-serif"><span style="font-size:12px;line-height:18px">My archfile is </span></font></div><div><font color="#000000" face="Arial, Helvetica, Verdana, Bitstream Vera Sans, sans-serif"><span style="font-size:12px;line-height:18px"> </span></font></div><div><font color="#000000" face="Arial, Helvetica, Verdana, Bitstream Vera Sans, sans-serif"><div style=""><span style="font-size:12px;line-height:18px">CC = gcc</span></div><div style=""><span style="font-size:12px;line-height:18px">CPP =</span></div><div style=""><span style="font-size:12px;line-height:18px">FC = mpif90</span></div><div style=""><span style="font-size:12px;line-height:18px">LD = mpif90</span></div><div style=""><span style="font-size:12px;line-height:18px">AR = ar -r</span></div><div style=""><span style="font-size:12px;line-height:18px">PREFIX = /home/rcohen</span></div><div style=""><span style="font-size:12px;line-height:18px">FFTW_INC = $(PREFIX)/include</span></div><div style=""><span style="font-size:12px;line-height:18px">FFTW_LIB = $(PREFIX)/lib</span></div><div style=""><span style="font-size:12px;line-height:18px">LIBINT_INC = $(PREFIX)/include</span></div><div style=""><span style="font-size:12px;line-height:18px">LIBINT_LIB = $(PREFIX)/lib</span></div><div style=""><span style="font-size:12px;line-height:18px">LIBXC_INC = $(PREFIX)/include</span></div><div style=""><span style="font-size:12px;line-height:18px">LIBXC_LIB = $(PREFIX)/lib</span></div><div style=""><span style="font-size:12px;line-height:18px">GCC_LIB = $(PREFIX)/gcc-trunk/lib</span></div><div style=""><span style="font-size:12px;line-height:18px">GCC_LIB64 = $(PREFIX)/gcc-trunk/lib64</span></div><div style=""><span style="font-size:12px;line-height:18px">GCC_INC = $(PREFIX)/gcc-trunk/include</span></div><div style=""><span style="font-size:12px;line-height:18px">DFLAGS = -D__FFTW3 -D__LIBINT -D__LIBXC2\</span></div><div style=""><span style="font-size:12px;line-height:18px"> -D__LIBINT_MAX_AM=7 -D__LIBDERIV_MAX_AM1=6 -D__MAX_CONTR=4\</span></div><div style=""><span style="font-size:12px;line-height:18px"> -D__parallel -D__SCALAPACK -D__HAS_smm_dnn -D__ELPA3 </span></div><div style=""><span style="font-size:12px;line-height:18px">CPPFLAGS =</span></div><div style=""><span style="font-size:12px;line-height:18px">FCFLAGS = $(DFLAGS) -O2 -ffast-math -ffree-form -ffree-line-length-none\</span></div><div style=""><span style="font-size:12px;line-height:18px"> -fopenmp -ftree-vectorize -funroll-loops\</span></div><div style=""><span style="font-size:12px;line-height:18px"> -mtune=native \</span></div><div style=""><span style="font-size:12px;line-height:18px"> -I$(FFTW_INC) -I$(LIBINT_INC) -I$(LIBXC_INC) -I$(MKLROOT)/include \</span></div><div style=""><span style="font-size:12px;line-height:18px"> -I$(GCC_INC) -I$(PREFIX)/include/elpa_openmp-2015.11.001/modules</span></div><div style=""><span style="font-size:12px;line-height:18px">LIBS = \</span></div><div style=""><span style="font-size:12px;line-height:18px"> $(PREFIX)/lib/libscalapack.a $(PREFIX)/lib/libsmm_dnn_sandybridge-2015-11-10.a \</span></div><div style=""><span style="font-size:12px;line-height:18px"> $(FFTW_LIB)/libfftw3.a\</span></div><div style=""><span style="font-size:12px;line-height:18px"> $(FFTW_LIB)/libfftw3_threads.a\</span></div><div style=""><span style="font-size:12px;line-height:18px"> $(LIBXC_LIB)/libxcf90.a\</span></div><div style=""><span style="font-size:12px;line-height:18px"> $(LIBXC_LIB)/libxc.a\</span></div><div style=""><span style="font-size:12px;line-height:18px"> $(PREFIX)/lib/liblapack.a $(PREFIX)/lib/libtmglib.a $(PREFIX)/lib/libgomp.a \</span></div><div style=""><span style="font-size:12px;line-height:18px"> $(PREFIX)/lib/libderiv.a $(PREFIX)/lib/libint.a -lelpa_openmp -lgomp -lopenblas</span></div><div style=""><span style="font-size:12px;line-height:18px">LDFLAGS = $(FCFLAGS) -L$(GCC_LIB64) -L$(GCC_LIB) -static-libgfortran -L$(PREFIX)/lib </span></div><div style="font-size:12px;line-height:18px"><br></div><div style="font-size:12px;line-height:18px">gcc is 6.0.0</div><div style="font-size:12px;line-height:18px">gfortran</div><div style="font-size:12px;line-height:18px">openmpi 1.10.2</div><div style="font-size:12px;line-height:18px">scalapack 2.0.2</div><div style=""><span style="font-size:12px;line-height:18px">elpa-2015.11.001</span><br></div><div style=""><span style="font-size:12px;line-height:18px">libint 1.1.5 (I tried libint 2.0.3 but seems to be missing derivs)</span></div><div style=""><span style="font-size:12px;line-height:18px">libxc 2.2.2</span></div><div style=""><span style="font-size:12px;line-height:18px">openblas xianyi-OpenBLAS-c679dd1</span></div><div style=""><span style="font-size:12px;line-height:18px">mm_dnn_sandybridge-2015-11-10</span><span style="font-size:12px;line-height:18px"><br></span></div><div style=""><span style="font-size:12px;line-height:18px">fftw 3.3.4</span></div><div style=""><span style="font-size:12px;line-height:18px"><br></span></div><div style=""><span style="font-size:12px;line-height:18px"><img src="cid:ii_153a9858f38a8bb9" alt="Inline image 1" width="362" height="218"><br></span></div><div style=""><span style="font-size:12px;line-height:18px">Speedup total is for the whole benchmark, setup plus 30 timesteps.</span></div><div style=""><span style="font-size:12px;line-height:18px">Speedup step is for the time for the last time step.</span></div><div style=""><span style="font-size:12px;line-height:18px">64 Molecules H2O FPMD NVT within LDA</span></div><div style="font-size:12px;line-height:18px"><br></div></font><div><div class="gmail_signature">---<br>Ronald Cohen<br>Geophysical Laboratory<br>Carnegie Institution<br>5251 Broad Branch Rd., N.W.<br>Washington, D.C. 20015<br><a href="mailto:rco...@carnegiescience.edu" target="_blank">rco...@carnegiescience.edu</a><br>office: 202-478-8937<br>skype: ronaldcohen<br><a href="https://twitter.com/recohen3" target="_blank">https://twitter.com/recohen3</a><br><a href="https://www.linkedin.com/profile/view?id=163327727" target="_blank">https://www.linkedin.com/profile/view?id=163327727</a><br></div></div><div class="gmail_signature"><br></div> </div></div>