<div dir="ltr"><div>Hi Ben,</div><div>it seems that ELPA problem is fixed. Now you are missing BLAS library linking, indeed I don't see the library in LIBS inside your arch file.</div><div>I think your directory is</div><div><br></div><div>/<wbr>usr/local/blas-3.5.0/lib<br></div><div><br></div><div>and the library should be -lrefblas</div><div><br></div><div>Alfio</div><br><br>Il giorno venerdì 15 gennaio 2016 17:23:08 UTC+1, Ben Fotovich ha scritto:<blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;"><div dir="ltr">Hey Michael,<br><br>When I modify cp2k-3.0/arch/Linux-x86-64-<wbr>gfortran.popt and change -D__ELPA2 to -D__ELPA3, below is what happens when I try to compile:<br><br>/usr/local/openmpi-1.8.4/bin/<wbr>mpif90 -D__FFTW3 -D__LIBINT -D__ELPA3 -D__LIBINT_MAX_AM=7 -D__LIBDERIV_MAX_AM1=6 -D__MAX_CONTR=4 -D__parallel -D__SCALAPACK -O2 -ffast-math -ffree-form -ffree-line-length-none -ftree-vectorize -funroll-loops -mtune=native -I/usr/local/fftw-3.3.4/<wbr>include -I/usr/local/libint-1.1.4/<wbr>include -I/usr/local/elpa-2015.11.001/<wbr>include/elpa/modules -D__COMPILE_ARCH="\"Linux-x86-<wbr>64-gfortran\"" -D__COMPILE_DATE="\"Fri Jan 15 09:11:28 MST 2016\"" -D__COMPILE_HOST="\"du-hpc\"" -D__COMPILE_REVISION="\"svn:<wbr>16458\"" -D__DATA_DIR="\"/root/c/cp2k-<wbr>3.0/data\"" -L -L/lib64 -L/root/c/cp2k-3.0/lib/Linux-<wbr>x86-64-gfortran/popt -o /root/c/cp2k-3.0/exe/Linux-<wbr>x86-64-gfortran/dbcsr_<wbr>performance_driver.popt dbcsr_performance_driver.o  -ldbcsr -ldbcsrarnoldi -ldbcsrvec -ldbcsrmm -ldbcsrops -ldbcsrbblock -ldbcsrbdist -ldbcsrbase -ldbcsrdata -ldbcsrerr -lcp2kmpiwrap -lmicsmm -lclsmm -lcp2kacc -lcp2kaccmic -lcp2kaccopencl -lcp2kbase /usr/local/scalapack-2.0.2-<wbr>gcc/lib/libscalapack-gnu.a /usr/local/fftw-3.3.4/lib/<wbr>libfftw3.a /usr/local/libint-1.1.4/lib/<wbr>libderiv.a /usr/local/libint-1.1.4/lib/<wbr>libint.a -L/usr/local/elpa-2015.11.001/<wbr>lib/libelpa_openmp.a<br>/root/c/cp2k-3.0/lib/Linux-<wbr>x86-64-gfortran/popt/libdbcsr.<wbr>a(dbcsr_test_methods.o): In function `__dbcsr_test_methods_MOD_<wbr>dbcsr_make_random_matrix':<br>dbcsr_test_methods.F:(.text+<wbr>0x1198): undefined reference to `dlarnv_'<br>/root/c/cp2k-3.0/lib/Linux-<wbr>x86-64-gfortran/popt/<wbr>libdbcsrmm.a(dbcsr_mm_hostdrv.<wbr>o): In function `__dbcsr_mm_hostdrv_MOD_blas_<wbr>process_mm_stack_s':<br>dbcsr_mm_hostdrv.F:(.text+<wbr>0x98): undefined reference to `sgemm_'<br>dbcsr_mm_hostdrv.F:(.text+<wbr>0x109): undefined reference to `sgemm_'<br>dbcsr_mm_hostdrv.F:(.text+<wbr>0x175): undefined reference to `sgemm_'<br>dbcsr_mm_hostdrv.F:(.text+<wbr>0x1cb): undefined reference to `sgemm_'<br>/root/c/cp2k-3.0/lib/Linux-<wbr>x86-64-gfortran/popt/<wbr>libdbcsrmm.a(dbcsr_mm_hostdrv.<wbr>o): In function `__dbcsr_mm_hostdrv_MOD_blas_<wbr>process_mm_stack_d':<br>dbcsr_mm_hostdrv.F:(.text+<wbr>0x288): undefined reference to `dgemm_'<br>dbcsr_mm_hostdrv.F:(.text+<wbr>0x2f9): undefined reference to `dgemm_'<br>dbcsr_mm_hostdrv.F:(.text+<wbr>0x365): undefined reference to `dgemm_'<br>dbcsr_mm_hostdrv.F:(.text+<wbr>0x3bb): undefined reference to `dgemm_'<br>/root/c/cp2k-3.0/lib/Linux-<wbr>x86-64-gfortran/popt/<wbr>libdbcsrmm.a(dbcsr_mm_hostdrv.<wbr>o): In function `__dbcsr_mm_hostdrv_MOD_blas_<wbr>process_mm_stack_c':<br>dbcsr_mm_hostdrv.F:(.text+<wbr>0x478): undefined reference to `cgemm_'<br>dbcsr_mm_hostdrv.F:(.text+<wbr>0x4e9): undefined reference to `cgemm_'<br>dbcsr_mm_hostdrv.F:(.text+<wbr>0x555): undefined reference to `cgemm_'<br>dbcsr_mm_hostdrv.F:(.text+<wbr>0x5ab): undefined reference to `cgemm_'<br>/root/c/cp2k-3.0/lib/Linux-<wbr>x86-64-gfortran/popt/<wbr>libdbcsrmm.a(dbcsr_mm_hostdrv.<wbr>o): In function `__dbcsr_mm_hostdrv_MOD_blas_<wbr>process_mm_stack_z':<br>dbcsr_mm_hostdrv.F:(.text+<wbr>0x678): undefined reference to `zgemm_'<br>dbcsr_mm_hostdrv.F:(.text+<wbr>0x6ff): undefined reference to `zgemm_'<br>dbcsr_mm_hostdrv.F:(.text+<wbr>0x77e): undefined reference to `zgemm_'<br>dbcsr_mm_hostdrv.F:(.text+<wbr>0x7e6): undefined reference to `zgemm_'<br>/root/c/cp2k-3.0/lib/Linux-<wbr>x86-64-gfortran/popt/<wbr>libdbcsrops.a(dbcsr_blas_<wbr>operations.o): In function `__dbcsr_blas_operations_MOD_<wbr>dbcsr_lapack_larnv':<br>dbcsr_blas_operations.F:(.<wbr>text+0x227): undefined reference to `slarnv_'<br>dbcsr_blas_operations.F:(.<wbr>text+0x301): undefined reference to `dlarnv_'<br>/root/c/cp2k-3.0/lib/Linux-<wbr>x86-64-gfortran/popt/<wbr>libdbcsrops.a(dbcsr_<wbr>operations.o): In function `__dbcsr_operations_MOD_dbcsr_<wbr>trace_ab_d':<br>dbcsr_operations.F:(.text+<wbr>0x13b7): undefined reference to `ddot_'<br>dbcsr_operations.F:(.text+<wbr>0x1445): undefined reference to `cdotu_'<br>dbcsr_operations.F:(.text+<wbr>0x1833): undefined reference to `zdotu_'<br>dbcsr_operations.F:(.text+<wbr>0x18e3): undefined reference to `sdot_'<br>/root/c/cp2k-3.0/lib/Linux-<wbr>x86-64-gfortran/popt/<wbr>libdbcsrops.a(dbcsr_<wbr>operations.o): In function `__dbcsr_operations_MOD_dbcsr_<wbr>trace_ab_c':<br>dbcsr_operations.F:(.text+<wbr>0x2726): undefined reference to `ddot_'<br>dbcsr_operations.F:(.text+<wbr>0x27d0): undefined reference to `ddot_'<br>dbcsr_operations.F:(.text+<wbr>0x28ae): undefined reference to `cdotu_'<br>dbcsr_operations.F:(.text+<wbr>0x2985): undefined reference to `cdotu_'<br>dbcsr_operations.F:(.text+<wbr>0x2dbd): undefined reference to `zdotu_'<br>dbcsr_operations.F:(.text+<wbr>0x2e70): undefined reference to `zdotu_'<br>dbcsr_operations.F:(.text+<wbr>0x2f56): undefined reference to `sdot_'<br>dbcsr_operations.F:(.text+<wbr>0x300f): undefined reference to `sdot_'<br>/root/c/cp2k-3.0/lib/Linux-<wbr>x86-64-gfortran/popt/<wbr>libdbcsrops.a(dbcsr_<wbr>operations.o): In function `__dbcsr_operations_MOD_dbcsr_<wbr>trace_ab_s':<br>dbcsr_operations.F:(.text+<wbr>0x4590): undefined reference to `ddot_'<br>dbcsr_operations.F:(.text+<wbr>0x4624): undefined reference to `cdotu_'<br>dbcsr_operations.F:(.text+<wbr>0x4a42): undefined reference to `zdotu_'<br>dbcsr_operations.F:(.text+<wbr>0x4afd): undefined reference to `sdot_'<br>/root/c/cp2k-3.0/lib/Linux-<wbr>x86-64-gfortran/popt/<wbr>libdbcsrops.a(dbcsr_<wbr>operations.o): In function `__dbcsr_operations_MOD_dbcsr_<wbr>trace_ab_z':<br>dbcsr_operations.F:(.text+<wbr>0x5fb5): undefined reference to `ddot_'<br>dbcsr_operations.F:(.text+<wbr>0x606b): undefined reference to `ddot_'<br>dbcsr_operations.F:(.text+<wbr>0x6136): undefined reference to `cdotu_'<br>dbcsr_operations.F:(.text+<wbr>0x6577): undefined reference to `zdotu_'<br>dbcsr_operations.F:(.text+<wbr>0x663e): undefined reference to `zdotu_'<br>dbcsr_operations.F:(.text+<wbr>0x670d): undefined reference to `sdot_'<br>dbcsr_operations.F:(.text+<wbr>0x67cf): undefined reference to `sdot_'<br>/root/c/cp2k-3.0/lib/Linux-<wbr>x86-64-gfortran/popt/<wbr>libdbcsrops.a(dbcsr_<wbr>operations.o): In function `__dbcsr_operations_MOD_dbcsr_<wbr>filter_anytype':<br>dbcsr_operations.F:(.text+<wbr>0xd57f): undefined reference to `dznrm2_'<br>dbcsr_operations.F:(.text+<wbr>0xd6bf): undefined reference to `sdot_'<br>dbcsr_operations.F:(.text+<wbr>0xd79c): undefined reference to `ddot_'<br>dbcsr_operations.F:(.text+<wbr>0xd89f): undefined reference to `scnrm2_'<br>/root/c/cp2k-3.0/lib/Linux-<wbr>x86-64-gfortran/popt/<wbr>libdbcsrops.a(dbcsr_<wbr>operations.o): In function `__dbcsr_operations_MOD_dbcsr_<wbr>init_random':<br>dbcsr_operations.F:(.text+<wbr>0x11955): undefined reference to `dlarnv_'<br>dbcsr_operations.F:(.text+<wbr>0x11a7f): undefined reference to `dcopy_'<br>dbcsr_operations.F:(.text+<wbr>0x11b3c): undefined reference to `dcopy_'<br>/root/c/cp2k-3.0/lib/Linux-<wbr>x86-64-gfortran/popt/<wbr>libdbcsrbblock.a(dbcsr_block_<wbr>access.o): In function `__dbcsr_block_access_MOD_<wbr>dbcsr_put_block_c':<br>dbcsr_block_access.F:(.text+<wbr>0x6de): undefined reference to `ccopy_'<br>dbcsr_block_access.F:(.text+<wbr>0x1563): undefined reference to `caxpy_'<br>dbcsr_block_access.F:(.text+<wbr>0x1c4c): undefined reference to `caxpy_'<br>dbcsr_block_access.F:(.text+<wbr>0x1d23): undefined reference to `ccopy_'<br>/root/c/cp2k-3.0/lib/Linux-<wbr>x86-64-gfortran/popt/<wbr>libdbcsrbblock.a(dbcsr_block_<wbr>access.o): In function `__dbcsr_block_access_MOD_<wbr>dbcsr_put_block_s':<br>dbcsr_block_access.F:(.text+<wbr>0x3025): undefined reference to `scopy_'<br>dbcsr_block_access.F:(.text+<wbr>0x3cf0): undefined reference to `saxpy_'<br>dbcsr_block_access.F:(.text+<wbr>0x431d): undefined reference to `saxpy_'<br>dbcsr_block_access.F:(.text+<wbr>0x43f9): undefined reference to `scopy_'<br>/root/c/cp2k-3.0/lib/Linux-<wbr>x86-64-gfortran/popt/<wbr>libdbcsrbblock.a(dbcsr_block_<wbr>access.o): In function `__dbcsr_block_access_MOD_<wbr>dbcsr_put_block_z':<br>dbcsr_block_access.F:(.text+<wbr>0x5661): undefined reference to `zcopy_'<br>dbcsr_block_access.F:(.text+<wbr>0x63e6): undefined reference to `zaxpy_'<br>dbcsr_block_access.F:(.text+<wbr>0x6ab7): undefined reference to `zaxpy_'<br>dbcsr_block_access.F:(.text+<wbr>0x6b8f): undefined reference to `zcopy_'<br>/root/c/cp2k-3.0/lib/Linux-<wbr>x86-64-gfortran/popt/<wbr>libdbcsrbblock.a(dbcsr_block_<wbr>access.o): In function `__dbcsr_block_access_MOD_<wbr>dbcsr_put_block_d':<br>dbcsr_block_access.F:(.text+<wbr>0x7d6a): undefined reference to `dcopy_'<br>dbcsr_block_access.F:(.text+<wbr>0x88d8): undefined reference to `daxpy_'<br>dbcsr_block_access.F:(.text+<wbr>0x8edd): undefined reference to `daxpy_'<br>dbcsr_block_access.F:(.text+<wbr>0x8fb9): undefined reference to `dcopy_'<br>/usr/local/scalapack-2.0.2-<wbr>gcc/lib/libscalapack-gnu.a(<wbr>clarnv.o): In function `clarnv_':<br>clarnv.f:(.text+0x7a): undefined reference to `slaruv_'<br>clarnv.f:(.text+0x169): undefined reference to `slaruv_'<br>/usr/local/scalapack-2.0.2-<wbr>gcc/lib/libscalapack-gnu.a(<wbr>zlarnv.o): In function `zlarnv_':<br>zlarnv.f:(.text+0x7f): undefined reference to `dlaruv_'<br>zlarnv.f:(.text+0x165): undefined reference to `dlaruv_'<br>collect2: error: ld returned 1 exit status<br>make[3]: *** [/root/c/cp2k-3.0/exe/Linux-<wbr>x86-64-gfortran/dbcsr_<wbr>performance_driver.popt] Error 1<br><br>Thanks!<br><br>Ben<br><br>On Friday, January 15, 2016 at 2:52:05 AM UTC-7, Michael wrote:<blockquote class="gmail_quote" style="margin:0;margin-left:0.8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,
<br>g
<br>On Thu, Jan 14, 2016 at 02:14:24PM -0800, Ben Fotovich wrote:
<br>> I spoke too soon!
<br>> 
<br>> Looking in elpa/src/elpa_c_interface.F90
<br>> Line 117 shows:
<br>> 
<br>> function solve_elpa2_evp_real_wrapper(<wbr>na, nev, ncols, a, lda, ev, q, ldq, 
<br>> nblk,    &
<br>>                                   mpi_comm_rows, mpi_comm_cols, 
<br>> mpi_comm_all, &
<br>>                                   THIS_REAL_ELPA_KERNEL_API, 
<br>> useQR)           &
<br>>                                   result(success) 
<br>> bind(C,name="elpa_solve_evp_<wbr>real_2stage")
<br>> 
<br>> and line 141 shows:
<br>> 
<br>> successFortran = solve_evp_real_2stage(na, nev, a, lda, ev, q, ldq, nblk, 
<br>> mpi_comm_rows, mpi_comm_cols, mpi_comm_all, &
<br>>                                            THIS_REAL_ELPA_KERNEL_API, 
<br>> useQRFortran)
<br>> 
<br>> I may be misreading, but this looks like it is slightly different than what 
<br>> was called from the CP2K code. Thoughts?
<br>
<br>Indeed, ELPA changed interface between 2015.05 and 2015.11. As you are
<br>using 2015.11, you should defined -D__ELPA3 instead of -D__ELPA2 I
<br>believe, does that work?
<br>
<br>
<br>Michael
<br></blockquote></div></blockquote></div>