I was loading it as it was needed for compilation. I have unloaded the module, but the error still occurs: <div><br /></div><div><font face="Courier New">```<br />LIBXSMM_VERSION: develop-1.17-3834 (25693946)<br />CLX/DP TRY JIT STA COL<br /> 0..13 2 2 0 0 <br /> 14..23 0 0 0 0 <br /> 24..64 0 0 0 0 <br />Registry and code: 13 MB + 16 KB (gemm=2)<br />Command (PID=15485): /lustre/pd01/hpc-kuchta-1716987452/software/cp2k/exe/local/cp2k.psmp -i H2O-9.inp -o H2O-9.out<br />Uptime: 1.757102 s<br /><br />===================================================================================<br />= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES<br />= RANK 0 PID 15485 RUNNING AT r30c01b01<br />= KILLED BY SIGNAL: 11 (Segmentation fault)<br />===================================================================================<br /><br />===================================================================================<br />= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES<br />= RANK 1 PID 15486 RUNNING AT r30c01b01<br />= KILLED BY SIGNAL: 9 (Killed)<br />===================================================================================<br />```</font><br /><br />and the last 100 lines:<br /><br /><font face="Courier New">```<br /> 000000:000002>> 11 37 pw_create_c1d start <br /> Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002<< 11 37 pw_create_c1d 0.000 <br /> Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002<< 10 64 pw_pool_create_pw 0.000<br /> Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002>> 10 25 pw_copy start Hostmem: <br /> 697 MB GPUmem: 0 MB<br /> 000000:000002<< 10 25 pw_copy 0.001 Hostmem: <br /> 697 MB GPUmem: 0 MB<br /> 000000:000002>> 10 17 pw_axpy start Hostmem: <br /> 697 MB GPUmem: 0 MB<br /> 000000:000002<< 10 17 pw_axpy 0.001 Hostmem: <br /> 697 MB GPUmem: 0 MB<br /> 000000:000002>> 10 19 mp_sum_d start Hostmem:<br /> 697 MB GPUmem: 0 MB<br /> 000000:000002<< 10 19 mp_sum_d 0.000 Hostmem:<br /> 697 MB GPUmem: 0 MB<br /> 000000:000002>> 10 3 pw_poisson_solve start <br /> Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002>> 11 3 pw_poisson_rebuild s<br /> tart Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002<< 11 3 pw_poisson_rebuild 0<br /> .000 Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002>> 11 65 pw_pool_create_pw st<br /> art Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002>> 12 38 pw_create_c1d sta<br /> rt Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002<< 12 38 pw_create_c1d 0.0<br /> 00 Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002<< 11 65 pw_pool_create_pw 0.<br /> 000 Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002>> 11 26 pw_copy start Hostme<br /> m: 697 MB GPUmem: 0 MB<br /> 000000:000002<< 11 26 pw_copy 0.001 Hostme<br /> m: 697 MB GPUmem: 0 MB<br /> 000000:000002>> 11 3 pw_multiply_with sta<br /> rt Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002<< 11 3 pw_multiply_with 0.0<br /> 01 Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002>> 11 27 pw_copy start Hostme<br /> m: 697 MB GPUmem: 0 MB<br /> 000000:000002<< 11 27 pw_copy 0.001 Hostme<br /> m: 697 MB GPUmem: 0 MB<br /> 000000:000002>> 11 3 pw_integral_ab start<br /> Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002>> 12 20 mp_sum_d start Ho<br /> stmem: 697 MB GPUmem: 0 MB<br /> 000000:000002<< 12 20 mp_sum_d 0.001 Ho<br /> stmem: 697 MB GPUmem: 0 MB<br /> 000000:000002<< 11 3 pw_integral_ab 0.004<br /> Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002>> 11 4 pw_poisson_set start<br /> Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002>> 12 66 pw_pool_create_pw <br /> start Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002>> 13 39 pw_create_c1d <br /> start Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002<< 13 39 pw_create_c1d <br /> 0.000 Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002<< 12 66 pw_pool_create_pw <br /> 0.000 Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002>> 12 28 pw_copy start Hos<br /> tmem: 697 MB GPUmem: 0 MB<br /> 000000:000002<< 12 28 pw_copy 0.001 Hos<br /> tmem: 697 MB GPUmem: 0 MB<br /> 000000:000002>> 12 7 pw_derive start H<br /> ostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002<< 12 7 pw_derive 0.002 H<br /> ostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002>> 12 67 pw_pool_create_pw <br /> start Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002>> 13 40 pw_create_c1d <br /> start Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002<< 13 40 pw_create_c1d <br /> 0.000 Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002<< 12 67 pw_pool_create_pw <br /> 0.000 Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002>> 12 29 pw_copy start Hos<br /> tmem: 697 MB GPUmem: 0 MB<br /> 000000:000002<< 12 29 pw_copy 0.001 Hos<br /> tmem: 697 MB GPUmem: 0 MB<br /> 000000:000002>> 12 8 pw_derive start H<br /> ostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002<< 12 8 pw_derive 0.002 H<br /> ostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002>> 12 68 pw_pool_create_pw <br /> start Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002>> 13 41 pw_create_c1d <br /> start Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002<< 13 41 pw_create_c1d <br /> 0.000 Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002<< 12 68 pw_pool_create_pw <br /> 0.000 Hostmem: 697 MB GPUmem: 0 MB<br /> 000000:000002>> 12 30 pw_copy start Hos<br /> tmem: 697 MB GPUmem: 0 MB<br /> 000000:000002<< 12 30 pw_copy 0.001 Hos<br /> tmem: 697 MB GPUmem: 0 MB<br /> 000000:000002>> 12 9 pw_derive start H<br /> ostmem: 697 MB GPUmem: 0 MB<br /> ```</font><br /><br /></div><div>This is the list of currently loaded modules (all come with intel):</div><div><br /></div><div><font face="Courier New">```</font></div><div><font face="Courier New">Currently Loaded Modulefiles:<br /> 1) GCCcore/13.3.0 7) impi/2021.13.0-intel-compilers-2024.2.0 <br /> 2) zlib/1.3.1-GCCcore-13.3.0 8) imkl/2024.2.0 <br /> 3) binutils/2.42-GCCcore-13.3.0 9) iimpi/2024a <br /> 4) intel-compilers/2024.2.0 10) imkl-FFTW/2024.2.0-iimpi-2024a <br /> 5) numactl/2.0.18-GCCcore-13.3.0 11) intel/2024a <br /> 6) UCX/1.16.0-GCCcore-13.3.0 </font></div><div><font face="Courier New">```</font></div><div class="gmail_quote"><div dir="auto" class="gmail_attr">wtorek, 22 października 2024 o 11:12:57 UTC+2 Frederick Stein napisał(a):<br/></div><blockquote class="gmail_quote" style="margin: 0 0 0 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div>Dear Bartosz,</div><div>I am currently running some tests with the latest Intel compiler myself. What bothers me about your setup is the module GCC13/13.3.0 . Why is it loaded? Can you unload it? This would at least reduce potential interferences with between the Intel and the GCC compilers.</div><div>Best,</div><div>Frederick<br></div><br><div class="gmail_quote"><div dir="auto" class="gmail_attr">bartosz mazur schrieb am Montag, 21. Oktober 2024 um 16:33:45 UTC+2:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">The error for ssmp is:<div><br></div><div><font face="Courier New">```</font></div><div><font face="Courier New"></font></div><div><font face="Courier New">LIBXSMM_VERSION: develop-1.17-3834 (25693946)<br>CLX/DP TRY JIT STA COL<br></font></div><div><font face="Courier New"> 0..13 4 4 0 0 <br></font></div><div><font face="Courier New"> 14..23 0 0 0 0 <br> 24..64 0 0 0 0 <br></font></div><div><font face="Courier New">Registry and code: 13 MB + 32 KB (gemm=4)<br>Command (PID=54845): /lustre/pd01/hpc-kuchta-1716987452/software/cp2k/exe/local/cp2k.ssmp -i H2O-9.inp -o H2O-9.out<br>Uptime: 2.861583 s<br>/var/spool/slurmd/r30c01b15/job3120330/slurm_script: line 36: 54845 Segmentation fault (core dumped) /lustre/pd01/hpc-kuchta-1716987452/software/cp2k/exe/local/cp2k.ssmp -i H2O-9.inp -o H2O-9.out</font></div><div><font face="Courier New">```</font></div><div><br></div><div>and the last 100 lines of output:</div><div><br></div><div><font face="Courier New">```</font></div><div><font face="Courier New"> 000000:000001>> 12 20 mp_sum_d start Ho<br> stmem: 380 MB GPUmem: 0 MB<br> 000000:000001<< 12 20 mp_sum_d 0.000 Ho<br> stmem: 380 MB GPUmem: 0 MB<br> 000000:000001<< 11 13 dbcsr_dot_sd 0.000 H<br> ostmem: 380 MB GPUmem: 0 MB<br> 000000:000001<< 10 12 calculate_ptrace_kp 0.0<br> 00 Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001<< 9 6 evaluate_core_matrix_traces <br> 0.000 Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001>> 9 6 rebuild_ks_matrix start Ho<br> stmem: 380 MB GPUmem: 0 MB<br> 000000:000001>> 10 6 qs_ks_build_kohn_sham_matrix <br> start Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001>> 11 140 pw_pool_create_pw st<br> art Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001>> 12 79 pw_create_c1d sta<br> rt Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001<< 12 79 pw_create_c1d 0.0<br> 00 Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001<< 11 140 pw_pool_create_pw 0.<br> 000 Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001>> 11 141 pw_pool_create_pw st<br> art Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001>> 12 80 pw_create_c1d sta<br> rt Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001<< 12 80 pw_create_c1d 0.0<br> 00 Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001<< 11 141 pw_pool_create_pw 0.<br> 000 Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001>> 11 61 pw_copy start Hostme<br> m: 380 MB GPUmem: 0 MB<br> 000000:000001<< 11 61 pw_copy 0.004 Hostme<br> m: 380 MB GPUmem: 0 MB<br> 000000:000001>> 11 35 pw_axpy start Hostme<br> m: 380 MB GPUmem: 0 MB<br> 000000:000001<< 11 35 pw_axpy 0.002 Hostme<br> m: 380 MB GPUmem: 0 MB<br> 000000:000001>> 11 6 pw_poisson_solve sta<br> rt Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001>> 12 6 pw_poisson_rebuild <br> start Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001<< 12 6 pw_poisson_rebuild <br> 0.000 Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001>> 12 142 pw_pool_create_pw <br> start Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001>> 13 81 pw_create_c1d <br> start Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001<< 13 81 pw_create_c1d <br> 0.000 Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001<< 12 142 pw_pool_create_pw <br> 0.000 Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001>> 12 62 pw_copy start Hos<br> tmem: 380 MB GPUmem: 0 MB<br> 000000:000001<< 12 62 pw_copy 0.003 Hos<br> tmem: 380 MB GPUmem: 0 MB<br> 000000:000001>> 12 6 pw_multiply_with <br> start Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001<< 12 6 pw_multiply_with <br> 0.002 Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001>> 12 63 pw_copy start Hos<br> tmem: 380 MB GPUmem: 0 MB<br> 000000:000001<< 12 63 pw_copy 0.003 Hos<br> tmem: 380 MB GPUmem: 0 MB<br> 000000:000001>> 12 6 pw_integral_ab st<br> art Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001<< 12 6 pw_integral_ab 0.<br> 005 Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001>> 12 7 pw_poisson_set st<br> art Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001>> 13 143 pw_pool_create_pw <br> start Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001>> 14 82 pw_create_c1d <br> start Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001<< 14 82 pw_create_c1d <br> 0.000 Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001<< 13 143 pw_pool_create_pw <br> 0.000 Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001>> 13 64 pw_copy start <br> Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001<< 13 64 pw_copy 0.003 <br> Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001>> 13 16 pw_derive star<br> t Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001<< 13 16 pw_derive 0.00<br> 6 Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001>> 13 144 pw_pool_create_pw <br> start Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001>> 14 83 pw_create_c1d <br> start Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001<< 14 83 pw_create_c1d <br> 0.000 Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001<< 13 144 pw_pool_create_pw <br> 0.000 Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001>> 13 65 pw_copy start <br> Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001<< 13 65 pw_copy 0.004 <br> Hostmem: 380 MB GPUmem: 0 MB<br> 000000:000001>> 13 17 pw_derive star<br> t Hostmem: 380 MB GPUmem: 0 MB</font></div><div><font face="Courier New">```</font></div><div><br></div><div>for psmp the last 100 lines is:</div><div><font face="Courier New"><br></font></div><div><font face="Courier New">```</font></div><div><font face="Courier New"> 000000:000002<< 9 7 evaluate_core_matrix_traces <br></font></div><div><font face="Courier New"> 0.000 Hostmem: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002>> 9 7 rebuild_ks_matrix start Ho</font></div><div><font face="Courier New"><br> stmem: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002>> 10 7 qs_ks_build_kohn_sham_matrix <br></font></div><div><font face="Courier New"> start Hostmem: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002>> 11 164 pw_pool_create_pw st<br> art Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002>> 12 93 pw_create_c1d sta<br> rt Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002<< 12 93 pw_create_c1d 0.0<br> 00 Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002<< 11 164 pw_pool_create_pw 0.<br> 000 Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002>> 11 165 pw_pool_create_pw st<br> art Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002>> 12 94 pw_create_c1d sta<br> rt Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002<< 12 94 pw_create_c1d 0.0<br> 00 Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002<< 11 165 pw_pool_create_pw 0.<br> 000 Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002>> 11 73 pw_copy start Hostme</font></div><div><font face="Courier New"><br> m: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002<< 11 73 pw_copy 0.001 Hostme</font></div><div><font face="Courier New"><br> m: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002>> 11 41 pw_axpy start Hostme</font></div><div><font face="Courier New"><br> m: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002<< 11 41 pw_axpy 0.001 Hostme</font></div><div><font face="Courier New"><br> m: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002>> 11 52 mp_sum_d start Hostm</font></div><div><font face="Courier New"><br> em: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002<< 11 52 mp_sum_d 0.000 Hostm</font></div><div><font face="Courier New"><br> em: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002>> 11 7 pw_poisson_solve sta<br> rt Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002>> 12 7 pw_poisson_rebuild <br></font></div><div><font face="Courier New"> start Hostmem: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002<< 12 7 pw_poisson_rebuild <br></font></div><div><font face="Courier New"> 0.000 Hostmem: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002>> 12 166 pw_pool_create_pw </font></div><div><font face="Courier New"><br> start Hostmem: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002>> 13 95 pw_create_c1d <br></font></div><div><font face="Courier New"> start Hostmem: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002<< 13 95 pw_create_c1d <br></font></div><div><font face="Courier New"> 0.000 Hostmem: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002<< 12 166 pw_pool_create_pw </font></div><div><font face="Courier New"><br> 0.000 Hostmem: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002>> 12 74 pw_copy start Hos</font></div><div><font face="Courier New"><br> tmem: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002<< 12 74 pw_copy 0.001 Hos</font></div><div><font face="Courier New"><br> tmem: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002>> 12 7 pw_multiply_with <br></font></div><div><font face="Courier New"> start Hostmem: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002<< 12 7 pw_multiply_with <br></font></div><div><font face="Courier New"> 0.001 Hostmem: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002>> 12 75 pw_copy start Hos</font></div><div><font face="Courier New"><br> tmem: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002<< 12 75 pw_copy 0.001 Hos</font></div><div><font face="Courier New"><br> tmem: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002>> 12 7 pw_integral_ab st<br> art Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002>> 13 53 mp_sum_d start</font></div><div><font face="Courier New"><br> Hostmem: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002<< 13 53 mp_sum_d 0.000</font></div><div><font face="Courier New"><br> Hostmem: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002<< 12 7 pw_integral_ab 0.<br> 003 Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002>> 12 8 pw_poisson_set st<br> art Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002>> 13 167 pw_pool_create_pw <br></font></div><div><font face="Courier New"> start Hostmem: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002>> 14 96 pw_create_c1d </font></div><div><font face="Courier New"><br> start Hostmem: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002<< 14 96 pw_create_c1d </font></div><div><font face="Courier New"><br> 0.000 Hostmem: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002<< 13 167 pw_pool_create_pw <br></font></div><div><font face="Courier New"> 0.000 Hostmem: 693 MB GPUmem: 0 MB<br></font></div><div><font face="Courier New"> 000000:000002>> 13 76 pw_copy start <br></font></div><div><font face="Courier New"> Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002<< 13 76 pw_copy 0.001 <br> Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002>> 13 19 pw_derive star<br> t Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002<< 13 19 pw_derive 0.00<br> 2 Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002>> 13 168 pw_pool_create_pw <br> start Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002>> 14 97 pw_create_c1d <br> start Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002<< 14 97 pw_create_c1d <br> 0.000 Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002<< 13 168 pw_pool_create_pw <br> 0.000 Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002>> 13 77 pw_copy start <br> Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002<< 13 77 pw_copy 0.001 <br> Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002>> 13 20 pw_derive star<br> t Hostmem: 693 MB GPUmem: 0 MB</font></div><div><font face="Courier New"></font></div><div><font face="Courier New">```</font></div><div><br></div><div>Thanks</div><div>Bartosz<br><br></div><div class="gmail_quote"><div dir="auto" class="gmail_attr">poniedziałek, 21 października 2024 o 08:58:34 UTC+2 Frederick Stein napisał(a):<br></div><blockquote class="gmail_quote" style="margin:0 0 0 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Dear Bartosz,</div><div>I have no idea about the issue with LibXSMM.</div><div>Regarding the trace, I do not know either as there is not much that could break in pw_derive (it just performs multiplications) and the sequence of operations is to unspecific. It may be that the code actually breaks somewhere else. Can you do the same with the ssmp and post the last 100 lines? This way, we remove the asynchronicity issues for backtraces with the psmp version.</div><div>Best,</div><div>Frederick<br></div><br><div class="gmail_quote"><div dir="auto" class="gmail_attr">bartosz mazur schrieb am Sonntag, 20. Oktober 2024 um 16:47:15 UTC+2:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">The error is:<div><br></div><div><font face="Courier New">```</font></div><div><font face="Courier New"></font></div><div><font face="Courier New">LIBXSMM_VERSION: develop-1.17-3834 (25693946)<br>CLX/DP TRY JIT STA COL<br></font></div><div><font face="Courier New"> 0..13 2 2 0 0<br> 14..23 0 0 0 0</font></div><div><font face="Courier New"><br> 24..64 0 0 0 0<br></font></div><div><font face="Courier New">Registry and code: 13 MB + 16 KB (gemm=2)<br>Command (PID=2607388): /lustre/pd01/hpc-kuchta-1716987452/software/cp2k/exe/local/cp2k.psmp -i H2O-9.inp -o H2O-9.out<br>Uptime: 5.288243 s</font></div><div><font face="Courier New"><br><br>===================================================================================<br>= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES<br></font></div><div><font face="Courier New">= RANK 0 PID 2607388 RUNNING AT r21c01b10</font></div><div><font face="Courier New"><br>= KILLED BY SIGNAL: 11 (Segmentation fault)<br>===================================================================================<br><br>===================================================================================<br>= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES<br></font></div><div><font face="Courier New">= RANK 1 PID 2607389 RUNNING AT r21c01b10<br>= KILLED BY SIGNAL: 9 (Killed)<br>===================================================================================<br></font></div><div><font face="Courier New">```</font></div><div><br></div><div>and the last 20 lines:</div><div><br></div><div><font face="Courier New">```</font></div><div><font face="Courier New"> 000000:000002<< 13 76 pw_copy 0.001<br> Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002>> 13 19 pw_derive star<br> t Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002<< 13 19 pw_derive 0.00<br> 2 Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002>> 13 168 pw_pool_create_pw<br> start Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002>> 14 97 pw_create_c1d<br> start Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002<< 14 97 pw_create_c1d<br> 0.000 Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002<< 13 168 pw_pool_create_pw<br> 0.000 Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002>> 13 77 pw_copy start<br> Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002<< 13 77 pw_copy 0.001<br> Hostmem: 693 MB GPUmem: 0 MB<br> 000000:000002>> 13 20 pw_derive star<br> t Hostmem: 693 MB GPUmem: 0 MB</font></div><div><font face="Courier New">```</font><br><br></div><div>Thanks!</div><div class="gmail_quote"><div dir="auto" class="gmail_attr">piątek, 18 października 2024 o 17:18:39 UTC+2 Frederick Stein napisał(a):<br></div><blockquote class="gmail_quote" style="margin:0 0 0 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Please pick one of the failing tests. Then, add the TRACE keyword to the &GLOBAL section and then run the test manually. This increases the size of the output file dramatically (to some million lines). Can you send me the last ~20 lines of the output?<br><div class="gmail_quote"><div dir="auto" class="gmail_attr">bartosz mazur schrieb am Freitag, 18. Oktober 2024 um 17:09:40 UTC+2:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I'm using do_regtests.py script, not make regtesting, but I assume it makes no difference. As I mentioned in previous message for `--ompthreads 1` all tests were passed both for ssmp and psmp. For ssmp with `--ompthreads 2` I observe similar errors as for psmp with the same setting, I provide example output as attachment. <div><br></div><div>Thanks</div><div>Bartosz<br><br></div><div class="gmail_quote"><div dir="auto" class="gmail_attr">piątek, 18 października 2024 o 16:24:16 UTC+2 Frederick Stein napisał(a):<br></div><blockquote class="gmail_quote" style="margin:0 0 0 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Dear Bartosz,<br></div><div>What happens if you set the number of OpenMP threads to 1 (add '--ompthreads 1' to TESTOPTS)? What errors do you observe in case of the ssmp?</div><div>Best,</div><div>Frederick<br></div><br><div class="gmail_quote"><div dir="auto" class="gmail_attr">bartosz mazur schrieb am Freitag, 18. Oktober 2024 um 15:37:43 UTC+2:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Frederick,<div><br></div><div>thanks again for help. So I have tested different simulation variants and I know that the problem occurs when using OMP. For MPI calculations without OMP all tests pass. I have also tested the effect of the <font face="Courier New">`OMP_PROC_BIND` </font>and <font face="Courier New">`OMP_PLACES`</font> parameters and apart from the effect on simulation time, they have no significant effect on the presence of errors. Below are the results for ssmp:</div><div><br></div><div><font face="Courier New">```</font></div><div><font face="Courier New">OMP_PROC_BIND, OMP_PLACES, correct, total, wrong, failed, time <br>spread, threads, 3850, 4144, 4, 290, 186min<br>spread, cores, 3831, 4144, 3, 310, 183min<br>spread, sockets, 3864, 4144, 3, 277, 104min<br>close, threads, 3879, 4144, 3, 262, 171min<br>close, cores, 3854, 4144, 0, 290, 168min<br>close, sockets, 3865, 4144, 3, 276, 104min<br>master, threads, 4121, 4144, 0, 23, 1002min<br>master, cores, 4121, 4144, 0, 23, 986min<br>master, sockets, 3942, 4144, 3, 199, 219min<br>false, threads, 3918, 4144, 0, 226, 178min<br>false, cores, 3919, 4144, 3, 222, 176min<br>false, sockets, 3856, 4144, 4, 284, 104min<br>```</font></div><div><br></div><div>and psmp:</div><div><br></div><div><font face="Courier New">```</font></div><div><font face="Courier New">OMP_PROC_BIND, OMP_PLACES, results<br>spread, threads, Summary: correct: 4097 / 4227; failed: 130; 495min<br>spread, cores, 26 / 362<br>spread, cores, 26 / 362<br>close, threads, Summary: correct: 4133 / 4227; failed: 94; 484min<br>close, cores, 60 / 362<br>close, sockets, 13 / 362<br>master, threads, 13 / 362<br>master, cores, 79 / 362<br>master, sockets, Summary: correct: 4153 / 4227; failed: 74; 563min<br>false, threads, Summary: correct: 4153 / 4227; failed: 74; 556min<br>false, cores, Summary: correct: 4106 / 4227; failed: 121; 511min<br>false, sockets, 96 / 362</font></div><div><font face="Courier New">not specified, not specified, Summary: correct: 4129 / 4227; failed: 98; 263min</font><br></div><div><font face="Courier New">```</font></div><div><br></div><div>Any ideas what I could do next to have more information about the source of the problem or maybe you see a potential solution at this stage? I would appreciate any further help. <br></div><div><br></div><div>Best</div><div>Bartosz</div><div><br></div><div><br></div><div class="gmail_quote"><div dir="auto" class="gmail_attr">piątek, 11 października 2024 o 14:30:25 UTC+2 Frederick Stein napisał(a):<br></div><blockquote class="gmail_quote" style="margin:0 0 0 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Dear Bartosz,</div><div>If I am not mistaken, you used 8 OpenMP threads. The test do not run that efficiently with such a large number of threads. 2 should be sufficient.</div><div>The test result suggests that most of the functionality may work but due to a missing backtrace (or similar information), it is hard to tell why they fail. You could also try to run some of the single-node tests to assess the stability of CP2K.<br></div><div>Best,</div><div>Frederick<br></div><br><div class="gmail_quote"><div dir="auto" class="gmail_attr">bartosz mazur schrieb am Freitag, 11. Oktober 2024 um 13:48:42 UTC+2:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Sorry, forgot attachments.<div><br></div></blockquote></div></blockquote></div></blockquote></div></blockquote></div></blockquote></div></blockquote></div></blockquote></div></blockquote></div></blockquote></div></blockquote></div>
<p></p>
-- <br />
You received this message because you are subscribed to the Google Groups "cp2k" group.<br />
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:cp2k+unsubscribe@googlegroups.com">cp2k+unsubscribe@googlegroups.com</a>.<br />
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/cp2k/d3cc4f35-c1e5-4685-831f-53e4eb90eb90n%40googlegroups.com?utm_medium=email&utm_source=footer">https://groups.google.com/d/msgid/cp2k/d3cc4f35-c1e5-4685-831f-53e4eb90eb90n%40googlegroups.com</a>.<br />