[CP2K-user] [CP2K:19916] Re: FFTW_PLAN_TYPE PATIENT and EXHAUSTIVE not working

Frederick Stein f.stein at hzdr.de
Fri Feb 16 09:49:51 UTC 2024


Dear Léon,
Regarding 1:
Did you optimize the accuracy-relevant parameters (CUTOFF, EPS_DEFAULT) to 
meet your demands regarding energies and gradients? If not, there is an how 
to concerning this topic (https://www.cp2k.org/howto:converging_cutoff). 
Choose the cutoff as high as necessary and as low as possible.
TLDR: From your timing report, I recommend you to employ the MEASURE mode, 
see below for a more lengthy explanation.
If you have optimized these parameters, check the timing report. The total 
CP2K runtime is given by the line with "CP2K" (ignoring warming up times of 
the cluster and of CP2K). Each line represents a routine within CP2K, from 
the name you may guess its purpose/task. Relevant to you are especially the 
last four columns. The last two columns indicate the average and maximum 
time spent in a given routine and its called routines. They give you an 
indication which functionality spends the most time. The fourth and third 
last columns indicate the the average and maximum time spent in the routine 
itself, i.e. without the routines called by the given routine. These two 
columns indicate in which compute kernel CP2K spends its most amount of 
time. If you check the routines with the largest self-time, you find the 
highest self-times of FFT-related routines, i.e. those containing 'fft' in 
their names, in case of fft3d_ps (10122.9 s, 3.9 % of CP2K) and fft_wrap_pw1pw2_200 
(5123.8 s, 2.0 % of CP2K). Even if you were able to reduce the timing of 
these routines to zero, you would gain ca. 6 % of computation time, in 
practice it will be much less. So, I think that a further reduction would 
not be able to reduce the computational time significantly.
For completeness, the difference to the total runtime of the 
fft_wrap_pw1pw2 routine of ~50000 s is related to the respective 
communication routines which do not have 'fft' in their name. These are of 
course not affected by the FFTW planning times.

Regarding 2:
Checking the source code of the routine fft_get_scratch in 
src/pw/fft_tools.F indicates that the issue might be related to an issue on 
a lower level routine within CP2K (probably src/fft/fftw3_lib.F) or in 
FFTW3 directly. I did some local tests with one of the single-node tests 
and did not observe any issues in these modes. So, the issue will need more 
thorough investigations.

HTH
Frederick
Léon Luntadila Lufungula schrieb am Freitag, 16. Februar 2024 um 10:01:04 
UTC+1:

> Dear Frederick,
>
> Thanks for guiding me in solving this issue!
>
> 1) I have not yet checked if this is indeed the case but I'm quite new to 
> AIMD calculations and the reference manual stated that this was recommended 
> for long AIMD trajectories, so I assumed this would be a good option to 
> enable. If you could show me how to check this, I would greatly appreciate 
> the help! For the same calculation with the default FFTW_PLAN_TYPE ESTIMATE 
> I get the following timings:
>
>
>  -------------------------------------------------------------------------------
>  -                                                                         
>     -
>  -                                T I M I N G                             
>      -
>  -                                                                         
>     -
>
>  -------------------------------------------------------------------------------
>  SUBROUTINE                       CALLS  ASD         SELF TIME       
>  TOTAL TIME
>                                 MAXIMUM       AVERAGE  MAXIMUM  AVERAGE 
>  MAXIMUM
>  CP2K                                 1  1.0      0.0      0.0 259198.3 
> 259201.0
>  qs_mol_dyn_low                       1  2.0      0.5      0.6 259198.0 
> 259200.7
>  velocity_verlet                  21600  3.0      4.7      5.3 259117.8 
> 259120.7
>  qs_forces                        21601  4.0      6.7      8.3 259054.1 
> 259057.9
>  qs_energies                      21601  5.0      1.3      2.0 216944.6 
> 216962.9
>  scf_env_do_scf                   21601  6.0      0.7      1.0 197423.7 
> 197426.9
>  scf_env_do_scf_inner_loop       172819  7.0      2.9      3.7 147536.9 
> 147542.8
>  rebuild_ks_matrix               194420  8.7      0.7      0.7 113213.6 
> 113417.8
>  qs_ks_build_kohn_sham_matrix    194420  9.7     21.9     25.3 113212.9 
> 113417.2
>  qs_ks_update_qs_env             194420  8.0      1.5      1.8  85131.3 
>  85315.6
> * pw_transfer                    4104421 12.9    259.1    293.8  53272.8 
>  54657.8*
>  sum_up_and_integrate            194420 10.7    158.9    210.4  54433.6 
>  54518.3
>  integrate_v_rspace              194420 11.7      6.8      8.9  54274.2 
>  54375.6
>  
> *fft_wrap_pw1pw2                3715581 14.0     41.8     46.1  52788.9 
>  54160.2 fft_wrap_pw1pw2_200            2160221 15.4   5123.8   5554.3 
>  50067.4  51727.8*
>  init_scf_loop                    21601  7.0      0.6      0.8  49795.6 
>  49797.8
>  qs_vxc_create                   194420 10.7      4.5      5.3  49235.6 
>  49729.8
>  xc_vxc_pw_create                194420 11.7    990.7   1608.7  49231.1 
>  49725.4
>  qs_rho_update_rho_low           194420  8.1      0.8      0.9  44519.7 
>  44594.7
>  calculate_rho_elec              194420  9.1    109.2    184.2  44518.9 
>  44593.9
>  fft3d_ps                       3715581 16.0  10122.9  14717.9  41927.7 
>  43177.5
>  dbcsr_multiply_generic         2981151 12.6    130.7    138.3  42145.8 
>  42708.1
>  prepare_preconditioner           21601  8.0      0.1      0.2  38552.7 
>  38578.0
>  grid_integrate_task_list        194420 12.7  34380.1  37401.8  34380.1 
>  37401.8
>  xc_pw_derive                   1166520 13.7      9.8     11.0  34752.0 
>  36587.5
>  make_preconditioner              21601  9.0      0.4      0.6  36451.6 
>  36464.5
>  qs_scf_new_mos                  172819  8.0      0.9      1.4  34340.9 
>  34469.5
>  qs_scf_loop_do_ot               172819  9.0      0.8      1.0  34340.0 
>  34468.7
>  make_full_all                    21601 10.0      4.6      5.3  34250.9 
>  34260.8
>  ot_scf_mini                     172819 10.0      4.0      4.7  31468.4 
>  31598.1
>  multiply_cannon                2981151 13.6    192.3    222.5  29228.9 
>  30932.4
>  mp_alltoall_z22v               3715581 18.0  27949.9  30346.5  27949.9 
>  30346.5
>  multiply_cannon_loop           2981151 14.6    215.7    231.9  27541.2 
>  28832.2
>  xc_pw_divergence                194420 12.7      4.0      4.7  26628.5 
>  28220.2
>  qs_ks_update_qs_env_forces       21601  5.0      0.1      0.2  28174.2 
>  28195.0
>  xc_rho_set_and_dset_create      194420 12.7    239.1    269.1  20595.1 
>  28022.0
>  ot_mini                         172819 11.0      1.0      1.4  23618.8 
>  23759.5
>  grid_collocate_task_list        194420 10.1  22590.9  23442.1  22590.9 
>  23442.1
>  rs_pw_transfer                 1987402 12.3     41.9     48.7  21225.8 
>  22117.5
>  yz_to_x                         993701 17.7    742.7   1149.4  19795.7 
>  21777.5
>  density_rs2pw                   194420 10.1      8.1      8.9  20317.1 
>  21233.4
>  mp_waitall_1                   ******* -4.3  17314.7  19114.1  17314.7 
>  19114.1
>  multiply_cannon_multrec        ******* 15.6  17135.6  18128.8  17149.2 
>  18142.9
>  cp_fm_diag_elpa                  64803 11.3      0.4      0.5  18118.8 
>  18125.4
>  cp_fm_diag_elpa_base             64803 12.3  17919.4  18033.2  18104.5 
>  18104.8
>  mp_waitany                     ******* 14.1  13555.6  14684.3  13555.6 
>  14684.3
>  xc_functional_eval              194420 13.7      2.0      2.6   8289.8 
>  14473.2
>  pbe_lda_eval                    194420 14.7   8287.8  14472.3   8287.8 
>  14472.3
>
>
> 2) Unfortunately, no pdbg-version was compiled on my HPC center, but I did 
> run a calculation with TRACE and TRACE_MASTER enabled. See attached files 
> for the output.
>
> Kind regards,
> Léon
> On Friday 16 February 2024 at 09:36:56 UTC+1 Frederick Stein wrote:
>
>> Dear Leon,
>>
>> I do not know the root of the error and I cannot suggest a solution. The 
>> options themself are tested within our regtest suite and we do not find any 
>> issues there. So, it seems to be a more complicated problem either on FFTW 
>> site or on CP2K site.
>>
>> I have two questions:
>>
>> 1. Did you check whether the FFT kernel actually needs an improvement? 
>> Check the runtime of the routines pw_transfer and those starting with 
>> fft_wrap_pw1pw2 (or similar).
>>
>> 2. Do you have a pdbg-version of CP2K available? If yes, can you run one 
>> of the failing tests with that one? It might also help to turn on the 
>> keywords TRACE and TRACE_MASTER in the GLOBAL section of your input files 
>> to identify the actual routine on CP2K site where the error occurs.
>>
>> Regards,
>>
>> Frederick
>>
>> Léon Luntadila Lufungula schrieb am Montag, 12. Februar 2024 um 10:42:16 
>> UTC+1:
>>
>>> Dear all,
>>>
>>> I've been running some AIMD calculations and am trying to speed up the 
>>> calculations a bit by playing with the FFTW_PLAN_TYPE option. 
>>> Unfortunately, only MEASURE and the default ESTIMATE are working. If I try 
>>> to set it to PATIENT (as recommended for long AIMD runs) or EXHAUSTIVE, the 
>>> calculation crashes almost immediately with the following error messages 
>>> (see also attached files):
>>>
>>> [PATIENT]
>>>
>>> ...
>>>
>>> corrupted double-linked list
>>>
>>> corrupted double-linked list (not small)
>>>
>>> cp2k.popt: malloc.c:4106: _int_malloc: Assertion `(unsigned long) (size) 
>>> >= (unsigned long) (nb)' failed.
>>>
>>>  
>>>
>>> Program received signal SIGABRT: Process abort signal.
>>>
>>> ...
>>>
>>> [EXHAUSTIVE]
>>>
>>> ...
>>>
>>> malloc_consolidate(): unaligned fastbin chunk detected
>>>
>>> malloc_consolidate(): unaligned fastbin chunk detected
>>>
>>>  
>>>
>>> Program received signal SIGABRT: Process abort signal.
>>>
>>>  
>>>
>>> Backtrace for this error:
>>>
>>> corrupted double-linked list
>>>
>>>  
>>>
>>> Program received signal SIGABRT: Process abort signal.
>>>
>>>  
>>>
>>> Backtrace for this error:
>>>
>>> malloc_consolidate(): unaligned fastbin chunk detected
>>>
>>>  
>>>
>>> Program received signal SIGABRT: Process abort signal.
>>>
>>>  
>>>
>>> Backtrace for this error:
>>>
>>>  
>>>
>>> Program received signal SIGABRT: Process abort signal.
>>>
>>>  
>>>
>>> Backtrace for this error:
>>>
>>> corrupted double-linked list
>>>
>>> cp2k.popt: malloc.c:4106: _int_malloc: Assertion `(unsigned long) (size) 
>>> >= (unsigned long) (nb)' failed.
>>>
>>> ...
>>>
>>> I'm running CP2K/2023.2-foss-2022a as compiled with Easybuild by our HPC 
>>> centre, but the same problems appear when I try the CP2K/2022.1-foss-2022a 
>>> version. However, when I run it with the CP2K/7.1-intel-2020a version which 
>>> is also available, both EXHAUSTIVE and PATIENT seem to be working 
>>> properly... Is this something that can be solved in some way or will this 
>>> require a different compilation of CP2K, possibly with the intel toolchain 
>>> instead of the foss toolchain? 
>>>
>>>
>>> Kind regards,
>>>
>>> Léon
>>>
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+unsubscribe at googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/70911d18-487c-47c3-8bbe-00521ee5c6cen%40googlegroups.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.cp2k.org/archives/cp2k-user/attachments/20240216/2e3e5740/attachment-0001.htm>


More information about the CP2K-user mailing list