[CP2K-user] [CP2K:19916] Re: FFTW_PLAN_TYPE PATIENT and EXHAUSTIVE not working
Frederick Stein
f.stein at hzdr.de
Fri Feb 16 09:49:51 UTC 2024
Dear Léon,
Regarding 1:
Did you optimize the accuracy-relevant parameters (CUTOFF, EPS_DEFAULT) to
meet your demands regarding energies and gradients? If not, there is an how
to concerning this topic (https://www.cp2k.org/howto:converging_cutoff).
Choose the cutoff as high as necessary and as low as possible.
TLDR: From your timing report, I recommend you to employ the MEASURE mode,
see below for a more lengthy explanation.
If you have optimized these parameters, check the timing report. The total
CP2K runtime is given by the line with "CP2K" (ignoring warming up times of
the cluster and of CP2K). Each line represents a routine within CP2K, from
the name you may guess its purpose/task. Relevant to you are especially the
last four columns. The last two columns indicate the average and maximum
time spent in a given routine and its called routines. They give you an
indication which functionality spends the most time. The fourth and third
last columns indicate the the average and maximum time spent in the routine
itself, i.e. without the routines called by the given routine. These two
columns indicate in which compute kernel CP2K spends its most amount of
time. If you check the routines with the largest self-time, you find the
highest self-times of FFT-related routines, i.e. those containing 'fft' in
their names, in case of fft3d_ps (10122.9 s, 3.9 % of CP2K) and fft_wrap_pw1pw2_200
(5123.8 s, 2.0 % of CP2K). Even if you were able to reduce the timing of
these routines to zero, you would gain ca. 6 % of computation time, in
practice it will be much less. So, I think that a further reduction would
not be able to reduce the computational time significantly.
For completeness, the difference to the total runtime of the
fft_wrap_pw1pw2 routine of ~50000 s is related to the respective
communication routines which do not have 'fft' in their name. These are of
course not affected by the FFTW planning times.
Regarding 2:
Checking the source code of the routine fft_get_scratch in
src/pw/fft_tools.F indicates that the issue might be related to an issue on
a lower level routine within CP2K (probably src/fft/fftw3_lib.F) or in
FFTW3 directly. I did some local tests with one of the single-node tests
and did not observe any issues in these modes. So, the issue will need more
thorough investigations.
HTH
Frederick
Léon Luntadila Lufungula schrieb am Freitag, 16. Februar 2024 um 10:01:04
UTC+1:
> Dear Frederick,
>
> Thanks for guiding me in solving this issue!
>
> 1) I have not yet checked if this is indeed the case but I'm quite new to
> AIMD calculations and the reference manual stated that this was recommended
> for long AIMD trajectories, so I assumed this would be a good option to
> enable. If you could show me how to check this, I would greatly appreciate
> the help! For the same calculation with the default FFTW_PLAN_TYPE ESTIMATE
> I get the following timings:
>
>
> -------------------------------------------------------------------------------
> -
> -
> - T I M I N G
> -
> -
> -
>
> -------------------------------------------------------------------------------
> SUBROUTINE CALLS ASD SELF TIME
> TOTAL TIME
> MAXIMUM AVERAGE MAXIMUM AVERAGE
> MAXIMUM
> CP2K 1 1.0 0.0 0.0 259198.3
> 259201.0
> qs_mol_dyn_low 1 2.0 0.5 0.6 259198.0
> 259200.7
> velocity_verlet 21600 3.0 4.7 5.3 259117.8
> 259120.7
> qs_forces 21601 4.0 6.7 8.3 259054.1
> 259057.9
> qs_energies 21601 5.0 1.3 2.0 216944.6
> 216962.9
> scf_env_do_scf 21601 6.0 0.7 1.0 197423.7
> 197426.9
> scf_env_do_scf_inner_loop 172819 7.0 2.9 3.7 147536.9
> 147542.8
> rebuild_ks_matrix 194420 8.7 0.7 0.7 113213.6
> 113417.8
> qs_ks_build_kohn_sham_matrix 194420 9.7 21.9 25.3 113212.9
> 113417.2
> qs_ks_update_qs_env 194420 8.0 1.5 1.8 85131.3
> 85315.6
> * pw_transfer 4104421 12.9 259.1 293.8 53272.8
> 54657.8*
> sum_up_and_integrate 194420 10.7 158.9 210.4 54433.6
> 54518.3
> integrate_v_rspace 194420 11.7 6.8 8.9 54274.2
> 54375.6
>
> *fft_wrap_pw1pw2 3715581 14.0 41.8 46.1 52788.9
> 54160.2 fft_wrap_pw1pw2_200 2160221 15.4 5123.8 5554.3
> 50067.4 51727.8*
> init_scf_loop 21601 7.0 0.6 0.8 49795.6
> 49797.8
> qs_vxc_create 194420 10.7 4.5 5.3 49235.6
> 49729.8
> xc_vxc_pw_create 194420 11.7 990.7 1608.7 49231.1
> 49725.4
> qs_rho_update_rho_low 194420 8.1 0.8 0.9 44519.7
> 44594.7
> calculate_rho_elec 194420 9.1 109.2 184.2 44518.9
> 44593.9
> fft3d_ps 3715581 16.0 10122.9 14717.9 41927.7
> 43177.5
> dbcsr_multiply_generic 2981151 12.6 130.7 138.3 42145.8
> 42708.1
> prepare_preconditioner 21601 8.0 0.1 0.2 38552.7
> 38578.0
> grid_integrate_task_list 194420 12.7 34380.1 37401.8 34380.1
> 37401.8
> xc_pw_derive 1166520 13.7 9.8 11.0 34752.0
> 36587.5
> make_preconditioner 21601 9.0 0.4 0.6 36451.6
> 36464.5
> qs_scf_new_mos 172819 8.0 0.9 1.4 34340.9
> 34469.5
> qs_scf_loop_do_ot 172819 9.0 0.8 1.0 34340.0
> 34468.7
> make_full_all 21601 10.0 4.6 5.3 34250.9
> 34260.8
> ot_scf_mini 172819 10.0 4.0 4.7 31468.4
> 31598.1
> multiply_cannon 2981151 13.6 192.3 222.5 29228.9
> 30932.4
> mp_alltoall_z22v 3715581 18.0 27949.9 30346.5 27949.9
> 30346.5
> multiply_cannon_loop 2981151 14.6 215.7 231.9 27541.2
> 28832.2
> xc_pw_divergence 194420 12.7 4.0 4.7 26628.5
> 28220.2
> qs_ks_update_qs_env_forces 21601 5.0 0.1 0.2 28174.2
> 28195.0
> xc_rho_set_and_dset_create 194420 12.7 239.1 269.1 20595.1
> 28022.0
> ot_mini 172819 11.0 1.0 1.4 23618.8
> 23759.5
> grid_collocate_task_list 194420 10.1 22590.9 23442.1 22590.9
> 23442.1
> rs_pw_transfer 1987402 12.3 41.9 48.7 21225.8
> 22117.5
> yz_to_x 993701 17.7 742.7 1149.4 19795.7
> 21777.5
> density_rs2pw 194420 10.1 8.1 8.9 20317.1
> 21233.4
> mp_waitall_1 ******* -4.3 17314.7 19114.1 17314.7
> 19114.1
> multiply_cannon_multrec ******* 15.6 17135.6 18128.8 17149.2
> 18142.9
> cp_fm_diag_elpa 64803 11.3 0.4 0.5 18118.8
> 18125.4
> cp_fm_diag_elpa_base 64803 12.3 17919.4 18033.2 18104.5
> 18104.8
> mp_waitany ******* 14.1 13555.6 14684.3 13555.6
> 14684.3
> xc_functional_eval 194420 13.7 2.0 2.6 8289.8
> 14473.2
> pbe_lda_eval 194420 14.7 8287.8 14472.3 8287.8
> 14472.3
>
>
> 2) Unfortunately, no pdbg-version was compiled on my HPC center, but I did
> run a calculation with TRACE and TRACE_MASTER enabled. See attached files
> for the output.
>
> Kind regards,
> Léon
> On Friday 16 February 2024 at 09:36:56 UTC+1 Frederick Stein wrote:
>
>> Dear Leon,
>>
>> I do not know the root of the error and I cannot suggest a solution. The
>> options themself are tested within our regtest suite and we do not find any
>> issues there. So, it seems to be a more complicated problem either on FFTW
>> site or on CP2K site.
>>
>> I have two questions:
>>
>> 1. Did you check whether the FFT kernel actually needs an improvement?
>> Check the runtime of the routines pw_transfer and those starting with
>> fft_wrap_pw1pw2 (or similar).
>>
>> 2. Do you have a pdbg-version of CP2K available? If yes, can you run one
>> of the failing tests with that one? It might also help to turn on the
>> keywords TRACE and TRACE_MASTER in the GLOBAL section of your input files
>> to identify the actual routine on CP2K site where the error occurs.
>>
>> Regards,
>>
>> Frederick
>>
>> Léon Luntadila Lufungula schrieb am Montag, 12. Februar 2024 um 10:42:16
>> UTC+1:
>>
>>> Dear all,
>>>
>>> I've been running some AIMD calculations and am trying to speed up the
>>> calculations a bit by playing with the FFTW_PLAN_TYPE option.
>>> Unfortunately, only MEASURE and the default ESTIMATE are working. If I try
>>> to set it to PATIENT (as recommended for long AIMD runs) or EXHAUSTIVE, the
>>> calculation crashes almost immediately with the following error messages
>>> (see also attached files):
>>>
>>> [PATIENT]
>>>
>>> ...
>>>
>>> corrupted double-linked list
>>>
>>> corrupted double-linked list (not small)
>>>
>>> cp2k.popt: malloc.c:4106: _int_malloc: Assertion `(unsigned long) (size)
>>> >= (unsigned long) (nb)' failed.
>>>
>>>
>>>
>>> Program received signal SIGABRT: Process abort signal.
>>>
>>> ...
>>>
>>> [EXHAUSTIVE]
>>>
>>> ...
>>>
>>> malloc_consolidate(): unaligned fastbin chunk detected
>>>
>>> malloc_consolidate(): unaligned fastbin chunk detected
>>>
>>>
>>>
>>> Program received signal SIGABRT: Process abort signal.
>>>
>>>
>>>
>>> Backtrace for this error:
>>>
>>> corrupted double-linked list
>>>
>>>
>>>
>>> Program received signal SIGABRT: Process abort signal.
>>>
>>>
>>>
>>> Backtrace for this error:
>>>
>>> malloc_consolidate(): unaligned fastbin chunk detected
>>>
>>>
>>>
>>> Program received signal SIGABRT: Process abort signal.
>>>
>>>
>>>
>>> Backtrace for this error:
>>>
>>>
>>>
>>> Program received signal SIGABRT: Process abort signal.
>>>
>>>
>>>
>>> Backtrace for this error:
>>>
>>> corrupted double-linked list
>>>
>>> cp2k.popt: malloc.c:4106: _int_malloc: Assertion `(unsigned long) (size)
>>> >= (unsigned long) (nb)' failed.
>>>
>>> ...
>>>
>>> I'm running CP2K/2023.2-foss-2022a as compiled with Easybuild by our HPC
>>> centre, but the same problems appear when I try the CP2K/2022.1-foss-2022a
>>> version. However, when I run it with the CP2K/7.1-intel-2020a version which
>>> is also available, both EXHAUSTIVE and PATIENT seem to be working
>>> properly... Is this something that can be solved in some way or will this
>>> require a different compilation of CP2K, possibly with the intel toolchain
>>> instead of the foss toolchain?
>>>
>>>
>>> Kind regards,
>>>
>>> Léon
>>>
>>>
>>>
--
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+unsubscribe at googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/70911d18-487c-47c3-8bbe-00521ee5c6cen%40googlegroups.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.cp2k.org/archives/cp2k-user/attachments/20240216/2e3e5740/attachment-0001.htm>
More information about the CP2K-user
mailing list