[CP2K-user] [CP2K:19278] On the time&memory consumption of ERI calculation in CP2K hybrid functional single point

[Roosevelt Gildner]

Dear Dr. Bussy

Really appreciate your detailed answer.
First apologizes for the lack of my detailed parameters description. In 
fact, I have considered about the MAX_MEMORY.
For the aforementioned example A and B, I have spared enough MAX_MEMORY. My 
setting strategy is: MAX_MEMORY* MPI ranks=90% of the physical total memory 
of the HPC array.
Therefore, with respect to the example B, I believe it is not of on-the-fly 
mode. For the stablized memory consumption is approximately 70%, no more 
than my setting, 90% of the total memory.
(The attached two figures are the HPC performance curves v.s. time. In 
which the deep yellow line refers to the real-time memory consumption 
percent, corresponding to the left Y axis)

The EPS_SCHWARZ and EPS_PGF_ORB are both set as 1E-8. I believe they are 
accurate enough, and actually it works well in example A. So I think these 
two parameters might not be the key factor.
Perhaps I have to loose the EPS_SCHWARZ and CUTOFF_RADIUS. However, it is 
really a pity that although I have enough physical memory space, when the 
memory consumption rises from 4TB (example A) to 7TB (example B), it seems 
to get stuck. It is really strange. Maybe the inherent limitation of 
CP2K?(I use psmp version, but the OpenMPI is default)

P.S. Thank you for your advice on ADMM. I have learned that ADMM is usually 
adopted with OT to realize the purification of the aux. basis. However, 
what I am interested in is the unoccupied orbitals of my structure. 
Therefore the strategy of ADMM+OT will not be an option.
Thank you again for your help. And for the creation of such a great ab 
initio software!

Regards,
RG
On Monday, September 25, 2023 at 5:04:18 PM UTC+8 Augustin Bussy wrote:

> Hi,
>
> what you observe is somewhat expected. The limiting factor in your B3LYP 
> calculation is the Hartree-Fock exchange, which computational performance 
> depends strongly on the choice of EPS_PGF_ORB, EPS_SCHWARZ,  CUTOFF_RADIUS 
> and MAX_MEMORY.
>
> In an ideal CP2K run, when performing HFX calculations, all of the ERIs 
> are initially calculated and stored in memory (during the first SCF step). 
> This allows for much faster timings for the next SCF steps, because ERIs 
> are simply fetched from memory. The keyword MAX_MEMORY from the &HF section 
> controls how much memory is dedicated to storing the ERIs (per MPI rank). 
> If there are more ERIs than available memory, the rest is computed on the 
> fly, at each SCF step. This is linked to your first question: when the 
> memory stabilizes, it means that you have reached that MAX_MEMORY upper 
> limit, and the new ERIs are not stored. You should adapt the value of 
> MAX_MEMORY to your hardware.
>
> For your second and third question: HFX performance in CP2K depends a lot 
> on the choice of EPS_SCHWARZ, EPS_PGF_ORB and CUTOFF_RADIUS. These 
> parameters control how many ERIs need to be calculated, and going from 
> 10E-8 to 10E-10 EPS_SCHWARZ while increasing the CUTOFF_RADIUS will lead to 
> a large number of additional ERIs to be calculated (and CP2K might seems 
> stuck because there is so much more to do). In most cases, I would say that 
> EPS_SCHWARZ = 1.0E-8,  EPS_PGF_ORB =1.0E-5 - 1.0E-6 and CUTOFF_RADIUS = 6 
> Angstrom is accurate enough and efficient.
>
> You can also consider the ADMM method to further accelerate your 
> calculation (https://pubs.acs.org/doi/full/10.1021/ct1002225).
>
> I hope that helps.
> Best,
> Augustin
> ------------------------------
> *From:* cp... at googlegroups.com <cp... at googlegroups.com> on behalf of 
> Roosevelt Gildner <roosevelt... at gmail.com>
> *Sent:* Saturday, September 23, 2023 4:29 PM
> *To:* cp2k <cp... at googlegroups.com>
> *Subject:* [CP2K:19271] On the time&memory consumption of ERI calculation 
> in CP2K hybrid functional single point 
>  
> Dear CP2K team,
>
> I am attempting to calculate the single point energy of a huge periodic 
> cluster (near 5k atoms, just contains C H O N, cell size 30+ angstrom, only 
> Gamma point). Due to the demand of following analysis, I used B3LYP hybrid 
> functional (with D3(BJ)) and 6-31G(d) basis for GAPW calculation.
> I have prepared enough memory space to do this (TB scale). Meanwhile I 
> want as accurate result (especially for DOS analysis, need unoccupied 
> orbitals) as possible, but not too excessive time comsuption.
> I have tried several parameter sets and found when the EPS_SCHWARZ and 
> CUTOFF_RADIUS is respectively over 8 (exponent) and 10 angstrom, the time 
> comsuption significant increases.
> Below are the two typical examples:
> A 
>
> EPS_SCHWARZ=1E-8.
>
> SCREEN_ON_INITIAL_P=T
>
> CUTOFF_RADIUS=8.0.
>
> The total time comsuption is about 25 hours, memory peak 4.2TB(not 
> on-the-fly). The HPC monitor tells me that the memory linear rised to the 
> stable level and soon the first SCF step completed.(total SCF=24 steps)
>
> [image: 微信图片_20230923213258.png]
>
> B
>
> EPS_SCHWARZ=1E-10.
>
> SCREEN_ON_INITIAL_P=T
>
> CUTOFF_RADIUS=10.0
> The memory comsuption stablized at about 7TB. But it has been 1 day (now 
> already 5 days) since the memory rised to the stable level, the first SCF 
> step is not completed yet.
> [image: 微信图片_20230922132420-2.png]
>
> Here are my confusions:
> 1 Whether the memory rising to the stable level means the ERI calculation 
> is completed? if yes, why the example B does not complete the first SCF 
> step even after 1-5 days? (the memory used is not 100%, so i believe it is 
> not on-the-fly mode)And why do example A and B take such different time 
> comsuption?
> 2 I am afraid the CP2K mission B got stuck. but I am not sure. Is there 
> any method to judge whether the mission is normal? And need I continue 
> waiting?
> 3 Which parameter(s) is the key factor(s) affecting the time comsuption of 
> hybrid functional calculation? EPS_DEFAULT? EPS_PGF_ORB? EPS_SCHWARZ? 
> CUTOFF_RADIUS?
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "cp2k" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to cp2k+uns... at googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/cp2k/57a56625-e467-4a50-ab54-665e7df6d445n%40googlegroups.com 
> <https://groups.google.com/d/msgid/cp2k/57a56625-e467-4a50-ab54-665e7df6d445n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+unsubscribe at googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/fc2df1c7-8475-4db6-a83d-ce6f917fc87an%40googlegroups.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.cp2k.org/archives/cp2k-user/attachments/20230925/d6582547/attachment.htm>