[CP2K-user] [CP2K:20728] Re: out of memory: C3N4 (56 atoms) HES06 + RI-HFX + ADMM

[Augustin Bussy]

Dear Jiang,

Starting from the default atomic guess, the first SCF step typically 
requires less memory (for somewhat technical reasons). This indicates that 
you run at the edge of what your machine can deal with.

I have a few comments on your new input file:
1) You stopped using ADMM. The cost of HFX calculations (both in computing 
time and memory) depends a lot on the basis set, and particularly on its 
diffuseness. MOLOPT basis sets are not optimized for HFX calculation. I 
suggest you switch back to ADMM.
2) You switched the interaction potential from SHORTRANGE to TC. While TC 
is typically more efficient, the cutoff radius you use (4.0 Ang) is very 
short. I would us at least 6.0 in your case. Note also that this is not the 
HSE06 functional any more, for this, you need to revert to SHORTRANGE. In a 
2D system like yours, it should be affordable. With the TC potential, you 
can use functionals like PBE0 or B3LYP.
3) The value of EPS_SCHWARZ is fairly large, for safety, I would try to run 
with 1.0E-7 or 1.0E-8
4) You should change your ratio of MPI ranks and OMP threads. When memory 
is limited, it is good practice to reduce the number of ranks. With your 
setup, I recommend 32 ranks and 4 threads.

Note that running with a PRINT_LEVEL MEDIUM can give you valuable extra 
information on the cost of your calculation (in relation to RI-HFXk 
parameters, for example).
Best,
Augustin
On Monday 23 September 2024 at 09:37:22 UTC+2 yj jiang wrote:

> Dear Bussy, Stein, and other CP2K experts,
>
> With your guidance, I have been able to run this job on a 512GB HPC. 
> However, I still encounter some issues.
>
> After running the calculations several times, I keep running into memory 
> problems right after the first SCF iteration ends (upon diagonalizing the 
> first Fock matrix). Could you please advise on where the memory consumption 
> lies and how I can optimize it to reduce memory usage?
>
> Moreover, could you please check if there are any other precision-related 
> issues in this input file? I am particularly concerned about the settings 
> in the INTERACTION_POTENTIAL section.
>
> Thank you very much for your assistance!
>
> Best regards,
> Jiang
>
> 在2024年9月22日星期日 UTC+8 17:13:50<yj jiang> 写道：
>
>> Deer Bussy
>>
>> Your suggestions have been extremely helpful. I am now able to run this 
>> system on a 512GB HPC.
>> The running speed is acceptable to me, considering the large amount of 
>> computation involved. 
>> Thank you very much for your help.
>>
>> Best wishes.
>> Jiang
>> 在2024年9月19日星期四 UTC+8 21:33:29<Augustin Bussy> 写道：
>>
>>> Hi Jiang,
>>> I was able to run your input file on a 512G system, but I'd like to add 
>>> a few comments:
>>> 1) I somehow missed the fact that you are running RI-HFX with K-points. 
>>> In such a case, the choice of the RI metric is not crucial for performance 
>>> (or memory usage), and the best course of action is to keep the default.
>>> 2) What really matters is the number of atoms in the unit cell, the 
>>> diffuseness of the basis set, and the range of the HFX potential. There is 
>>> not much you can do here, especially since you already use ADMM. The range 
>>> of the HFX potential really matters, because this influences how many 
>>> periodic images of your system need to be considered. Typically, this  has 
>>> a limited impact for 2D systems, because there should only be images along 
>>> the XY axes. However, if the extent of the cell in the Z direction is too 
>>> short, periodic images get involved as well. *I found that increasing 
>>> the Z cell parameters from 20 Ang to 30 Ang speeds up the calculations by a 
>>> factor 2*, because less images are needed.
>>> 3) Unless you reach a very high number of K-points (~1000), this has 
>>> practically no impact on cost, and only a little on memory. You might as 
>>> well use a denser grid than 3x3x1
>>> 4) I am not sure if your system is a metal. If it is not, you should 
>>> avoid using Broyden mixing and smearing, as SCF convergence slows down.
>>> 5) While you can run this calculation on a single HPC node with 512G, 
>>> this will be extremely slow. I highly recommand using more resources, and 
>>> increasing the KP_NGROUPS keyword the the HF%RI section. This is not a 
>>> small calculation you are attempting.
>>>
>>> I invite you to read the paper written for the method: 
>>> https://doi.org/10.1063/5.0189659, this might help you understand some 
>>> of the points above.
>>> I hope that helps. Best,
>>> Augustin
>>> On Wednesday 18 September 2024 at 16:54:17 UTC+2 yj jiang wrote:
>>>
>>>> Thank you all very much for your help. A few days ago, I thought I had 
>>>> uploaded the output file here, but it seems that due to the large file 
>>>> size, the upload may not have been successful. I apologize for this.
>>>>
>>>> Yesterday, I seemed to have found the reason. The supercomputing job 
>>>> system I am using has a memory allocation limit for each process, and this 
>>>> issue is not exclusive to cp2k jobs. I am now planning to set the maximum 
>>>> memory usage for cp2k to see if it resolves the problem.
>>>>
>>>> Thank you again for your help!
>>>>
>>>> Jiang
>>>>
>>>> 在2024年9月17日星期二 UTC+8 15:33:33<Augustin Bussy> 写道：
>>>>
>>>> Hi,
>>>> everything that Frederick mentioned is very relevant. Additionally, I'd 
>>>> like to add that the method was developed with small unit cells in mind. 
>>>> Your cell is quite large, and this might cause memory issues, although 
>>>> having a 2D material should help. I'll see if I am able to run it.
>>>> Best,
>>>> Augustin
>>>>
>>>> On Monday 16 September 2024 at 08:18:57 UTC+2 Frederick Stein wrote:
>>>>
>>>> Due to my lack of experience, I can just guess from my knowledge on RI 
>>>> methods, the manual and what I was told by the developer. Please also 
>>>> consider the respective manual page: 
>>>> https://manual.cp2k.org/trunk/CP2K_INPUT/FORCE_EVAL/DFT/XC/HF/RI.html#CP2K_INPUT.FORCE_EVAL.DFT.XC.HF.RI 
>>>> .
>>>> - Change RI_METRIC to TRUNCATED and CUTOFF_RADIUS in the &RI section to 
>>>> a value between 1.0 and 2.0 . Lower values improve sparsity, larger values 
>>>> increase accuracy. The shortrange Coulomb metric has a rather large range. 
>>>> If you would like to stick to it, set the OMEGA parameter of the &RI 
>>>> section to a value larger then 0.11 (let's say 1-10).
>>>> - Just for testing, try a single k-point.
>>>> - Try other values of MEMORY_CUT in &RI. Larger values (try 10) should 
>>>> decrease the block sizes. I do not really know how it works but it does 
>>>> have an effect on memory usage.
>>>> - Increase EPS_FILTER in &RI to a larger value (double-check later 
>>>> whether you have to decrease it again).
>>>> If nothing of my suggestions work and the developer of the feature does 
>>>> not chime in, it may be that you do not have enough memory.
>>>> Best,
>>>> Frederick
>>>>
>>>>
>>>> Frederick Stein schrieb am Sonntag, 15. September 2024 um 18:42:31 
>>>> UTC+2:
>>>>
>>>> Hi,
>>>> I am not an expert with RI-HFX but I would try to run potentially 
>>>> cheaper settings first (single k-point, smaller basis sets, larger value of 
>>>> omega, identity metric). It may also help to have an CP2K output file or 
>>>> supplementary input/output file if you submit jobs with a scheduler like 
>>>> Slurm. If there is no function stack in the output file(s) given (either by 
>>>> the compiler or by CP2K), you could also add the keywords TRACE and 
>>>> TRACE_MASTER to your GLOBAL section.
>>>> Best,
>>>> Frederick
>>>>
>>>> yj jiang schrieb am Sonntag, 15. September 2024 um 18:13:42 UTC+2:
>>>>
>>>> Hi, I'm calculating band structure for C3N4 with a Ni atom 
>>>> (attachment). I want to use HES06 + RI-HFX + ADMM. There is a "out of 
>>>> memory" error. How can I address this? Thx. a lot.
>>>>
>>>> I can access a hpc with max memory of 512G.
>>>>
>>>>

-- 
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+unsubscribe at googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/2380524f-ed0b-47ed-b28c-01b6fc7ad529n%40googlegroups.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.cp2k.org/archives/cp2k-user/attachments/20240923/0dfe2984/attachment.htm>