[CP2K-user] [CP2K:19509] Deallocate memory used by Hamiltonian-related subroutines

[Jürg Hutter]

Yes, I was using TRACE to get detailed information and the Linux "top"
command for an external measure.

regards
JH

________________________________________
From: cp2k at googlegroups.com <cp2k at googlegroups.com> on behalf of Cindy Pham <cindypham196 at gmail.com>
Sent: Wednesday, November 15, 2023 3:07 AM
To: cp2k at googlegroups.com
Subject: Re: [CP2K:19502] Deallocate memory used by Hamiltonian-related subroutines

Hi Prof.Hutter,

My setup was with gcc-7.3.0 and openmpi-3.1.2. I tried to use gcc-12.3.0 and openmpi-4.1.5 as you suggested but I could not compile using these versions with my makefile. May I ask if you also used TRACE as the memory check tool or others?

Best,
Cindy.

On Wed, Nov 1, 2023 at 4:34 AM Jürg Hutter <hutter at chem.uzh.ch<mailto:hutter at chem.uzh.ch>> wrote:
Hi

I run your input for 15000 steps and don't see any memory increase.
The run was on 9 CPU's using the Trunk version, compiled with
gcc-13.2.0 and openmpi-4.1.5.

Can you verify your setup? Especially some mpi versions are known to
have internal memory leaks.

regards
JH

________________________________________
From: cp2k at googlegroups.com<mailto:cp2k at googlegroups.com> <cp2k at googlegroups.com<mailto:cp2k at googlegroups.com>> on behalf of Cindy Pham <cindypham196 at gmail.com<mailto:cindypham196 at gmail.com>>
Sent: Monday, October 30, 2023 6:52 PM
To: cp2k at googlegroups.com<mailto:cp2k at googlegroups.com>
Subject: Re: [CP2K:19445] Deallocate memory used by Hamiltonian-related subroutines

Hi Prof.Hutter,

The memory used CP2K (as read by TRACE from /proc/self/statm file) keeps increasing even if I run a standard OT job (the input file is attached). The TRACE helped us to establish that the memory increase happens within qs_rho_update_rho_low() subroutine (the increase is shown in the attached plot). A closer look into the TRACE printout of subroutines called by qs_rho_update_rho_low() produces confusing data. TRACE tells that memory allocation can increase not only in complex subroutines, but even in simple subroutines such as pw_zero(). How is it possible that pw_zero() increases TRACE printout? Is it because allocated memory cannot be measured precisely in Unix (via /proc/self/statm)?

If so, it still does not explain the SYSTEMATIC memory increase in more complex subroutines called by qs_rho_update_rho_low(). I have noticed two arrays, grids_c and npts_local, within the grid_collocate_task_list subroutine that are not explicitly deallocated. Do you know if it is by design? I know that valgrind tells that the memory is properly deallocated (somehow) in the END of the job, but the minor increases DURING a long run eventually lead to "out of memory" problem.

Best regards,
Cindy.

On Wed, Oct 4, 2023 at 8:56 PM Cindy Pham <cindypham196 at gmail.com<mailto:cindypham196 at gmail.com><mailto:cindypham196 at gmail.com<mailto:cindypham196 at gmail.com>>> wrote:
Hi Prof. Hutter,

Thank you for your suggestion!

Best regards,
Cindy.



On Wed, Oct 4, 2023 at 4:15 AM Jürg Hutter <hutter at chem.uzh.ch<mailto:hutter at chem.uzh.ch><mailto:hutter at chem.uzh.ch<mailto:hutter at chem.uzh.ch>>> wrote:
Hi

without knowing the details of your program it is impossible to point to
a easy solution. As there are no memory leaks in CP2K, you must miss some
routines that clean up at the end of the SCF loop.
I would suggest you compile the code with memory leak detection in order
to find the problematic structure and then write a routine to deallocate them.
See the sdbg arch files for the gfortran options needed for leak detection.

regards
JH

________________________________________
From: cp2k at googlegroups.com<mailto:cp2k at googlegroups.com><mailto:cp2k at googlegroups.com<mailto:cp2k at googlegroups.com>> <cp2k at googlegroups.com<mailto:cp2k at googlegroups.com><mailto:cp2k at googlegroups.com<mailto:cp2k at googlegroups.com>>> on behalf of Cindy Pham <cindypham196 at gmail.com<mailto:cindypham196 at gmail.com><mailto:cindypham196 at gmail.com<mailto:cindypham196 at gmail.com>>>
Sent: Tuesday, October 3, 2023 10:16 PM
To: cp2k at googlegroups.com<mailto:cp2k at googlegroups.com><mailto:cp2k at googlegroups.com<mailto:cp2k at googlegroups.com>>
Subject: [CP2K:19284] Deallocate memory used by Hamiltonian-related subroutines

Hi CP2K forum,

I am running a lengthy SCF calculation (over 10k iterations) and noticed a gradual increase in the allocated memory (I used TRACE keyword to print current allocated memory). It appears that the step-by-step increase in memory allocation happens when the Kohn-Sham Hamiltonian is re-calculated, specifically within the qs_ks_did_change function (in qs_ks_types.F).

The SCF routine is my own code that relies on the CP2K built-in Hamiltonian subroutines. It functions properly, but its only problem is the ever increasing memory consumption.

Since I do not need to keep previous Hamiltonians (for any kind of DIIS extrapolation), is there any way to deallocate all memory used by the Hamiltonian-related subroutines (at least once in a while, say, after 1000 SCF iterations)?

Alternatively, are there any input keywords that can ensure that the Hamiltonian structures are reset once in a while?

Thank you in advance for your time and your suggestions.

Best regards,
Cindy Pham.

--
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+unsubscribe at googlegroups.com<mailto:cp2k%2Bunsubscribe at googlegroups.com><mailto:cp2k%2Bunsubscribe at googlegroups.com<mailto:cp2k%252Bunsubscribe at googlegroups.com>><mailto:cp2k+unsubscribe at googlegroups.com<mailto:cp2k%2Bunsubscribe at googlegroups.com><mailto:cp2k%2Bunsubscribe at googlegroups.com<mailto:cp2k%252Bunsubscribe at googlegroups.com>>>.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/CAN4Jpm3f3BUuJ%2B3EDEuqpp0TbBEkaHfOjQm%2Bn9%3D6vYquYQNQvQ%40mail.gmail.com<https://groups.google.com/d/msgid/cp2k/CAN4Jpm3f3BUuJ%2B3EDEuqpp0TbBEkaHfOjQm%2Bn9%3D6vYquYQNQvQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.

--
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+unsubscribe at googlegroups.com<mailto:cp2k%2Bunsubscribe at googlegroups.com><mailto:cp2k%2Bunsubscribe at googlegroups.com<mailto:cp2k%252Bunsubscribe at googlegroups.com>>.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/ZR0P278MB07591FF04A66D61DBAA83F3A9FCBA%40ZR0P278MB0759.CHEP278.PROD.OUTLOOK.COM.

--
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+unsubscribe at googlegroups.com<mailto:cp2k%2Bunsubscribe at googlegroups.com><mailto:cp2k+unsubscribe at googlegroups.com<mailto:cp2k%2Bunsubscribe at googlegroups.com>>.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/CAN4Jpm0fQ2eh-B5VbkYRoMcphU9eYDL5f6Jki7FkvY%3Duh9h3GQ%40mail.gmail.com<https://groups.google.com/d/msgid/cp2k/CAN4Jpm0fQ2eh-B5VbkYRoMcphU9eYDL5f6Jki7FkvY%3Duh9h3GQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.

--
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+unsubscribe at googlegroups.com<mailto:cp2k%2Bunsubscribe at googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/ZR0P278MB075927A9473369B2C0BD1F419FA7A%40ZR0P278MB0759.CHEP278.PROD.OUTLOOK.COM.

--
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+unsubscribe at googlegroups.com<mailto:cp2k+unsubscribe at googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/CAN4Jpm0bV%2BsPKUX6-hwEFhk4pfyTrVAXka27tBGvO4zcYq8%2Bow%40mail.gmail.com<https://groups.google.com/d/msgid/cp2k/CAN4Jpm0bV%2BsPKUX6-hwEFhk4pfyTrVAXka27tBGvO4zcYq8%2Bow%40mail.gmail.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+unsubscribe at googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/ZR0P278MB0759397C227B3C8312CD03299FB1A%40ZR0P278MB0759.CHEP278.PROD.OUTLOOK.COM.