[CP2K-user] [CP2K:19509] Deallocate memory used by Hamiltonian-related subroutines
Jürg Hutter
hutter at chem.uzh.ch
Wed Nov 15 16:07:55 UTC 2023
Yes, I was using TRACE to get detailed information and the Linux "top"
command for an external measure.
regards
JH
________________________________________
From: cp2k at googlegroups.com <cp2k at googlegroups.com> on behalf of Cindy Pham <cindypham196 at gmail.com>
Sent: Wednesday, November 15, 2023 3:07 AM
To: cp2k at googlegroups.com
Subject: Re: [CP2K:19502] Deallocate memory used by Hamiltonian-related subroutines
Hi Prof.Hutter,
My setup was with gcc-7.3.0 and openmpi-3.1.2. I tried to use gcc-12.3.0 and openmpi-4.1.5 as you suggested but I could not compile using these versions with my makefile. May I ask if you also used TRACE as the memory check tool or others?
Best,
Cindy.
On Wed, Nov 1, 2023 at 4:34 AM Jürg Hutter <hutter at chem.uzh.ch<mailto:hutter at chem.uzh.ch>> wrote:
Hi
I run your input for 15000 steps and don't see any memory increase.
The run was on 9 CPU's using the Trunk version, compiled with
gcc-13.2.0 and openmpi-4.1.5.
Can you verify your setup? Especially some mpi versions are known to
have internal memory leaks.
regards
JH
________________________________________
From: cp2k at googlegroups.com<mailto:cp2k at googlegroups.com> <cp2k at googlegroups.com<mailto:cp2k at googlegroups.com>> on behalf of Cindy Pham <cindypham196 at gmail.com<mailto:cindypham196 at gmail.com>>
Sent: Monday, October 30, 2023 6:52 PM
To: cp2k at googlegroups.com<mailto:cp2k at googlegroups.com>
Subject: Re: [CP2K:19445] Deallocate memory used by Hamiltonian-related subroutines
Hi Prof.Hutter,
The memory used CP2K (as read by TRACE from /proc/self/statm file) keeps increasing even if I run a standard OT job (the input file is attached). The TRACE helped us to establish that the memory increase happens within qs_rho_update_rho_low() subroutine (the increase is shown in the attached plot). A closer look into the TRACE printout of subroutines called by qs_rho_update_rho_low() produces confusing data. TRACE tells that memory allocation can increase not only in complex subroutines, but even in simple subroutines such as pw_zero(). How is it possible that pw_zero() increases TRACE printout? Is it because allocated memory cannot be measured precisely in Unix (via /proc/self/statm)?
If so, it still does not explain the SYSTEMATIC memory increase in more complex subroutines called by qs_rho_update_rho_low(). I have noticed two arrays, grids_c and npts_local, within the grid_collocate_task_list subroutine that are not explicitly deallocated. Do you know if it is by design? I know that valgrind tells that the memory is properly deallocated (somehow) in the END of the job, but the minor increases DURING a long run eventually lead to "out of memory" problem.
Best regards,
Cindy.
On Wed, Oct 4, 2023 at 8:56 PM Cindy Pham <cindypham196 at gmail.com<mailto:cindypham196 at gmail.com><mailto:cindypham196 at gmail.com<mailto:cindypham196 at gmail.com>>> wrote:
Hi Prof. Hutter,
Thank you for your suggestion!
Best regards,
Cindy.
On Wed, Oct 4, 2023 at 4:15 AM Jürg Hutter <hutter at chem.uzh.ch<mailto:hutter at chem.uzh.ch><mailto:hutter at chem.uzh.ch<mailto:hutter at chem.uzh.ch>>> wrote:
Hi
without knowing the details of your program it is impossible to point to
a easy solution. As there are no memory leaks in CP2K, you must miss some
routines that clean up at the end of the SCF loop.
I would suggest you compile the code with memory leak detection in order
to find the problematic structure and then write a routine to deallocate them.
See the sdbg arch files for the gfortran options needed for leak detection.
regards
JH
________________________________________
From: cp2k at googlegroups.com<mailto:cp2k at googlegroups.com><mailto:cp2k at googlegroups.com<mailto:cp2k at googlegroups.com>> <cp2k at googlegroups.com<mailto:cp2k at googlegroups.com><mailto:cp2k at googlegroups.com<mailto:cp2k at googlegroups.com>>> on behalf of Cindy Pham <cindypham196 at gmail.com<mailto:cindypham196 at gmail.com><mailto:cindypham196 at gmail.com<mailto:cindypham196 at gmail.com>>>
Sent: Tuesday, October 3, 2023 10:16 PM
To: cp2k at googlegroups.com<mailto:cp2k at googlegroups.com><mailto:cp2k at googlegroups.com<mailto:cp2k at googlegroups.com>>
Subject: [CP2K:19284] Deallocate memory used by Hamiltonian-related subroutines
Hi CP2K forum,
I am running a lengthy SCF calculation (over 10k iterations) and noticed a gradual increase in the allocated memory (I used TRACE keyword to print current allocated memory). It appears that the step-by-step increase in memory allocation happens when the Kohn-Sham Hamiltonian is re-calculated, specifically within the qs_ks_did_change function (in qs_ks_types.F).
The SCF routine is my own code that relies on the CP2K built-in Hamiltonian subroutines. It functions properly, but its only problem is the ever increasing memory consumption.
Since I do not need to keep previous Hamiltonians (for any kind of DIIS extrapolation), is there any way to deallocate all memory used by the Hamiltonian-related subroutines (at least once in a while, say, after 1000 SCF iterations)?
Alternatively, are there any input keywords that can ensure that the Hamiltonian structures are reset once in a while?
Thank you in advance for your time and your suggestions.
Best regards,
Cindy Pham.
--
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+unsubscribe at googlegroups.com<mailto:cp2k%2Bunsubscribe at googlegroups.com><mailto:cp2k%2Bunsubscribe at googlegroups.com<mailto:cp2k%252Bunsubscribe at googlegroups.com>><mailto:cp2k+unsubscribe at googlegroups.com<mailto:cp2k%2Bunsubscribe at googlegroups.com><mailto:cp2k%2Bunsubscribe at googlegroups.com<mailto:cp2k%252Bunsubscribe at googlegroups.com>>>.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/CAN4Jpm3f3BUuJ%2B3EDEuqpp0TbBEkaHfOjQm%2Bn9%3D6vYquYQNQvQ%40mail.gmail.com<https://groups.google.com/d/msgid/cp2k/CAN4Jpm3f3BUuJ%2B3EDEuqpp0TbBEkaHfOjQm%2Bn9%3D6vYquYQNQvQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+unsubscribe at googlegroups.com<mailto:cp2k%2Bunsubscribe at googlegroups.com><mailto:cp2k%2Bunsubscribe at googlegroups.com<mailto:cp2k%252Bunsubscribe at googlegroups.com>>.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/ZR0P278MB07591FF04A66D61DBAA83F3A9FCBA%40ZR0P278MB0759.CHEP278.PROD.OUTLOOK.COM.
--
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+unsubscribe at googlegroups.com<mailto:cp2k%2Bunsubscribe at googlegroups.com><mailto:cp2k+unsubscribe at googlegroups.com<mailto:cp2k%2Bunsubscribe at googlegroups.com>>.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/CAN4Jpm0fQ2eh-B5VbkYRoMcphU9eYDL5f6Jki7FkvY%3Duh9h3GQ%40mail.gmail.com<https://groups.google.com/d/msgid/cp2k/CAN4Jpm0fQ2eh-B5VbkYRoMcphU9eYDL5f6Jki7FkvY%3Duh9h3GQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+unsubscribe at googlegroups.com<mailto:cp2k%2Bunsubscribe at googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/ZR0P278MB075927A9473369B2C0BD1F419FA7A%40ZR0P278MB0759.CHEP278.PROD.OUTLOOK.COM.
--
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+unsubscribe at googlegroups.com<mailto:cp2k+unsubscribe at googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/CAN4Jpm0bV%2BsPKUX6-hwEFhk4pfyTrVAXka27tBGvO4zcYq8%2Bow%40mail.gmail.com<https://groups.google.com/d/msgid/cp2k/CAN4Jpm0bV%2BsPKUX6-hwEFhk4pfyTrVAXka27tBGvO4zcYq8%2Bow%40mail.gmail.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+unsubscribe at googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/ZR0P278MB0759397C227B3C8312CD03299FB1A%40ZR0P278MB0759.CHEP278.PROD.OUTLOOK.COM.
More information about the CP2K-user
mailing list