[CP2K-user] [CP2K:19502] Deallocate memory used by Hamiltonian-related subroutines

[Cindy Pham]

Hi Prof.Hutter,

My setup was with gcc-7.3.0 and openmpi-3.1.2. I tried to use gcc-12.3.0
and openmpi-4.1.5 as you suggested but I could not compile using these
versions with my makefile. May I ask if you also used TRACE as the memory
check tool or others?

Best,
Cindy.

On Wed, Nov 1, 2023 at 4:34 AM Jürg Hutter <hutter at chem.uzh.ch> wrote:

> Hi
>
> I run your input for 15000 steps and don't see any memory increase.
> The run was on 9 CPU's using the Trunk version, compiled with
> gcc-13.2.0 and openmpi-4.1.5.
>
> Can you verify your setup? Especially some mpi versions are known to
> have internal memory leaks.
>
> regards
> JH
>
> ________________________________________
> From: cp2k at googlegroups.com <cp2k at googlegroups.com> on behalf of Cindy
> Pham <cindypham196 at gmail.com>
> Sent: Monday, October 30, 2023 6:52 PM
> To: cp2k at googlegroups.com
> Subject: Re: [CP2K:19445] Deallocate memory used by Hamiltonian-related
> subroutines
>
> Hi Prof.Hutter,
>
> The memory used CP2K (as read by TRACE from /proc/self/statm file) keeps
> increasing even if I run a standard OT job (the input file is attached).
> The TRACE helped us to establish that the memory increase happens within
> qs_rho_update_rho_low() subroutine (the increase is shown in the attached
> plot). A closer look into the TRACE printout of subroutines called by
> qs_rho_update_rho_low() produces confusing data. TRACE tells that memory
> allocation can increase not only in complex subroutines, but even in simple
> subroutines such as pw_zero(). How is it possible that pw_zero() increases
> TRACE printout? Is it because allocated memory cannot be measured precisely
> in Unix (via /proc/self/statm)?
>
> If so, it still does not explain the SYSTEMATIC memory increase in more
> complex subroutines called by qs_rho_update_rho_low(). I have noticed two
> arrays, grids_c and npts_local, within the grid_collocate_task_list
> subroutine that are not explicitly deallocated. Do you know if it is by
> design? I know that valgrind tells that the memory is properly deallocated
> (somehow) in the END of the job, but the minor increases DURING a long run
> eventually lead to "out of memory" problem.
>
> Best regards,
> Cindy.
>
> On Wed, Oct 4, 2023 at 8:56 PM Cindy Pham <cindypham196 at gmail.com<mailto:
> cindypham196 at gmail.com>> wrote:
> Hi Prof. Hutter,
>
> Thank you for your suggestion!
>
> Best regards,
> Cindy.
>
>
>
> On Wed, Oct 4, 2023 at 4:15 AM Jürg Hutter <hutter at chem.uzh.ch<mailto:
> hutter at chem.uzh.ch>> wrote:
> Hi
>
> without knowing the details of your program it is impossible to point to
> a easy solution. As there are no memory leaks in CP2K, you must miss some
> routines that clean up at the end of the SCF loop.
> I would suggest you compile the code with memory leak detection in order
> to find the problematic structure and then write a routine to deallocate
> them.
> See the sdbg arch files for the gfortran options needed for leak detection.
>
> regards
> JH
>
> ________________________________________
> From: cp2k at googlegroups.com<mailto:cp2k at googlegroups.com> <
> cp2k at googlegroups.com<mailto:cp2k at googlegroups.com>> on behalf of Cindy
> Pham <cindypham196 at gmail.com<mailto:cindypham196 at gmail.com>>
> Sent: Tuesday, October 3, 2023 10:16 PM
> To: cp2k at googlegroups.com<mailto:cp2k at googlegroups.com>
> Subject: [CP2K:19284] Deallocate memory used by Hamiltonian-related
> subroutines
>
> Hi CP2K forum,
>
> I am running a lengthy SCF calculation (over 10k iterations) and noticed a
> gradual increase in the allocated memory (I used TRACE keyword to print
> current allocated memory). It appears that the step-by-step increase in
> memory allocation happens when the Kohn-Sham Hamiltonian is re-calculated,
> specifically within the qs_ks_did_change function (in qs_ks_types.F).
>
> The SCF routine is my own code that relies on the CP2K built-in
> Hamiltonian subroutines. It functions properly, but its only problem is the
> ever increasing memory consumption.
>
> Since I do not need to keep previous Hamiltonians (for any kind of DIIS
> extrapolation), is there any way to deallocate all memory used by the
> Hamiltonian-related subroutines (at least once in a while, say, after 1000
> SCF iterations)?
>
> Alternatively, are there any input keywords that can ensure that the
> Hamiltonian structures are reset once in a while?
>
> Thank you in advance for your time and your suggestions.
>
> Best regards,
> Cindy Pham.
>
> --
> You received this message because you are subscribed to the Google Groups
> "cp2k" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cp2k+unsubscribe at googlegroups.com<mailto:
> cp2k%2Bunsubscribe at googlegroups.com><mailto:
> cp2k+unsubscribe at googlegroups.com<mailto:
> cp2k%2Bunsubscribe at googlegroups.com>>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/cp2k/CAN4Jpm3f3BUuJ%2B3EDEuqpp0TbBEkaHfOjQm%2Bn9%3D6vYquYQNQvQ%40mail.gmail.com
> <
> https://groups.google.com/d/msgid/cp2k/CAN4Jpm3f3BUuJ%2B3EDEuqpp0TbBEkaHfOjQm%2Bn9%3D6vYquYQNQvQ%40mail.gmail.com?utm_medium=email&utm_source=footer
> >.
>
> --
> You received this message because you are subscribed to the Google Groups
> "cp2k" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cp2k+unsubscribe at googlegroups.com<mailto:
> cp2k%2Bunsubscribe at googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/cp2k/ZR0P278MB07591FF04A66D61DBAA83F3A9FCBA%40ZR0P278MB0759.CHEP278.PROD.OUTLOOK.COM
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "cp2k" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cp2k+unsubscribe at googlegroups.com<mailto:
> cp2k+unsubscribe at googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/cp2k/CAN4Jpm0fQ2eh-B5VbkYRoMcphU9eYDL5f6Jki7FkvY%3Duh9h3GQ%40mail.gmail.com
> <
> https://groups.google.com/d/msgid/cp2k/CAN4Jpm0fQ2eh-B5VbkYRoMcphU9eYDL5f6Jki7FkvY%3Duh9h3GQ%40mail.gmail.com?utm_medium=email&utm_source=footer
> >.
>
> --
> You received this message because you are subscribed to the Google Groups
> "cp2k" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cp2k+unsubscribe at googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/cp2k/ZR0P278MB075927A9473369B2C0BD1F419FA7A%40ZR0P278MB0759.CHEP278.PROD.OUTLOOK.COM
> .
>

-- 
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+unsubscribe at googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/CAN4Jpm0bV%2BsPKUX6-hwEFhk4pfyTrVAXka27tBGvO4zcYq8%2Bow%40mail.gmail.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.cp2k.org/archives/cp2k-user/attachments/20231114/803394d8/attachment-0001.htm>