<div dir="ltr">Hi Sean,<div><br></div><div>are you sure you have dedicated access to the resource you run on? Those large fluctuations look likely to be caused by other jobs running on the node(s) you are using (though the general linearish trend is surprising).</div><div><br></div><div>Matt <br><br>On Friday, October 11, 2019 at 1:41:03 PM UTC+1, Sean Fischer wrote:<blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;"><div dir="ltr">I'm attempting to run MD using QS on a 64 molecule water box. So far I have been unable to achieve consistent performance across the MD trajectory. After a while the time per MD step dramatically increases, but the number of SCF iterations per step stays basically constant, as you can see in the included image.<p style="text-align:center;clear:both"><a style="margin-left:1em;margin-right:1em"><img src="https://groups.google.com/group/cp2k/attach/cc04bffa8a9b6/md_timing.png?part=0.2&view=1&authuser=0" alt="md_timing.png" width="320" height="240" border="0"></a></p><p style="text-align:left;clear:both">To further illustrate this, the following is from one of the first steps in the trajectory</p><p style="text-align:left;clear:both"><br></p><p><span><span> </span>------------------------------<wbr>----- OT ------------------------------<wbr>---------</span></p><p><span><span> </span>Minimizer<span> </span>: DIIS<span> </span>: direct inversion</span></p><p><span><span> </span>in the iterative subspace</span></p><p><span><span> </span>using <span> </span>7 DIIS vectors</span></p><p><span><span> </span>safer DIIS on</span></p><p><span><span> </span>Preconditioner : FULL_ALL<span> </span>: diagonalization, state selective</span></p><p><span><span> </span>Precond_solver : DEFAULT</span></p><p><span><span> </span>stepsize <span> </span>:<span> </span>0.15000000<span> </span>energy_gap <span> </span>:<span> </span>0.08000000</span></p><p><span><span> </span>eps_taylor <span> </span>: <span> </span>0.10000E-15<span> </span>max_taylor <span> </span>: <span> </span>4</span></p><p><span><span> </span>------------------------------<wbr>----- OT ------------------------------<wbr>---------</span></p><p><span></span><br></p><p><span><span> </span>Step <span> </span>Update method<span> </span>Time<span> </span>Convergence <span> </span>Total energy<span> </span>Change</span></p><p><span><span> </span>------------------------------<wbr>------------------------------<wbr>------------------</span></p><p><span><span> </span>1 OT DIIS <span> </span>0.15E+00<span> </span>2.3 <span> </span>0.00030148 <span> </span>-1107.9047914512 -1.11E+03</span></p><p><span><span> </span>2 OT DIIS <span> </span>0.15E+00<span> </span>1.6 <span> </span>0.00012826 <span> </span>-1107.9089080136 -4.12E-03</span></p><p><span><span> </span>3 OT DIIS <span> </span>0.15E+00<span> </span>1.6 <span> </span>0.00007994 <span> </span>-1107.9095257164 -6.18E-04</span></p><p><span><span> </span>4 OT DIIS <span> </span>0.15E+00<span> </span>1.6 <span> </span>0.00000744 <span> </span>-1107.9096948950 -1.69E-04</span></p><p><span><span> </span>5 OT DIIS <span> </span>0.15E+00<span> </span>2.0 <span> </span>0.00000377 <span> </span>-1107.9096964458 -1.55E-06</span></p><p><span><span> </span>6 OT DIIS <span> </span>0.15E+00<span> </span>1.6 <span> </span>0.00000067 <span> </span>-1107.9096967653 -3.19E-07</span></p><p><span></span><br></p><p><span><span> </span>*** SCF run converged in <span> </span>6 steps ***</span></p><p><span></span><br></p><p><span></span><br></p><p><span><span> </span>Electronic density on regular grids: <span> </span>-511.9999993695<span> </span>0.0000006305</span></p><p><span><span> </span>Core density on regular grids:<span> </span>511.9999999846 <span> </span>-0.0000000154</span></p><p><span><span> </span>Total charge density on r-space grids:<span> </span>0.0000006151</span></p><p style="text-align:left;clear:both">
</p><p><span><span> </span>Total charge density g-space grids: <span> </span>0.0000006151</span></p><p style="text-align:left;clear:both"><br></p><p style="text-align:left;clear:both"><br></p><p style="text-align:left;clear:both">While after ~2500 steps, the same section now looks like</p><p style="text-align:left;clear:both"><br></p><p><span><span> </span>------------------------------<wbr>----- OT ------------------------------<wbr>---------</span></p><p><span><span> </span>Minimizer<span> </span>: DIIS<span> </span>: direct inversion</span></p><p><span><span> </span>in the iterative subspace</span></p><p><span><span> </span>using <span> </span>7 DIIS vectors</span></p><p><span><span> </span>safer DIIS on</span></p><p><span><span> </span>Preconditioner : FULL_ALL<span> </span>: diagonalization, state selective</span></p><p><span><span> </span>Precond_solver : DEFAULT</span></p><p><span><span> </span>stepsize <span> </span>:<span> </span>0.15000000<span> </span>energy_gap <span> </span>:<span> </span>0.08000000</span></p><p><span><span> </span>eps_taylor <span> </span>: <span> </span>0.10000E-15<span> </span>max_taylor <span> </span>: <span> </span>4</span></p><p><span><span> </span>------------------------------<wbr>----- OT ------------------------------<wbr>---------</span></p><p><span></span><br></p><p><span><span> </span>Step <span> </span>Update method<span> </span>Time<span> </span>Convergence <span> </span>Total energy<span> </span>Change</span></p><p><span><span> </span>------------------------------<wbr>------------------------------<wbr>------------------</span></p><p><span><span> </span>1 OT DIIS <span> </span>0.15E+00<span> </span>6.3 <span> </span>0.00003184 <span> </span>-1108.0316770158 -1.11E+03</span></p><p><span><span> </span>2 OT DIIS <span> </span>0.15E+00 <span> </span>17.0 <span> </span>0.00001599 <span> </span>-1108.0317180830 -4.11E-05</span></p><p><span><span> </span>3 OT DIIS <span> </span>0.15E+00 <span> </span>18.4 <span> </span>0.00000956 <span> </span>-1108.0317247395 -6.66E-06</span></p><p><span><span> </span>4 OT DIIS <span> </span>0.15E+00 <span> </span>21.9 <span> </span>0.00000119 <span> </span>-1108.0317273970 -2.66E-06</span></p><p><span><span> </span>5 OT DIIS <span> </span>0.15E+00<span> </span>6.8 <span> </span>0.00000063 <span> </span>-1108.0317274283 -3.14E-08</span></p><p><span></span><br></p><p><span><span> </span>*** SCF run converged in <span> </span>5 steps ***</span></p><p><span></span><br></p><p><span></span><br></p><p><span><span> </span>Electronic density on regular grids: <span> </span>-511.9999994472<span> </span>0.0000005528</span></p><p><span><span> </span>Core density on regular grids:<span> </span>511.9999999845 <span> </span>-0.0000000155</span></p><p><span><span> </span>Total charge density on r-space grids:<span> </span>0.0000005373</span></p><p style="text-align:left;clear:both">
</p><p><span><span> </span>Total charge density g-space grids: <span> </span>0.0000005373</span></p><p style="text-align:left;clear:both"><br></p><p style="text-align:left;clear:both"><br></p><p style="text-align:left;clear:both">This has happened with different functionals, pseudopotentials, and SCF parameters (PS vs ASPC extrapolation, different orders of extrapolation, OT DIIS vs DIIS/Diag., different preconditioners). I recompiled the whole code with the toolchain install script generating all dependencies and got the same result. There is no indication from the dynamics of the system itself as to what the problem might be as the potential and kinetic energies are well behaved, and the constant of motion for the dynamics is reasonably conserved. I've watched the resulting trajectory itself, and there is nothing unusually happening (no bonding breaking, etc.) The input file I used for run in the plot above is attached. The starting geometry for the water molecules was taken from the 64 molecule benchmark in the tests/QS directory.</p><p style="text-align:left;clear:both"><br></p><p style="text-align:left;clear:both">If anyone has any suggestions on possible causes or where I can look for more information, I would be very appreciative.</p></div></blockquote></div></div>