[CP2K-user] CP2K 7.1-Cuda Bandgap and HF energies different from previous versions

[Fabian Ducry]

Dear Andres,

I can confirm and reproduce the issue. Apparently it appears when combining 
CUDA + OMP in hybrid calculations. In that case the energy becomes a 
function of #OMP threads per rank. For your input I got (cp2k 8.0, revision 
3e7b916, run on Piz Daint)
                                                                no cuda    
                  OMP_NUM_THREADS = 1          OMP_NUM_THREADS = 3          
OMP_NUM_THREADS = 6
  Exchange-correlation energy:          -433.84964308969535               
-433.84964308969302                -435.33426106395467                  
-435.96513615032325
  Hartree-Fock Exchange energy:      -127.87395928499694                
-127.87395928499325                -125.97109874333140                  
-125.24809389970088
  Total energy:                                -1976.39722899739672         
      -1976.39722899739013              
-1975.95046919253809                 -1975.87080541858177

Without OMP parallelization the energies agrees with the calculation 
without CUDA accelleration. Increasing OMP_NUM_THREADS beyond 1 increases 
the Hartree-Fock Exchange energy.
Apparently you have to disable OMP to obtain correct results. This is 
obviously not very satisfying and I hope this gets fixed. I see that you 
used 1 MPI/12 OMP ranks per node. Try increasing the number of MPI ranks 
per node. To do so you have to set 
export CRAY_CUDA_MPS=1 in the submission script.

I hope this helps.

Best,
Fabian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.cp2k.org/archives/cp2k-user/attachments/20200504/e6588d2e/attachment.htm>