<div dir="ltr"><div>Thanks for that Tiziano,</div><div>I'll give it a go later today.</div><div>kind regards,</div><div>Chris On Friday, 31 May 2019 09:24:51 UTC+1, Tiziano Müller wrote:</div><blockquote class="gmail_quote" style="margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;">Hi Chris, an arch/ file for CP2K with P100 GPUs can be found as part of the regtester output from Piz Daint here: <a onmousedown="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.cp2k.org%2Fstatic%2Fregtest%2Ftrunk%2Fcscs-daint-xc50_gpu%2FCRAY_XC50-gfortran_gpu.psmp.out\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEV3E1CGYSCg7z-ZdV_-UzUvyDJKg';return true;" onclick="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.cp2k.org%2Fstatic%2Fregtest%2Ftrunk%2Fcscs-daint-xc50_gpu%2FCRAY_XC50-gfortran_gpu.psmp.out\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEV3E1CGYSCg7z-ZdV_-UzUvyDJKg';return true;" href="https://www.cp2k.org/static/regtest/trunk/cscs-daint-xc50_gpu/CRAY_XC50-gfortran_gpu.psmp.out" target="_blank" rel="nofollow">https://www.cp2k.org/static/regtest/trunk/cscs-daint-xc50_gpu/CRAY_XC50-gfortran_gpu.psmp.out</a> Those outputs are usually available from here: <a onmousedown="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fdashboard.cp2k.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHgqILIfQ8_hWhJX0qYPMTOuEJwCQ';return true;" onclick="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fdashboard.cp2k.org%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHgqILIfQ8_hWhJX0qYPMTOuEJwCQ';return true;" href="https://dashboard.cp2k.org/" target="_blank" rel="nofollow">https://dashboard.cp2k.org/</a> (click the link in the Status column) Best regards, Tiziano Am 30.05.19 um 10:15 schrieb CNelson: > Hi Both, > would it be possible to get a copy of the ARCH file you used to build > CP2K with the new V100 GPUs? > cheers, > Chris. > > On Sunday, 4 November 2018 21:06:12 UTC, Alfio Lazzaro wrote: > > OK, the best way is if you can attach the arch file, the input file, > and the output that you got from CP2K. > The only GPU accelerated part in CP2K is DBCSR, but can be that you > are bound from something else. > > I agree with you that the reoptimization is not that important at > this stage... > > Alfio > > > Il giorno domenica 4 novembre 2018 19:13:02 UTC+1, <a>fo...@gmail.com</a> > ha scritto: > > Thanks Alfio for the response. > > Yes. 8 V100 GPUs is extreme. The test I had used takes around > 500 seconds on a system with Intel SKL G-6148 40 cores(20 > cores/socket). Do you think this test is not large enough to run > on GPUs? If yes, can you recommend any test from CP2K tests folder? > > I had tried runs with 1 & 2 V100 gpus also. The performance was > slower than the 8 V100 gpus run. > > CP2K was able to recognize all the 8 gpus, as per "DBCSR| ACC: > Number of devices/node". > > I had tried reoptimizing the kernels for V100. But could not > determine what block size values have to be passed to tune.py > script. > > As CP2K-6.1 already has optimized kernel parameters for P100, > even 2xP100 GPUs run was slower than CPU only benchmark. > > On Sunday, November 4, 2018 at 2:33:11 PM UTC+5:30, Alfio > Lazzaro wrote: > > You may take a look at this issue on > github: <a onmousedown="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fcp2k%2Fcp2k%2Fissues%2F73\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHay9GWtMchhngkD6hFPONyIcg7Ww';return true;" onclick="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fcp2k%2Fcp2k%2Fissues%2F73\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHay9GWtMchhngkD6hFPONyIcg7Ww';return true;" href="https://github.com/cp2k/cp2k/issues/73" target="_blank" rel="nofollow">https://github.com/cp2k/cp2k/issues/73</a> > <<a onmousedown="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fcp2k%2Fcp2k%2Fissues%2F73\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHay9GWtMchhngkD6hFPONyIcg7Ww';return true;" onclick="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fcp2k%2Fcp2k%2Fissues%2F73\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHay9GWtMchhngkD6hFPONyIcg7Ww';return true;" href="https://github.com/cp2k/cp2k/issues/73" target="_blank" rel="nofollow">https://github.com/cp2k/cp2k/issues/73</a>> > > In your particular case, your setup of 8 V100 is pretty > extreme and it would require a large computation. Which test > are you using for benchmarking? > > Then, your setup of 8 ranks + 5 threads should be OK. CP2K > attaches ranks to GPU in a round-robin manner, therefore in > your case there is a rank talking to each GPU. > We don't have a large experience of multi-gpu nodes, hence I > would suggest to do some scalability test by running 1 rank, > 2 ranks, ... 8 ranks (always 5 threads) to check how the > performance scales. BTW, make sure CP2K is able to recognize > 8 GPUs by checking the following output at the beginning: > > DBCSR| ACC: Number of devices/node > 1 > > Eventually, you might consider reoptimizing the kernels for > the V100, but this is not a priority... > > Alfio > > > > Il giorno sabato 3 novembre 2018 07:55:09 UTC+1, > <a>fo...@gmail.com</a> ha scritto: > > HI, > > How is the CP2K performance on GPUs in general? > > I'm getting very low performance on GPUs(Nvidia V100 > SXM2). It is a single node benchmark with 8 GPUs and > Intel Skylake Gold 6148 dual processors. > > The CP2K time on 8 GPUs (CP2K-6.1 psmp version, > ifort-2017, CUDA-9.2, 8mpi ranks + 5 threads per rank) > is still slower than CP2K time of CPU only benchmark. > > For CPU runs, the CP2K-6.1 is built with LIBXSMM-1.8.3. > > For GPU runs, have tried both with and without LIBXSMM. > There is no performance difference. But both's > performance is still slower than CPU only benchmark even > after using all the 8 GPUs & all 40 cores of CPU. Can > some one please share their experience on CP2K > performance with GPUs. > > The CUDA specific DFLAGS used are: -D__ACC -D__DBCSR_ACC > -D__PW_CUDA. > > -- > You received this message because you are subscribed to the Google > Groups "cp2k" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to <a onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;" href="javascript:" target="_blank" rel="nofollow" gdf-obfuscated-mailto="0iRqnhQgCQAJ">cp...@googlegroups.com</a> > <mailto:<a onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;" href="javascript:" target="_blank" rel="nofollow" gdf-obfuscated-mailto="0iRqnhQgCQAJ">cp...@googlegroups.com</a>>. > To post to this group, send email to <a onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;" href="javascript:" target="_blank" rel="nofollow" gdf-obfuscated-mailto="0iRqnhQgCQAJ">c...@googlegroups.com</a> > <mailto:<a onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;" href="javascript:" target="_blank" rel="nofollow" gdf-obfuscated-mailto="0iRqnhQgCQAJ">c...@googlegroups.com</a>>. > Visit this group at <a onmousedown="this.href='https://groups.google.com/group/cp2k';return true;" onclick="this.href='https://groups.google.com/group/cp2k';return true;" href="https://groups.google.com/group/cp2k" target="_blank" rel="nofollow">https://groups.google.com/group/cp2k</a>. > To view this discussion on the web visit > <a onmousedown="this.href='https://groups.google.com/d/msgid/cp2k/4920c538-3d63-4754-8dc3-76396262d543%40googlegroups.com';return true;" onclick="this.href='https://groups.google.com/d/msgid/cp2k/4920c538-3d63-4754-8dc3-76396262d543%40googlegroups.com';return true;" href="https://groups.google.com/d/msgid/cp2k/4920c538-3d63-4754-8dc3-76396262d543%40googlegroups.com" target="_blank" rel="nofollow">https://groups.google.com/d/msgid/cp2k/4920c538-3d63-4754-8dc3-76396262d543%40googlegroups.com</a> > <<a onmousedown="this.href='https://groups.google.com/d/msgid/cp2k/4920c538-3d63-4754-8dc3-76396262d543%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter';return true;" onclick="this.href='https://groups.google.com/d/msgid/cp2k/4920c538-3d63-4754-8dc3-76396262d543%40googlegroups.com?utm_medium\x3demail\x26utm_source\x3dfooter';return true;" href="https://groups.google.com/d/msgid/cp2k/4920c538-3d63-4754-8dc3-76396262d543%40googlegroups.com?utm_medium=email&utm_source=footer" target="_blank" rel="nofollow">https://groups.google.com/d/msgid/cp2k/4920c538-3d63-4754-8dc3-76396262d543%40googlegroups.com?utm_medium=email&utm_source=footer</a>>. > For more options, visit <a onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;" href="https://groups.google.com/d/optout" target="_blank" rel="nofollow">https://groups.google.com/d/optout</a>. -- Tiziano Müller University of Zurich Department of Chemistry Winterthurerstrasse 190 CH-8057 Zürich Tel: +41 44 63 54234 <a onmousedown="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.chem.uzh.ch\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEaSd80M-dQwlA8q5GEKkKY5wRCiQ';return true;" onclick="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.chem.uzh.ch\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEaSd80M-dQwlA8q5GEKkKY5wRCiQ';return true;" href="http://www.chem.uzh.ch" target="_blank" rel="nofollow">www.chem.uzh.ch</a> <a onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;" href="javascript:" target="_blank" rel="nofollow" gdf-obfuscated-mailto="0iRqnhQgCQAJ">tiz...@chem.uzh.ch</a> </blockquote></div>