[CP2K:3880] Re: cp2k on IBM Blue Gene

[Iain Bethune]

Hi Wei,

We did not see this error on BG/P.  I have not yet tried compiling the code for BG/Q however.  An internal compiler error should be reported to the system support people, and it might be addressed in a future version of the compiler.  Most likely it is related to hitting some internal limit in the compiler.  We have seen in the past with the IBM XL compilers problems with large modules, long 'use' clauses, and large subroutine argument lists.

You could also try compiling this file at a lower optimisation level (e.g. -O0), as there should not be anything performance-critical in this module.  Good luck!

Cheers

- Iain

--

Iain Bethune
Applications Consultant, EPCC

Email: ibet... at epcc.ed.ac.uk
Twitter: @IainBethune
Tel/Fax: +44 (0)131 650 5201/6555
Mob: +44 (0)7598317015
Addr: 2404 JCMB, The King's Buildings, Mayfield Road, Edinburgh, EH9 3JZ






On 27 Jun 2012, at 18:54, Wei wrote:

> Dear Iain, 
> 
> today I have tried to compile cp2k on Blue Gene/Q with the arch file you provided in the paper, I got some internal compiler error after some time, 
> 
> ** topology_psf   === End of Compilation 1 ===
> 1501-510  Compilation successful for file topology_psf.f90.
> ** fist_environment_types   === End of Compilation 1 ===
> 1501-510  Compilation successful for file fist_environment_types.f90.
> ** qs_environment_types   === End of Compilation 1 ===
> 1585-100  Internal compiler error -- please contact your service representative. For more information visit: http://www.ibm.com/support/docview.wss?uid=swg21110810
> 1501-511  Compilation failed for file qs_environment_types.f90.
> make[1]: *** [qs_environment_types.o] Error 255
> 
> Do you maybe have tried it already? or have you seen this error previously on BG/P?
> 
> I download the most recent version from the trunk, and my acrh file is 
> 
> CC = xlc
> CPP = cpp -traditional -E
> FC = mpixlf95_r -qsuffix=f=f90
> LD = mpixlf95_r
> AR = ar -r
> DFLAGS = -D__HAS_NO_ISO_C_BINDING -D__AIX -D__ESSL -D__FFTSG -D__parallel \
> -D__BLACS -D__SCALAPACK -D__FFTW3
> FFTW3_BASE = /bgsys/local/fftw3/3.3.2/fftw_g/
> CPPFLAGS = -C $(DFLAGS) -P -I$(FFTW3_BASE)/include
> FCFLAGS = -O2 -qarch=450d -qtune=450 -qstrict -qessl
> FCFLAGS2 = -O0 -qarch=450d -qtune=450 -qessl
> LDFLAGS = $(FCFLAGS)
> LIBS = /bgsys/local/scalapack/2.0.1_g/lib/libscalapack.a \
> /bgsys/local/lapack/3.3.0_g/lib/liblapack.a \
> $(FFTW3_BASE)/lib/libfftw3.a \
> /opt/ibmmath/essl/5.1/lib64/libesslbg.a
> 
> OBJECTS_ARCHITECTURE = machine_aix.o
> bibliography.o: bibliography.F
>          $(FC) -c $(FCFLAGS2) $<
> 
> Since we don't need LIBINT, I just removed all the related library path to make the compilation easier.. Also I didn't include libblacsF77init.a and libblacs.a, I cannot find them on our system, but our admin said they are integrated in libscalapack.a.
> 
> Thanks a lot in advance!
> 
> Best
> 
> Wei
> 
> On Wednesday, March 7, 2012 10:51:13 AM UTC+1, IBethune wrote:
> Wei,
> In CP2K, since some data relating to the atoms in the system (topology, neighbour lists etc.) are replicated, this implies that the system size is ultimately limited by the amount of memory per node, and although in practice this is only an issue for very large systems perhaps 100,000s of atoms.  Using more CPU cores will help to an extent since other data structures such as the sparse matrices will be distributed over more cores, and take up less memory per core, but for such a large system size you will likely need a large number of cores to accomplish your calculation in a reasonable amount of time in any case.
> 
> The scaling results reported in the PRACE white paper are for a 6144-atom system (DFT with MOLOPT basis set), and good scaling (close to linear) was achieved up to 8192 cores, which is the largest run we attempted.  In general, one would expect better scalability on the BG/P due to the fact it has slower cores and faster (lower latency, higher bandwidth) interconnect than a standard linux cluster.  I don't have any specific results to compare with however.
> 
> Re: BG/Q, given that IBM provides a linux-like environment with standard fortran Compilers I don't forsee huge issues with porting, but we have not tested this (yet).
> 
> Cheers
> 
> - Iain
> 
> --
> 
> Iain Bethune
> Applications Consultant, EPCC
> 
> Email: ibet... at epcc.ed.ac.uk
> Tel/Fax: +44 (0)131 650 5201/6555
> Mob: +44 (0)7598317015
> Addr: 2404 JCMB, The King's Buildings, Mayfield Road, Edinburgh, EH9 3JZ
> 
> 
> 
> On 6 Mar 2012, at 16:12, Wei wrote:
> 
> > Hi Iain,
> > 
> > thanks a lot for your fast reply!
> > 
> > I have read the document, and it is a huge surprise that you have
> > tried with 1 million atom system with DFT.. Now I have two further
> > questions:
> > 
> > 1. Does this newly compiled cp2k on Blue Gene has a lower limit of
> > atom size for a specific job (4000 cores)? Have you ever tried to run
> > md or relaxation for many steps for a smaller system size (i.e. 10,000
> > atom), and how big is the speedup factor compare to the calculation on
> > normal linux-cluster with intel cpus (let's say 200 cores with 3.06
> > GHz)?
> > 
> > 2. will the compilation be the same for Blue Gene/Q?
> > 
> > thanks a lot!
> > 
> > Best Regards,
> > 
> > Wei
> > 
> > 
> > On Mar 6, 3:04 pm, Iain Bethune <ibet... at epcc.ed.ac.uk> wrote:
> >> Hi Wei,
> >> 
> >> We ported CP2K to the BlueGene/P under PRACE last year.  See here :http://prace-portal.cscs.ch/uploads/tx_pracetmo/CP2K.pdffor details, incuding ARCH files and compiler details.
> >> 
> >> There were some problems, particularly with the mixed-mode (psmp) version of the code, but if you just want to do MPI it will be fine.
> >> 
> >> Cheers
> >> 
> >> - Iain
> >> 
> >> --
> >> 
> >> Iain Bethune
> >> Applications Consultant, EPCC
> >> 
> >> Email: ibet... at epcc.ed.ac.uk
> >> Tel/Fax: +44 (0)131 650 5201/6555
> >> Mob: +44 (0)7598317015
> >> Addr: 2404 JCMB, The King's Buildings, Mayfield Road, Edinburgh, EH9 3JZ
> >> 
> >> On 6 Mar 2012, at 14:02, Wei wrote:
> >> 
> >> 
> >> 
> >> 
> >> 
> >> 
> >> 
> >> 
> >> 
> >>> Dear all,
> >> 
> >>> I wonder if there is an efficient version of cp2k designed for IBM
> >>> Blue Gene? How could I find more information about it? Any suggestion
> >>> will be highly appreciated.
> >> 
> >>> Thanks in advance!
> >> 
> >>> Best Regards,
> >> 
> >>> Wei Zhang
> >> 
> >>> --
> >>> You received this message because you are subscribed to the Google Groups "cp2k" group.
> >>> To post to this group, send email to cp... at googlegroups.com.
> >>> To unsubscribe from this group, send email to cp2k+uns... at googlegroups.com.
> >>> For more options, visit this group athttp://groups.google.com/group/cp2k?hl=en.
> >> 
> >> --
> >> The University of Edinburgh is a charitable body, registered in
> >> Scotland, with registration number SC005336.
> > 
> > -- 
> > You received this message because you are subscribed to the Google Groups "cp2k" group.
> > To post to this group, send email to cp... at googlegroups.com.
> > To unsubscribe from this group, send email to cp2k+uns... at googlegroups.com.
> > For more options, visit this group at http://groups.google.com/group/cp2k?hl=en.
> > 
> >
> 
> 
> -- 
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups "cp2k" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/cp2k/-/XGzfWtQ-PpMJ.
> To post to this group, send email to cp... at googlegroups.com.
> To unsubscribe from this group, send email to cp2k+uns... at googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/cp2k?hl=en.


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.