[CP2K-user] [CP2K:17088] Running Cp2k in parallel using thread in a PC

[Matthew Graneri]

Hi Pierre,

Sorry it's taken so long to reply. Your reply really helped. Thank you!

Regards,

Matthew

On Friday, May 20, 2022 at 7:16:43 PM UTC+8 wave... at gmail.com wrote:

> Hi Everyone
>
> While I haven't figured out the GPU side of things (btw, only part of cp2k 
> is GPU-optimized), I found this approach useful for mpirun. Note that many 
> people do not recommend using hyper-threading for this kind of application, 
> so this will not give hyper-threading.
>
>      mpirun -n 2 --bind-to numa --map-by numa -display-map cp2k.psmp -i 
> my-cp2k-run.inp > my-cp2k-run.out
>
>
>    1. The 'bind-to numa' and 'map-by numa' make use of the os's 
>    understanding of the processor.
>    2. These two together neatly puts the mpi ranks per cpu socket.
>    3. The '-display-map' writes the mpi assignments at the beginning of 
>    the output.
>
> Hope this helps!
>
> Kind Regards
>
> Sam
> On Wednesday, May 18, 2022 at 12:23:50 PM UTC+2 pierre.an... at gmail.com 
> wrote:
>
>> Hi Matthew,
>>
>>  
>>
>> Unfortunately, there’s no single way to determine the best MPI/OpenMP 
>> load. It is system, calculation type, and hardware dependant. I recommend 
>> testing the performance. The first thing you could try is check if your 
>> CPUs are multithreaded. For example, if they are made of 34 cores and 2 
>> virtual cores per physical core (68 virtual cores in total), you could try 
>> OMP_NUM_THREADS=2 and keep your mpirun -np (34*#nodes).
>>
>>  
>>
>> Roughly speaking, MPI creates multiple replica of the calculation (called 
>> process), each replica dealing with part of the calculation. CP2K is 
>> efficiently parallelized with MPI. OpenMP generated multiple threads on the 
>> fly, generally to parallelize a loop. OpenMP can be used in a MPI thread 
>> but not the other way around. Typically, having more MPI processed consumes 
>> more memory than the same number of OpenMP threads. To use multiple nodes, 
>> MPI is mandatory and more efficient. These are generalities and, again, 
>> combining both is best but the ideal ratio varies. Testing is the best 
>> course of action, check which combination yields the largest number of 
>> ps/day with the minimum hardware resources. Doubling the hardware does not 
>> double the output, so increasing the number of nodes becomes a waste of 
>> resources at some point.  A rule of thumb, if the increase in output is 
>> less than 75-80% of the ideal case, then, it is not worth it.
>>
>>  
>>
>> As you can see, there is a lot of try and error, no systematic rule I am 
>> afraid.
>>
>>  
>>
>> Regards,
>>
>> Pierre
>>
>>  
>>
>>  
>>
>>  
>>
>> *From: *cp... at googlegroups.com <cp... at googlegroups.com> on behalf of 
>> Matthew Graneri <mhvg... at gmail.com>
>> *Date: *Wednesday, 18 May 2022 at 10:35
>> *To: *cp2k <cp... at googlegroups.com>
>> *Subject: *Re: [CP2K:16997] Running Cp2k in parallel using thread in a PC
>>
>> Hi Pierre,
>>
>>  
>>
>> I found this really valuable! Unfortunately, being very new to AIMD and 
>> very unfamiliar with computation in general, I was wondering if I might be 
>> able to get some advice? We have a HPC at my university where each node has 
>> 34 processors, and ~750 GB RAM available for use. It runs on a slurm 
>> queuing system.
>>
>>  
>>
>> Until now, I've run all my jobs using: mpirun -np $SLURM_NTASKS 
>> cp2k.popt -I input.inp -o output.out
>>
>> where $SLURM_NTASKS is whatever number of processors I've allocated to 
>> the job via the --ntasks=x flag.
>>
>>  
>>
>> So instead, I'm thinking it might be more appropriate to use the .psmp 
>> executable, but I'm not sure what the difference between the OpenMP and the 
>> MPI threads are, and what kind of ratios between the OMP and MPI threads 
>> would be most effective for speeding up an AIMD job, and how many threads 
>> of each type you can add before the parallelisation becomes less efficient.
>>
>>  
>>
>> Do you (or anyone else) have any advice on the matter? Is it better to 
>> have more OMP or MPI threads? And how many OMP threads per MPI thread would 
>> be appropriate? What kinds of ratios are most effective at speeding up 
>> calculations?
>>
>>  
>>
>> I would really appreciate any help I can get!
>>
>>  
>>
>> Regards,
>>
>>  
>>
>> Matthew
>>
>> On Friday, September 20, 2019 at 10:45:55 PM UTC+8 pierre.an... at gmail.com 
>> wrote:
>>
>> Hello Nikhil,
>>
>> Withe command "mpirun -n 42 cp2k.pop -i inp.inp -o -out.out", you are 
>> requesting 42 MPI threads and not 42 OpenMP threads. MPI usually relies on 
>> replicated data which means that, for a poorly program software, it will 
>> request a total amount of memory which the amount of memory required by a 
>> scalar execution times the number of threads. This can very quickly become 
>> problematic, in particular for QM calculations. OpenMP, however relies on 
>> shared memory, the data is normally not replicated but shared between 
>> threads and therefore, in an ideal scenario, the amount of memory needed 
>> for 42 OpenMP threads is the same as a single one.
>>
>> This might explains why you calculation freezes. You are out of memory. 
>> On your workstation, you should only use the executable "cp2k.ssmp" which 
>> is the OpenMP version. Then you don't need the mpirun command:
>>
>> cp2k.ssmp -i inp.inp -o -out.out
>>
>> To control the number of OpenMP threads, set the env variable: 
>> OMP_NUM_THREADS, e.g. in bash, export OMP_NUM_THREADS=48
>>
>> Now, if you need to balance between MPI and OpenMP, you should use the 
>> executable named cp2k.psmp. Here is such an example:
>>
>> export OMP_NUM_THREADS=24
>> mpirun -n 2 cp2k.psmp -i inp.inp -o -out.out
>>
>> In this example, I am requesting two MPI threads and each of them can use 
>> up to 24 OpenMP threads.
>>
>> Hope this clarifies things for you.
>>
>> Regards,
>> Pierre
>>
>>  
>>
>> On 20/09/2019 14:09, Nikhil Maroli wrote:
>>
>> Dear all, 
>>
>>  
>>
>> I have installed all the versions of CP2K in my workstation with 2 x 12 
>> core processor, total thread=48
>>
>>  
>>
>> I wanted to run cp2k in parallel using 42 threads, can anyone share the 
>> commands that i can use.
>>
>>  
>>
>> I have tried 
>>
>>  
>>
>> mpirun -n 42 cp2k.pop -i inp.inp -o -out.out
>>
>>  
>>
>> After this command there is a rise in memory to 100 % and the whole 
>> system freezes. (i have 128GB ram).
>>
>>  
>>
>> Any suggestion will be greatly appreciated,
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "cp2k" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to cp2k+uns... at googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/cp2k/39284c57-f6eb-463e-81a6-3a123596a9f2%40googlegroups.com 
>> <https://groups.google.com/d/msgid/cp2k/39284c57-f6eb-463e-81a6-3a123596a9f2%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>>
>>
>> -- 
>>
>> Dr Pierre Cazade, PhD
>>
>> AD3-023, Bernal Institute,
>>
>> University of Limerick,
>>
>> Plassey Park Road,
>>
>> Castletroy, co. Limerick,
>>
>> Ireland
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "cp2k" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to cp2k+uns... at googlegroups.com.
>>
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/cp2k/010a2dd7-dc2c-4475-8a9b-17cdbb10d20dn%40googlegroups.com 
>> <https://groups.google.com/d/msgid/cp2k/010a2dd7-dc2c-4475-8a9b-17cdbb10d20dn%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+unsubscribe at googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/b2fa6ff2-d78f-4fc4-992e-1b771d1ee509n%40googlegroups.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.cp2k.org/archives/cp2k-user/attachments/20220606/90b0f52b/attachment.htm>