[CP2K-user] [CP2K:18446] Running Cp2k in parallel using thread in a PC

[Corrado Bacchiocchi]

 Hi Everyone,

thanks for the many suggestions.

I have a single node server with two Xeon Gold 6154 CPU @ 3.00GHz, 18 cores 
each, 36 total, 72 threads.
I have found that the following launch command:

mpirun -np 72 --bind-to hwthread  cp2k.psmp -i cp2k.inp -o cp2k.out 

performs about 5x faster than

mpirun -n 2 --bind-to numa --map-by numa -display-map cp2k.psmp -i cp2k.inp 
-o cp2k.out

Regards
Corrado
On Monday, June 6, 2022 at 1:58:42 PM UTC+2 Matthew Graneri wrote:

> Hi Pierre,
>
> Sorry it's taken so long to reply. Your reply really helped. Thank you!
>
> Regards,
>
> Matthew
>
> On Friday, May 20, 2022 at 7:16:43 PM UTC+8 wave... at gmail.com wrote:
>
>> Hi Everyone
>>
>> While I haven't figured out the GPU side of things (btw, only part of 
>> cp2k is GPU-optimized), I found this approach useful for mpirun. Note that 
>> many people do not recommend using hyper-threading for this kind of 
>> application, so this will not give hyper-threading.
>>
>>      mpirun -n 2 --bind-to numa --map-by numa -display-map cp2k.psmp -i 
>> my-cp2k-run.inp > my-cp2k-run.out
>>
>>
>>    1. The 'bind-to numa' and 'map-by numa' make use of the os's 
>>    understanding of the processor.
>>    2. These two together neatly puts the mpi ranks per cpu socket.
>>    3. The '-display-map' writes the mpi assignments at the beginning of 
>>    the output.
>>
>> Hope this helps!
>>
>> Kind Regards
>>
>> Sam
>> On Wednesday, May 18, 2022 at 12:23:50 PM UTC+2 pierre.an... at gmail.com 
>> wrote:
>>
>>> Hi Matthew,
>>>
>>>  
>>>
>>> Unfortunately, there’s no single way to determine the best MPI/OpenMP 
>>> load. It is system, calculation type, and hardware dependant. I recommend 
>>> testing the performance. The first thing you could try is check if your 
>>> CPUs are multithreaded. For example, if they are made of 34 cores and 2 
>>> virtual cores per physical core (68 virtual cores in total), you could try 
>>> OMP_NUM_THREADS=2 and keep your mpirun -np (34*#nodes).
>>>
>>>  
>>>
>>> Roughly speaking, MPI creates multiple replica of the calculation 
>>> (called process), each replica dealing with part of the calculation. CP2K 
>>> is efficiently parallelized with MPI. OpenMP generated multiple threads on 
>>> the fly, generally to parallelize a loop. OpenMP can be used in a MPI 
>>> thread but not the other way around. Typically, having more MPI processed 
>>> consumes more memory than the same number of OpenMP threads. To use 
>>> multiple nodes, MPI is mandatory and more efficient. These are generalities 
>>> and, again, combining both is best but the ideal ratio varies. Testing is 
>>> the best course of action, check which combination yields the largest 
>>> number of ps/day with the minimum hardware resources. Doubling the hardware 
>>> does not double the output, so increasing the number of nodes becomes a 
>>> waste of resources at some point.  A rule of thumb, if the increase in 
>>> output is less than 75-80% of the ideal case, then, it is not worth it.
>>>
>>>  
>>>
>>> As you can see, there is a lot of try and error, no systematic rule I am 
>>> afraid.
>>>
>>>  
>>>
>>> Regards,
>>>
>>> Pierre
>>>
>>>  
>>>
>>>  
>>>
>>>  
>>>
>>> *From: *cp... at googlegroups.com <cp... at googlegroups.com> on behalf of 
>>> Matthew Graneri <mhvg... at gmail.com>
>>> *Date: *Wednesday, 18 May 2022 at 10:35
>>> *To: *cp2k <cp... at googlegroups.com>
>>> *Subject: *Re: [CP2K:16997] Running Cp2k in parallel using thread in a 
>>> PC
>>>
>>> Hi Pierre,
>>>
>>>  
>>>
>>> I found this really valuable! Unfortunately, being very new to AIMD and 
>>> very unfamiliar with computation in general, I was wondering if I might be 
>>> able to get some advice? We have a HPC at my university where each node has 
>>> 34 processors, and ~750 GB RAM available for use. It runs on a slurm 
>>> queuing system.
>>>
>>>  
>>>
>>> Until now, I've run all my jobs using: mpirun -np $SLURM_NTASKS 
>>> cp2k.popt -I input.inp -o output.out
>>>
>>> where $SLURM_NTASKS is whatever number of processors I've allocated to 
>>> the job via the --ntasks=x flag.
>>>
>>>  
>>>
>>> So instead, I'm thinking it might be more appropriate to use the .psmp 
>>> executable, but I'm not sure what the difference between the OpenMP and the 
>>> MPI threads are, and what kind of ratios between the OMP and MPI threads 
>>> would be most effective for speeding up an AIMD job, and how many threads 
>>> of each type you can add before the parallelisation becomes less efficient.
>>>
>>>  
>>>
>>> Do you (or anyone else) have any advice on the matter? Is it better to 
>>> have more OMP or MPI threads? And how many OMP threads per MPI thread would 
>>> be appropriate? What kinds of ratios are most effective at speeding up 
>>> calculations?
>>>
>>>  
>>>
>>> I would really appreciate any help I can get!
>>>
>>>  
>>>
>>> Regards,
>>>
>>>  
>>>
>>> Matthew
>>>
>>> On Friday, September 20, 2019 at 10:45:55 PM UTC+8 
>>> pierre.an... at gmail.com wrote:
>>>
>>> Hello Nikhil,
>>>
>>> Withe command "mpirun -n 42 cp2k.pop -i inp.inp -o -out.out", you are 
>>> requesting 42 MPI threads and not 42 OpenMP threads. MPI usually relies on 
>>> replicated data which means that, for a poorly program software, it will 
>>> request a total amount of memory which the amount of memory required by a 
>>> scalar execution times the number of threads. This can very quickly become 
>>> problematic, in particular for QM calculations. OpenMP, however relies on 
>>> shared memory, the data is normally not replicated but shared between 
>>> threads and therefore, in an ideal scenario, the amount of memory needed 
>>> for 42 OpenMP threads is the same as a single one.
>>>
>>> This might explains why you calculation freezes. You are out of memory. 
>>> On your workstation, you should only use the executable "cp2k.ssmp" which 
>>> is the OpenMP version. Then you don't need the mpirun command:
>>>
>>> cp2k.ssmp -i inp.inp -o -out.out
>>>
>>> To control the number of OpenMP threads, set the env variable: 
>>> OMP_NUM_THREADS, e.g. in bash, export OMP_NUM_THREADS=48
>>>
>>> Now, if you need to balance between MPI and OpenMP, you should use the 
>>> executable named cp2k.psmp. Here is such an example:
>>>
>>> export OMP_NUM_THREADS=24
>>> mpirun -n 2 cp2k.psmp -i inp.inp -o -out.out
>>>
>>> In this example, I am requesting two MPI threads and each of them can 
>>> use up to 24 OpenMP threads.
>>>
>>> Hope this clarifies things for you.
>>>
>>> Regards,
>>> Pierre
>>>
>>>  
>>>
>>> On 20/09/2019 14:09, Nikhil Maroli wrote:
>>>
>>> Dear all, 
>>>
>>>  
>>>
>>> I have installed all the versions of CP2K in my workstation with 2 x 12 
>>> core processor, total thread=48
>>>
>>>  
>>>
>>> I wanted to run cp2k in parallel using 42 threads, can anyone share the 
>>> commands that i can use.
>>>
>>>  
>>>
>>> I have tried 
>>>
>>>  
>>>
>>> mpirun -n 42 cp2k.pop -i inp.inp -o -out.out
>>>
>>>  
>>>
>>> After this command there is a rise in memory to 100 % and the whole 
>>> system freezes. (i have 128GB ram).
>>>
>>>  
>>>
>>> Any suggestion will be greatly appreciated,
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "cp2k" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to cp2k+uns... at googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/cp2k/39284c57-f6eb-463e-81a6-3a123596a9f2%40googlegroups.com 
>>> <https://groups.google.com/d/msgid/cp2k/39284c57-f6eb-463e-81a6-3a123596a9f2%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>>
>>>
>>> -- 
>>>
>>> Dr Pierre Cazade, PhD
>>>
>>> AD3-023, Bernal Institute,
>>>
>>> University of Limerick,
>>>
>>> Plassey Park Road,
>>>
>>> Castletroy, co. Limerick,
>>>
>>> Ireland
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "cp2k" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to cp2k+uns... at googlegroups.com.
>>>
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/cp2k/010a2dd7-dc2c-4475-8a9b-17cdbb10d20dn%40googlegroups.com 
>>> <https://groups.google.com/d/msgid/cp2k/010a2dd7-dc2c-4475-8a9b-17cdbb10d20dn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+unsubscribe at googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/4bec67f0-32db-4c70-9dea-bc2562010fd3n%40googlegroups.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.cp2k.org/archives/cp2k-user/attachments/20230214/8ff44108/attachment.htm>