Alliance Doc

MPI/en: Difference between revisions

Page Discussion

MPI/en (view source)

Revision as of 15:24, 24 October 2019

7,602 bytes added ,  5 years ago
Updating to match new version of source page
(Updating to match new version of source page)
(Updating to match new version of source page)
Line 23: Line 23:
== What is MPI? ==
== What is MPI? ==


The Message Passing Interface (MPI) is, strictly speaking, a ''standard'' describing a set of subroutines, functions, objects, ''etc.'', with which one can write parallel programs in a distributed memory environment. Many different ''implementations'' of the standard have been produced, such as Open MPI, MPICH, and MVAPICH. The standard describes how MPI should be called from Fortran, C, and C++ languages, but unofficial "bindings" can be found for several other languages.
The Message Passing Interface (MPI) is, strictly speaking, a ''standard'' describing a set of subroutines, functions, objects, ''etc.'', with which one can write parallel programs in a distributed memory environment. Many different ''implementations'' of the standard have been produced, such as Open MPI, Intel MPI, MPICH, and MVAPICH. The standard describes how MPI should be called from Fortran, C, and C++ languages, but unofficial "bindings" can be found for several other languages. Note that MPI 3.0 dropped official C++ bindings but instead you can use the C bindings from C++, or [https://www.boost.org/doc/libs/1_71_0/doc/html/mpi.html Boost MPI]. For Python we give examples using the MPI for Python package [https://mpi4py.readthedocs.io mpi4py].


Since MPI is an open, non-proprietary standard, an MPI program can easily be ported to many different computers. Applications that use it can run on a large number of cores at once, often with good parallel efficiency (see the [[Scalability | scalability page]] for more details). Given that memory is local to each process, some aspects of debugging are simplified --- it isn't possible for one process to interfere with the memory of another, and if a program generates a segmentation fault the resulting core file can be processed by standard serial debugging tools. However, due to the need to manage communication and synchronization explicitly, MPI programs may appear more complex than programs written with tools that support implicit communication. Furthermore, in designing an MPI program one should take care to minimize communication overhead to achieve a good speed-up from the parallel computation.
Since MPI is an open, non-proprietary standard, an MPI program can easily be ported to many different computers. Applications that use it can run on a large number of cores at once, often with good parallel efficiency (see the [[Scalability | scalability page]] for more details). Given that memory is local to each process, some aspects of debugging are simplified --- it isn't possible for one process to interfere with the memory of another, and if a program generates a segmentation fault the resulting core file can be processed by standard serial debugging tools. However, due to the need to manage communication and synchronization explicitly, MPI programs may appear more complex than programs written with tools that support implicit communication. Furthermore, in designing an MPI program one should take care to minimize communication overhead to achieve a good speed-up from the parallel computation.
Line 30: Line 30:


== MPI Programming Basics ==
== MPI Programming Basics ==
This tutorial will present the development of an MPI code in C and Fortran, but the concepts apply to any language for which MPI bindings exist. For simplicity our goal will be to parallelize the venerable "Hello, World!" program, which appears below for reference.
This tutorial will present the development of an MPI code in C, C++, Fortran, and Python, but the concepts apply to any language for which MPI bindings exist. For simplicity our goal will be to parallelize the venerable "Hello, World!" program, which appears below for reference.
<tabs>
<tabs>
<tab name="C">
<tab name="C">
Line 72: Line 72:
   
   
  end program hello
  end program hello
}}
</tab>
<tab name="Python">
{{File
  |name=hello.py
  |lang="python"
  |contents=
print('Hello, world!')
}}
}}
</tab>
</tab>
Line 88: Line 96:


=== Framework ===
=== Framework ===
Each MPI program must include the relevant header file (<tt>mpi.h</tt> for C/C++, <tt>mpif.h</tt> for Fortran), and be compiled and linked against the desired MPI implementation. Most MPI implementations provide a handy script, often called a ''compiler wrapper'', that handles all set-up issues with respect to <code>include</code> and <code>lib</code> directories, linking flags, ''etc.'' Our examples will all use these compiler wrappers:
Each MPI program must include the relevant header file or use the relevant module (<tt>mpi.h</tt> for C/C++, <tt>mpif.h</tt>, <tt>use mpi</tt>, or <tt>use mpi_f08</tt> for Fortran, where <tt>mpif.h</tt> is strongly discouraged and <tt>mpi_f08</tt> recommended for new Fortran 2008 code), and be compiled and linked against the desired MPI implementation. Most MPI implementations provide a handy script, often called a ''compiler wrapper'', that handles all set-up issues with respect to <code>include</code> and <code>lib</code> directories, linking flags, ''etc.'' Our examples will all use these compiler wrappers:
* C language wrapper: <tt>mpicc</tt>
* C language wrapper: <tt>mpicc</tt>
* Fortran: <tt>mpif90</tt>
* Fortran: <tt>mpifort</tt> (recommended) or <tt>mpif90</tt>
* C++: <tt>mpiCC</tt>
* C++: <tt>mpiCC</tt> or <tt>mpicxx</tt>


The copies of an MPI program, once they start running, must coordinate with one another somehow. This cooperation starts when each one calls an initialization function before it uses any other MPI features. The prototype for this function appears below:
The copies of an MPI program, once they start running, must coordinate with one another somehow. This cooperation starts when each one calls an initialization function before it uses any other MPI features. The prototype for this function appears below:
Line 109: Line 117:
  MPI_INIT(IERR)
  MPI_INIT(IERR)
  INTEGER :: IERR
  INTEGER :: IERR
</source>
</tab>
<tab name="Fortran 2008">
<source lang="fortran">
MPI_Init(ierr)
INTEGER, OPTIONAL, INTENT(OUT) :: ierr
</source>
</tab>
<tab name="Python (mpi4py)">
<source lang="Python">
# importing automatically initializes MPI with mpi4py
MPI.Init()
  </source>
  </source>
</tab>
</tab>
</tabs>
</tabs>
The arguments to the C <code>MPI_Init</code> are pointers to the <code>argc</code> and <code>argv</code> variables that represent the command-line arguments to the program. Like all C MPI functions, the return value represents the error status of the function. Fortran MPI subroutines return the error status in an additional argument, <code>IERR</code>.
The arguments to the C <code>MPI_Init</code> are pointers to the <code>argc</code> and <code>argv</code> variables that represent the command-line arguments to the program. Like all C MPI functions, the return value represents the error status of the function. Fortran MPI subroutines return the error status in an additional argument, <code>IERR</code>, which is optional if you <tt>use mpi_f08</tt>.


Similarly, we must call a function <code>MPI_Finalize</code> to do any clean-up that might be required before our program exits. The prototype for this function appears below:
Similarly, we must call a function <code>MPI_Finalize</code> to do any clean-up that might be required before our program exits. The prototype for this function appears below:
Line 129: Line 149:
  INTEGER :: IERR
  INTEGER :: IERR
</source>
</source>
</tab>
<tab name="Fortran 2008">
<source lang="fortran">
MPI_Finalize(ierr)
INTEGER, OPTIONAL, INTENT(OUT) :: ierr
</source>
</tab>
<tab name="Python (mpi4py)">
<source lang="Python">
# mpi4py installs a termination hook so there is no need to explicitly call MPI.Finalize.
MPI.Finalize()
</source>
</tab>
</tab>
</tabs>
</tabs>
Line 178: Line 210:
program phello0
program phello0
   
   
     include "mpif.h"
     use mpi
    implicit none
   
   
     integer :: ierror
     integer :: ierror
Line 187: Line 220:


  end program phello0
  end program phello0
}}
</tab>
<tab name="Fortran 2008">
{{File
  |name=phello0.f90
  |lang="fortran"
  |contents=
program phello0
    use mpi_f08
    implicit none
    call MPI_Init()
    print *, 'Hello, world!'
    call MPI_Finalize()
end program phello0
}}
</tab>
<tab name="Python (mpi4py)">
{{File
  |name=phello0.py
  |lang="python"
  |contents=
from mpi4py import MPI
print('Hello, world!')
}}
}}
</tab>
</tab>
Line 212: Line 271:
  MPI_COMM_RANK(COMM, RANK, IERR)
  MPI_COMM_RANK(COMM, RANK, IERR)
  INTEGER :: COMM, RANK, IERR
  INTEGER :: COMM, RANK, IERR
</source>
</tab>
<tab name="Fortran 2008">
<source lang="fortran">
MPI_Comm_size(comm, size, ierr)
TYPE(MPI_Comm), INTENT(IN) :: comm
INTEGER, INTENT(OUT) :: size
INTEGER, OPTIONAL, INTENT(OUT) :: ierr
MPI_Comm_rank(comm, rank, ierr)
TYPE(MPI_Comm), INTENT(IN) :: comm
INTEGER, INTENT(OUT) :: rank
INTEGER, OPTIONAL, INTENT(OUT) :: ierr
</source>
</tab>
<tab name="Python (mpi4py)">
<source lang="python">
MPI.Intracomm.Get_rank(self)
MPI.Intracomm.Get_size(self)
</source>
</source>
</tab>
</tab>
Line 271: Line 350:
  program phello1
  program phello1
   
   
     include "mpif.h"
     use mpi
    implicit none
 
     integer :: rank, size, ierror
     integer :: rank, size, ierror
   
   
Line 284: Line 364:
   
   
  end program phello1
  end program phello1
}}
</tab>
<tab name="Fortran 2008">
{{File
  |name=phello1.f90
  |lang="fortran"
  |contents=
program phello1
    use mpi_f08
    implicit none
    integer :: rank, size
    call MPI_Init()
    call MPI_Comm_size(MPI_COMM_WORLD, size)
    call MPI_Comm_rank(MPI_COMM_WORLD, rank)
    print *, 'Hello from process ', rank, ' of ', size
    call MPI_Finalize(ierror)
end program phello1
}}
</tab>
<tab name="Python (mpi4py)">
{{File
  |name=phello1.py
  |lang="python"
  |contents=
from mpi4py import MPI
comm = MPI.COMM_WORLD
size = comm.Get_size()
rank = comm.Get_rank()
print('Hello from process %d of %d'%(rank, size))
}}
}}
</tab>
</tab>
Line 300: Line 417:
If you are using the Boost version, you should compile with:  
If you are using the Boost version, you should compile with:  
  [~]$ mpic++ --std=c++11 phello1.cpp -lboost_mpi-mt -lboost_serialization-mt -o phello1
  [~]$ mpic++ --std=c++11 phello1.cpp -lboost_mpi-mt -lboost_serialization-mt -o phello1
If you are using the Python version, you don't need to compile but can run with:
[~]$ mpirun -np 4 python phello1.py


=== Communication ===
=== Communication ===
Line 337: Line 457:
  <type> MESSAGE(*)
  <type> MESSAGE(*)
  INTEGER :: COUNT, DATATYPE, DEST, TAG, COMM, IERR
  INTEGER :: COUNT, DATATYPE, DEST, TAG, COMM, IERR
</source>
</tab>
<tab name="Fortran 2008">
<source lang="fortran">
MPI_Send(message, count, datatype, dest, tag, comm, ierr)
TYPE(*), DIMENSION(..), INTENT(IN) :: message
INTEGER, INTENT(IN) :: count, dest, tag
TYPE(MPI_Datatype), INTENT(IN) :: datatype
TYPE(MPI_Comm), INTENT(IN) :: comm
INTEGER, OPTIONAL, INTENT(OUT) :: ierr
</source>
</tab>
<tab name="Python (mpi4py)">
<source lang="python">
# For general Python objects (pickled)
MPI.Intracomm.send(self, obj, int dest, int tag=0)
# For numpy arrays (fast)
MPI.Intracomm.Send(self, buf, int dest, int tag=0)
</source>
</source>
</tab>
</tab>
Line 342: Line 481:
Note that the <tt>datatype</tt> argument, specifying the type of data contained in the <tt>message</tt> buffer, is a variable. This is intended to provide a layer of compatibility between processes that could be running on architectures for which the native format for these types differs. It is possible to register new data types, but for this tutorial we will only use the predefined types provided by MPI. There is a predefined MPI type for each atomic data type in the source language (for C: <tt>MPI_CHAR</tt>, <tt>MPI_FLOAT</tt>, <tt>MPI_SHORT</tt>, <tt>MPI_INT</tt>, etc. and for Fortran: <tt>MPI_CHARACTER</tt>, <tt>MPI_INTEGER</tt>, <tt>MPI_REAL</tt>, etc.). You can find a full list of these types in the references provided below.
Note that the <tt>datatype</tt> argument, specifying the type of data contained in the <tt>message</tt> buffer, is a variable. This is intended to provide a layer of compatibility between processes that could be running on architectures for which the native format for these types differs. It is possible to register new data types, but for this tutorial we will only use the predefined types provided by MPI. There is a predefined MPI type for each atomic data type in the source language (for C: <tt>MPI_CHAR</tt>, <tt>MPI_FLOAT</tt>, <tt>MPI_SHORT</tt>, <tt>MPI_INT</tt>, etc. and for Fortran: <tt>MPI_CHARACTER</tt>, <tt>MPI_INTEGER</tt>, <tt>MPI_REAL</tt>, etc.). You can find a full list of these types in the references provided below.


<tt>MPI_Recv</tt> works in much the same way as <tt>MPI_Send</tt>. Referring to the function prototypes below, <tt>message</tt> is now a pointer to an allocated buffer of sufficient size to store <tt>count</tt> instances of <tt>datatype</tt>, to be received from process <tt>rank</tt>. <tt>MPI_Recv</tt> takes one additional argument, <tt>status</tt>, which should, in C, be a reference to an allocated <tt>MPI_Status</tt> structure, and, in Fortran, be an array of <tt>MPI_STATUS_SIZE</tt> integers. Upon return it will contain some information about the received message. Although we will not make use of it in this tutorial, the argument must be present.
<tt>MPI_Recv</tt> works in much the same way as <tt>MPI_Send</tt>. Referring to the function prototypes below, <tt>message</tt> is now a pointer to an allocated buffer of sufficient size to store <tt>count</tt> instances of <tt>datatype</tt>, to be received from process <tt>rank</tt>. <tt>MPI_Recv</tt> takes one additional argument, <tt>status</tt>, which should, in C, be a reference to an allocated <tt>MPI_Status</tt> structure, and, in Fortran, be an array of <tt>MPI_STATUS_SIZE</tt> integers or, for <tt>mpi_f08</tt>, a derived <tt>TYPE(MPI_Status)</tt> variable. Upon return it will contain some information about the received message. Although we will not make use of it in this tutorial, the argument must be present.
<tabs>
<tabs>
<tab name="C">
<tab name="C">
Line 372: Line 511:
  <type> :: MESSAGE(*)
  <type> :: MESSAGE(*)
  INTEGER :: COUNT, DATATYPE, SOURCE, TAG, COMM, STATUS(MPI_STATUS_SIZE), IERR
  INTEGER :: COUNT, DATATYPE, SOURCE, TAG, COMM, STATUS(MPI_STATUS_SIZE), IERR
</source>
</tab>
<tab name="Fortran 2008">
<source lang="fortran">
MPI_Recv(message, count, datatype, source, tag, comm, status, ierr)
TYPE(*), DIMENSION(..) :: message
INTEGER, INTENT(IN) :: count, source, tag
TYPE(MPI_Datatype), INTENT(IN) :: datatype
TYPE(MPI_Comm), INTENT(IN) :: comm
TYPE(MPI_Status) :: status
INTEGER, OPTIONAL, INTENT(OUT) :: ierr
</source>
</tab>
<tab name="Python (mpi4py)">
<source lang="python">
# For general Python objects (pickled)
MPI.Intracomm.recv(self, buf=None, int source=ANY_SOURCE, int tag=ANY_TAG, Status status=None)
# For numpy arrays (fast)
MPI.Intracomm.Recv(self, buf, int source=ANY_SOURCE, int tag=ANY_TAG, Status status=None)
</source>
</source>
</tab>
</tab>
Line 458: Line 617:


     implicit none
     implicit none
     include 'mpif.h'
     use mpi
     integer, parameter :: BUFMAX=81
     integer, parameter :: BUFMAX=81
     character(len=BUFMAX) :: outbuf, inbuf, tmp
     character(len=BUFMAX) :: outbuf, inbuf, tmp
Line 487: Line 646:
end program phello2
end program phello2
}}
}}
</tab>
<tab name="Fortran 2008">
{{File
  |name=phello2.f90
  |lang="fortran"
  |contents=
program phello2
    implicit none
    use mpi_f08
    integer, parameter :: BUFMAX=81
    character(len=BUFMAX) :: outbuf, inbuf, tmp
    integer :: rank, num_procs
    integer :: sendto, recvfrom
    type(MPI_Status) :: status
    call MPI_Init()
    call MPI_Comm_rank(MPI_COMM_WORLD, rank)
    call MPI_Comm_size(MPI_COMM_WORLD, num_procs)
    outbuf = 'Hello, world! from process '
    write(tmp,'(i2)') rank
    outbuf = outbuf(1:len_trim(outbuf)) // tmp(1:len_trim(tmp))
    write(tmp,'(i2)') num_procs
    outbuf = outbuf(1:len_trim(outbuf)) // ' of ' // tmp(1:len_trim(tmp))
    sendto = mod((rank + 1), num_procs)
    recvfrom = mod((rank + num_procs - 1), num_procs)
    call MPI_Send(outbuf, BUFMAX, MPI_CHARACTER, sendto, 0, MPI_COMM_WORLD)
    call MPI_Recv(inbuf, BUFMAX, MPI_CHARACTER, recvfrom, 0, MPI_COMM_WORLD, status)
    print *, 'Process', rank, ': Process', recvfrom, ' said:', inbuf
    call MPI_Finalize()
end program phello2
}}
</tab>
<tab name="Python (mpi4py)">
{{File
  |name=phello2.py
  |lang="python"
  |contents=
from mpi4py import MPI
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()
outbuf = "Hello, world! from process %d of %d" % (rank, size)
sendto = (rank + 1) % size;
recvfrom = (rank + size - 1) % size;
comm.send(outbuf, dest=sendto, tag=0)
inbuf = comm.recv(source=recvfrom, tag=0)
print('[P_%d] process %d said: "%s"]' % (rank, recvfrom, inbuf))
</tab>
</tab>
</tabs>
</tabs>
Line 655: Line 873:


     implicit none
     implicit none
     include 'mpif.h'
     use mpi


     integer, parameter :: BUFMAX=81
     integer, parameter :: BUFMAX=81
Line 689: Line 907:


end program phello3
end program phello3
}}
</tab>
<tab name="Fortran 2008">
{{File
  |name=phello3.f90
  |lang="fortran"
  |contents=
program phello3
    implicit none
    use mpi_f08
    integer, parameter :: BUFMAX=81
    character(len=BUFMAX) :: outbuf, inbuf, tmp
    integer :: rank, num_procs
    integer :: sendto, recvfrom
    type(MPI_Status) :: status
    call MPI_Init()
    call MPI_Comm_rank(MPI_COMM_WORLD, rank)
    call MPI_Comm_size(MPI_COMM_WORLD, num_procs)
    outbuf = 'Hello, world! from process '
    write(tmp,'(i2)') rank
    outbuf = outbuf(1:len_trim(outbuf)) // tmp(1:len_trim(tmp))
    write(tmp,'(i2)') num_procs
    outbuf = outbuf(1:len_trim(outbuf)) // ' of ' // tmp(1:len_trim(tmp))
    sendto = mod((rank + 1), num_procs)
    recvfrom = mod(((rank + num_procs) - 1), num_procs)
    if (MOD(rank,2) == 0) then
        call MPI_Send(outbuf, BUFMAX, MPI_CHARACTER, sendto, 0, MPI_COMM_WORLD)
        call MPI_Recv(inbuf, BUFMAX, MPI_CHARACTER, recvfrom, 0, MPI_COMM_WORLD, status)
    else
        call MPI_RECV(inbuf, BUFMAX, MPI_CHARACTER, recvfrom, 0, MPI_COMM_WORLD, status)
        call MPI_SEND(outbuf, BUFMAX, MPI_CHARACTER, sendto, 0, MPI_COMM_WORLD)
    endif
    print *, 'Process', rank, ': Process', recvfrom, ' said:', inbuf
    call MPI_Finalize()
end program phello3
}}
</tab>
<tab name="Python (mpi4py)">
{{File
  |name=phello3.py
  |lang="python"
  |contents=
from mpi4py import MPI
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()
outbuf = "Hello, world! from process %d of %d" % (rank, size)
sendto = (rank + 1) % size;
recvfrom = ((rank + size) - 1) % size;
if rank % 2 == 0:
    comm.send(outbuf, dest=sendto, tag=0)
    inbuf = comm.recv(source=recvfrom, tag=0)
else:
    inbuf = comm.recv(source=recvfrom, tag=0)
    comm.send(outbuf, dest=sendto, tag=0)
print('[P_%d] process %d said: "%s"]' % (rank, recvfrom, inbuf))
}}
}}
</tab>
</tab>
FuzzyBot
Bots
38,757

edits

Retrieved from "https://docs.alliancecan.ca/wiki/Special:MobileDiff/77510"