Avoiding MPI Deadlock or Race Conditions

Deadlock or race conditions occur when the message passing cannot be completed.


Consider the following and assume that the MPI_Send does not complete until the correspondingMPI_Recv is posted and visa versa. The MPI_Send commands will never be completed and the program will deadlock.

 if (rank == 0) {
      MPI_Send(..., 1, tag, MPI_COMM_WORLD);
      MPI_Recv(..., 1, tag, MPI_COMM_WORLD, &status);
 } else if (rank == 1) {
      MPI_Send(..., 0, tag, MPI_COMM_WORLD);
      MPI_Recv(..., 0, tag, MPI_COMM_WORLD, &status);
 }

  

There are a couple of ways to fix this problem. One way is to reverse the order of one of the send/receive pairs:

 if (rank == 0) {
            MPI_Send(..., 1, tag, MPI_COMM_WORLD);
            MPI_Recv(..., 1, tag, MPI_COMM_WORLD, &status);
 } else if (rank == 1) {
            MPI_Recv(..., 0, tag, MPI_COMM_WORLD, &status);
            MPI_Send(..., 0, tag, MPI_COMM_WORLD);
}

Another way is to make the send be a  non-blocking one (MPI_Isend)

if (rank == 0) {
            MPI_Isend(..., 1, tag, MPI_COMM_WORLD, &req);
            MPI_Recv(..., 1, tag, MPI_COMM_WORLD, &status);
            MPI_Wait(&req, &status);
 } else if (rank == 1) {
            MPI_Recv(..., 0, tag, MPI_COMM_WORLD, &status);
            MPI_Send(..., 0, tag, MPI_COMM_WORLD);
}