mpiexec and mvapich with InfiniBand Multicast-based MPI Broadcast

[Pete Wyckoff]

thomas.zeiser at rrze.uni-erlangen.de wrote on Mon, 02 May 2005 10:37 +0200:
> mvapich (0.9.4) has a compile-time option (MCST_SUPPORT) to
> activate "InfiniBand Multicast-based MPI Broadcast". 
> 
> When I starting a MCST_SUPPORT-enabled MPI-executable with the
> mpirun_rsh from mvapich everything is fine. When mpiexec is used
> instead, I get the following error message
> 
>    mpiexec -comm ib ./pi3
>    [1]pmgr_exchange_mcs_group_sync: read grp info 0(expected:8)
>    [2]pmgr_exchange_mcs_group_sync: read grp info 0(expected:8)
>    [3]pmgr_exchange_mcs_group_sync: read grp info 0(expected:8)
>    pmgr_exchange_mcs_group_sync: root read: Success
>    mpiexec: Warning: tasks 0-3 exited with status 1.
> 
> It seems that mvapich uses some additional initialisation sequences
> in this case ... (see also mpid/vapi/process/mpirun_rsh.c and
> mpid/vapi/process/pmgr_client_mpirun_rsh.c in the mvapich sources).

That is exactly the case.  There is extra setup required to support
this compile-time mvapich option that is not available in mpiexec.
There has not been enough interest around here to bother with that
option and thus fix up mpiexec to deal with it.

If you want to add code to mpiexec to do this, beware that your patched
version of mpiexec can no longer run non-MCST_SUPPORT mpich binaries.
We need to convince the mvapich people that they need to use version
numbers in the protocol to distinguish among the various different
startup sequences they use.

		-- Pete