order of assign MPI_rank to nodes

[Pete Wyckoff]

f.paladin at cineca.it wrote on Thu, 02 Dec 2004 11:33 +0100:
>   I have an MPI application that generate data locally to nodes in which
> it is dispatched. More, another part of this application will
> collect/use this nodes.
> 
> The two part have to dispatched in the same order, that is, the same
> MPI_rank have to go in the same nodes (as with mpirun -pgfile).
> 
> Is that garanteed by mpiexec?
> If not, there is a way to force mpiexec to do it?
> 
> We have to use mpiexec under pbs to launch the 2 part, in the same pbs
> session or, worst case, in 2 separated pbs session (in this case, we
> have obviusly to use pbs keywords to force to go in the same nodes)

If I understand you correctly, you have two parallel programs that must
run concurrently on the same set of nodes, in the same order.  The first
thing to worry about is that PBS allows only a single mpiexec in a batch
job, so you cannot do, say, this:

    mpiexec -np 2 job1 &
    mpiexec -np 2 job2

If you do manage to convince PBS to start two jobs on the same set of
nodes, MPI will honor the node list order as given by PBS in starting
the jobs.  The order of the nodes given in the output of "qstat -n",
which is the same order as given in $PBS_NODEFILE, will be used to assign
tasks to processors, starting from 0.

		-- Pete