multiple processes per node

[Pete Wyckoff]

jrbcast at teleline.es said on Fri, 23 May 2003 11:15 +0200:
> Due to the small size of our cluster sometimes we need to run a number
> of copies of our mpi programs which exceeds the number of processors. We
> used to do it with mpirun but this had the disadvantages that pbs didn't
> track the works properly so we passed to mpiexec. Now we are bnot able
> to do that (Runnning multiple processes on a smaller number of
> processors), somting like this:
> 
> #PBS -l nodes=4
> mpiexec -np 10 ....

This kind of goes outside a different goal of using mpiexec in a batch
environment, as users could consume more processors per SMP node
than they had been allocated, for instance.  But it should be fairly
easy to toss in as a compile-time option, if it would be useful for
someone.

How would you think it would allocate the processes in your example
above?  Round robin like this, perhaps?

    node0 - proc0, proc4, proc8
    node1 - proc1, proc5, proc9
    node2 - proc2, proc6
    node3 - proc3, proc7

Or packed to take advantage of maybe nearest-neighbor communication?

    node0 - proc0, proc1, proc2
    node1 - proc3, proc4, proc5
    node2 - proc6, proc7
    node3 - proc8, proc9

I'll see how hard this is to implement.

		-- Pete