mpiexec and jobs that spawn subtask themselves

[Pete Wyckoff]

Luc-Henri.Jolly at lct.jussieu.fr wrote on Mon, 07 Feb 2005 14:52 +0100:
> I am trying to figure out if it is possible to have mpiexec running a 
> job which does it's on spawning.
> 
> What I want is mpiexec  running the master job with the list of nodes 
> that have been reserved by OpenPBS,
> but not mpiexec running as many copies of the master job as defined by 
> the ncpus option .
> 
> The standard mpirun is working well, but cannotbe correctly monitored by 
> OpenPBS.

You can certainly start mpiexec using "-np 1" or otherwise to limit the
number of tasks started to fewer than what was allocated by PBS.  But
then to spawn new ones, there's no API exported by mpiexec to do so.
Your only option would be to use rsh/ssh just like mpirun does and you
would be in the same unhappy situation.

It's not clear to me what the right interface should be to spawn tasks
during the course of a job.  Are you using an API like MPI-2.0 that has
MPI_Comm_spawn() to start tasks in the middle of a run?  Perhaps the
best thing to do there is to write a PBS-specific environment handler
inside your MPI library to interact with PBS the same way that mpiexec
does.

Others have infrequently asked for a way to start multiple concurrent
mpiexec processes from the command line in a single batch job.  Here
each individual spawned set of processes (subjob) would not know about
or communicate with the other ones, presumably.

Any suggestions?

		-- Pete