multiple MPI jobs in a single allocation?

[Pete Wyckoff]

rwgree at sandia.gov said on Thu, 25 Mar 2004 17:50 +0000:
> A user here would like to do run multiple mpi jobs within a single PBS 
> allocation.  Something like this (csh/tcsh):
> 
> qsub -I -l nodes=2:ppn=2
> host2$    mpiexec -nostdin -n 2 a.out >& out1.txt & ; mpiexec -nostdin 
> -n 2 a.out >& out2.txt &
> 
> The first job starts and runs as expected.  The second job errs with:
> $ mpiexec: Error: get_hosts: tm_init: tm: not connected.
> 
> As a newbie, my feable understanding is that the first mpiexec instance 
> opens a fixed port to the pbs_mom on host2.  The second mpiexec, I'm 
> assuming, tries to open the same port and finds it occupied.
> 
> Is there any way to run multiple mpi jobs within a single allocation?
> 
> This user has a program that basically grabs a chunk of the machine and 
> then runs multiple jobs within the allocation, basically doing his own 
> load balancing and scheduling.
> The obvious answer is to use qsubs for each subtasks - this is not 
> acceptable to the user since they'd like the jobs to start immediately 
> so that they can be monitored.

I wish.  PBS seems to get really annoyed if you try to connect to it, in
the tm_init sense, from multiple processes within the same job.  Even if
you tried to run a second mpiexec on a different node within the
allocation it would fail too.  It's not a matter of port limitations or
such, at least, just a PBS design issue.

Given your constraint about not using multiple qsubs, I think you're
basically out of luck.  Of course, there's always "mpirun" via rsh but
then you have to figure out how to clean up everything when it dies.

One thing that _could_ be done is to hack the "a.out" code to compare
the process number not to 0 and 1, but to odd and even, then set up a
four-process job and have each of the two pairs go their separate ways,
maybe setting up a sub-communicator for each sub-task.  You would have
to manually filter the output to separate files too.

It is quite possible to modify mpiexec to do what you want, but not
trivial.  You'd have to have the second invocation look to see if
another is running, then contact it to spawn two more tasks as a
separate MPI job.  None of that functionality is available now.

		-- Pete