multiple MPI jobs in a single allocation?

[Ron W. Green]

Pete,

Thanks for the prompt response.  I'll work with our user and convince 
them to use qsubs
and allow PBS to do their scheduling.

thanks

ron

Pete Wyckoff wrote:

>rwgree at sandia.gov said on Thu, 25 Mar 2004 17:50 +0000:
>  
>
>>A user here would like to do run multiple mpi jobs within a single PBS 
>>allocation.  Something like this (csh/tcsh):
>>
>>qsub -I -l nodes=2:ppn=2
>>host2$    mpiexec -nostdin -n 2 a.out >& out1.txt & ; mpiexec -nostdin 
>>-n 2 a.out >& out2.txt &
>>
>>The first job starts and runs as expected.  The second job errs with:
>>$ mpiexec: Error: get_hosts: tm_init: tm: not connected.
>>
>>As a newbie, my feable understanding is that the first mpiexec instance 
>>opens a fixed port to the pbs_mom on host2.  The second mpiexec, I'm 
>>assuming, tries to open the same port and finds it occupied.
>>
>>Is there any way to run multiple mpi jobs within a single allocation?
>>
>>This user has a program that basically grabs a chunk of the machine and 
>>then runs multiple jobs within the allocation, basically doing his own 
>>load balancing and scheduling.
>>The obvious answer is to use qsubs for each subtasks - this is not 
>>acceptable to the user since they'd like the jobs to start immediately 
>>so that they can be monitored.
>>    
>>
>
>I wish.  PBS seems to get really annoyed if you try to connect to it, in
>the tm_init sense, from multiple processes within the same job.  Even if
>you tried to run a second mpiexec on a different node within the
>allocation it would fail too.  It's not a matter of port limitations or
>such, at least, just a PBS design issue.
>
>Given your constraint about not using multiple qsubs, I think you're
>basically out of luck.  Of course, there's always "mpirun" via rsh but
>then you have to figure out how to clean up everything when it dies.
>
>One thing that _could_ be done is to hack the "a.out" code to compare
>the process number not to 0 and 1, but to odd and even, then set up a
>four-process job and have each of the two pairs go their separate ways,
>maybe setting up a sub-communicator for each sub-task.  You would have
>to manually filter the output to separate files too.
>
>It is quite possible to modify mpiexec to do what you want, but not
>trivial.  You'd have to have the second invocation look to see if
>another is running, then contact it to spawn two more tasks as a
>separate MPI job.  None of that functionality is available now.
>
>		-- Pete
>  
>

-- 
Ron W. Green
rwgree at sandia.gov
+1-505-284-1600

Sr. Engineer, ICC Applications Support