mpiexec conf file versus pg file

[Christopher D. Maestas]

Pete,

Thanks for the feedback.  We do split up the MPI_COMM_WORLD with the two executables. 
I shoulda detailed this example first.  We do the following:

qsub -I -l nodes=2:vis+2:storage
mkpbspgfile --pgfile=mypgfile storage:/usr/local/ioproc/ioproc vis:"/usr/local/ccaffeine/dccafe/cxx/dc/user_iface/demo-client --file /home/user1/processors --port 4539"
cat mypgfile
--- mypgfile
vis1     0 /usr/local/ccaffeine/dccafe/cxx/dc/user_iface/demo-client --file /home/user1/processors --port 4539
vis2     1 /usr/local/ccaffeine/dccafe/cxx/dc/user_iface/demo-client --file /home/user1/processors --port 4539
storage1 1 /usr/local/ioproc/ioproc
storage2 1 /usr/local/ioproc/ioproc
--- mypgfile

The problem is that mpirun and pg files don't deal well with arguments passed into an mpi executable.  
(I submitted a feature request to mpi-maint at mcs.anl.gov).  So when you do:
	mpirun -pg mypgfile echo	# doesn't matter for exe arg
it fails to read in the arguments for the executables.

So I'd like to request trying to do this with mpiexec instead.

qsub -I -l nodes=2:vis+2:storage
mkpbsmpiexecfile --mpiexecfile=mympiexecfile storage:/usr/local/ioproc/ioproc vis:"/usr/local/ccaffeine/dccafe/cxx/dc/user_iface/demo-client --file /home/user1/processors --port 4539"
cat mympiexecfile
--- mympiexec file
storage1 : /usr/local/ioproc/ioproc
vis1 : /usr/local/ccaffeine/dccafe/cxx/dc/user_iface/demo-client --file /home/user1/processors --port 4539
--- mympiecec file

mpiexec -config=mympiexecfile

Is this possible?

Regards,

On Fri, 7 Feb 2003, Pete Wyckoff wrote:

> cdmaest at sandia.gov said:
> > All 4 tasks started.
> > read_gm_startup_ports: waiting for info
> > read_gm_startup_ports: id 2 port 2 board 0 gm_node_id 203 pid  2878
> > read_gm_startup_ports: id 3 port 2 board 0 gm_node_id 202 pid 20417
> > hello
> > goodbye
> [..]
> > And it hangs ... Is it not possible to do a multiple program job launch with mpiexec using two
> > mpiprograms?
> 
> It would not be easy.
> 
> What's happening above is that mpiexec thinks you have a single MPI job
> with four tasks.  It is waiting to get information from each of them so
> that it can construct a global map of the GM connections and send it
> back out to all the tasks which can then proceed past MPI_Init.  But
> "echo" isn't acting like an MPICH/GM code, obviously.  :)
> 
> And you can't even get away with something like:
>     mpiexec -np 2 code1 &
>     mpiexec -np 2 code2
> because the TM interface in PBS doesn't allow more than one startup
> connection per job for some reason.
> 
> It is true that mpiexec could be modified to be able to express what
> you're trying to do with some config file extensions, but can I
> recommend that you just get two PBS jobs, each with half the nodes (or
> processors), and run a single mpiexec in each?  You could also break up
> MPI_COMM_WORLD into a pair of communicators, one for each of your
> sub-jobs, and let them proceed essentially independently but within the
> context of a single MPI parallel process.
> 
> 		-- Pete
> 

-- Chris