confusion

[Pete Wyckoff]

vanallsburg at hope.edu wrote on Mon, 07 Nov 2005 15:16 -0500:
> I am trying to learn how to correctly add mpiexec-0.80 from OSU and 
> LAM/MPI on a small cluster running FC3, torque-1.2.0p2 and Intel's 
> Cluster tools(MPI  library).  I want to be able to use Torque to 
> submit/control MPI jobs the cluster. Users currently use torque and 
> submit(qsub) jobs, of which none of these apps require MPI (as of yet).
> 
> My confusion revolves around too many versions of mpiexec, and to what 
> extent does mpiexec-0.80 setup the MPI environment.  As a rookie, I'm 
> swamped in old/new documentation and can't see the forest for the trees.
> 
> Torque will manage the resources & start jobs with available node lists,
> if mpiexec-0.80  is used to start an MPI aware app
>    is it still required to do
> 	lamboot
> 	mpiexec
> 	and lamhalt?
>    Or does mpiexec-0.80 setup the boot-schema?
> 
> The same question exists with Intel's environment of
> 	mpdboot
> 	mpiexec
> 	and mpdallexit

First you don't have to bother with PBS integration if you just want
to run MPI jobs.  Just use the node list in $PBS_NODEFILE with the
non-integrated tools, e.g. lamboot for LAM or mpdboot for Intel MPI
(or you can use good-old mpirun with Intel MPI probably too).

But assuming you want PBS to have control and accounting of your MPI
tasks, you need to do something different.  In the LAM world, I'd
suggest that you ignore the OSC PBS mpiexec and try their native TM
interface.  It took me a while to find this on the LAM site:

    http://www.lam-mpi.org/faq/category4.php3#question6

It appears you still need to do lamboot etc, but bug them or their
manual for more details.

If you want to use Intel's MPI, configure the OSC PBS mpiexec-0.80
with the argument --with-default-comm=mpich2-pmi and use that
to launch your jobs:  no need for MPD at all.

Sorry it's all confusing.  If you have any suggestions for
documentation in mpiexec, I'll be happy to add something to
www.osc.edu/~pw/mpiexec/ that may help others figure this out too.

		-- Pete