Using mpiexec on non-PBS node for short test?

[Pete Wyckoff]

dog at lanl.gov wrote on Tue, 09 Oct 2007 12:56 -0600:
> We have mvapich2 running on an IB cluster here, under PBS control.   
> We build the standard mvapich2 including the mpich2 mpirun and  
> mpiexecs but at the tail end of that we delete mpiexec and mpirun in  
> favor of the PBS-enabled mpiexec.
> 
> Now we have run into a problem.  We have a user who has a complicated  
> build setup and needs to run a short "mpirun -n 1" test job as part  
> of his compilation process.  This breaks since mpiexec doesn't seem  
> to allow this sort of mode:
> 
> $ mpirun -n 1 my.exec
> mpirun: Error: PBS_JOBID not set in environment.  Code must be run  
> from a
>   PBS script, perhaps interactively using "qsub -I".
> 
> Are there any flags that can be passed to mpiexec to allow it to do a  
> simple shmem type of start up on a generic node?

It really doesn't like to do that.  There's too much information
that has to be gotten from PBS.  Like:  what nodes are available
to run the tasks?  And PBS handles all the work of startup and
tracking of tasks.

In fact, with your "mpirun -n 1" example, that relies on the fact
that you happen to be able to run on localhost in your cluster.  Our
sysadmins would get snippy if somebody tried to run on the login
nodes, and the program would fail as it's missing the communication
hardware.

So I'd advise you to get the user not to do that.  But if he really
wants to, mpich2 jobs just happen to work without a launcher.  Just
do "./my.exec".

The other approach is:  echo "cd $(pwd) ; mpiexec -n 1 my.exec" | qsub
or some such.  The runtests.pl script in mpiexec does the moral
equivalent of this to do lots of teensy test jobs.

		-- Pete