mpich/p4 failure when $SHELL=(t)csh

[Pete Wyckoff]

ben at bellatrix.pcl.ox.ac.uk said:
> The mpiexec manpage says:
> 
> 'It is functionally equivalent to
> rsh node "cd $cwd; $SHELL -c exec executable arguments"'
> 
> This is not strictly speaking true. According to the code, it is actually 
> equivalent to
> 
> rsh node "$SHELL -c cd $cwd; exec executable arguments"
> 
> i.e. the "cd" is executed by the user's shell. In most cases this works 
> out as being much the same thing. However, parallel jobs on our cluster 
> generally run from a temporary directory on the master node (or, as PBS 
> whimsically puts it, the "mother superior"). This directory does not exist 
> on the slaves, which get all the data they require from the master via. 
> MPI. Thus, when running an "mpiexec" command, lots of "no such directory" 
> errors pop up for the slave nodes. This is annoying, but not fatal, when 
> using bash as the shell; however, tcsh refuses to execute the second 
> command at all if the first "cd" command fails, which is pretty obviously 
> quite fatal.
> 
> Is there any "elegant" way of fixing this problem? Ideas that spring to 
> mind include:

This is a bit painful.  For mpich/gm the code doesn't change the working
directory on its own, but users of good old mpirun I think expect the
slave processes to end up in the same dir on each node.  We build a
/tmp/<jobid> on each compute node and tell users to cd into that from
the PBS script if they want a fast local filesystem and lots of free
space.  Just to point out that there may be lots of options here.

> 1. Spawn a "bash" process to execute the cd $cwd command, then exec the 
>    user's shell which in turn execs the binary.
> 2. Don't use $SHELL at all, but always use bash.
> 3. Tell my users that they can't use tcsh. ;)
> 4. Get rid of the first "cd" command entirely.
> 5. Patch the tm code in PBS so that a remote working directory can be 
>    specified for the tm_spawn function.
> 6. Set a variable for tcsh (and any other shell with this behaviour) to 
>    tell it to "continue on error" or some such.

I think I'd default with your patch, but maybe with a config option to
change the sense of the default runtime arg.

First, though, I'll go play around with

    rsh node "bash -c 'if [ -d $dir ] ; then cd $dir ; fi ; $SHELL -c
	\'exec executable arguments\''"

but it will be necessary to run through the executable and arguments and
change all ' to '\''.  Hope that's sufficient.  Ugh.

I initially wanted to use the same shell as $SHELL just in case the
somebody perverse wanted to take advantage of shell features on the
remote side:

    mpiexec -c=n foo \> foo-out.\$\(hostname\) 2\>\&1

Or whatever the icky equivalent is in some non-bash shell.

> 	I've gone for option 4 on our cluster, largely because almost all 
> of our mpiexec invocations are to start mpich/p4 jobs, and these have an 
> explicit working directory (via. the -p4wd option) anyway, and because a 
> "cd $pwd" command makes little sense for our setup. The attached patch 
> implements it. It's not a perfect solution, because if you do specify the 
> "-cd" option that I introduce in my patch, it still won't work if you use 
> tcsh. Perhaps a better long-term solution would include option 1 as 
> well... comments?

My take, then is that hopefully a modified (1) can get rid of both the
warning messages from bash and fix the non-bash-shell problem, while
still allowing both mpich/p4 and mpich/gm and ad-hoc usage.  Is there a
case where even testing the $dir would be a bad thing?

		-- Pete