mpirun with mpich-gm and pbs

[Adam Gray]

We're using open pbs 2.3.12 with the mpiexec patches, mpiexec 0.76, mpich-gm 
2.0.23 with myrinet cards.

We can get mpirun to work with and without pbs, mpiexec just never works.

When we run mpiexec in verbose mode we get the following:

resolve_exe: using absolute exe "/usr/local/cth/bin/mpicth"
node  0: name = shrikenode04, mpname = shrikenode04, cpu = 1
node  1: name = shrikenode03, mpname = shrikenode03, cpu = 1
node  2: name = shrikenode02, mpname = shrikenode02, cpu = 1
node  3: name = shrikenode01, mpname = shrikenode01, cpu = 1
[1] Error: Unable to connect to the master !
[2] Error: Unable to connect to the master !
[3] Error: Unable to connect to the master !
[1] Error: Unable to connect to the master to abort !
[2] Error: Unable to connect to the master to abort !
[3] Error: Unable to connect to the master to abort !
forrtl: error (78): process killed (SIGTERM)

We can rsh and ssh without passwords between nodes. the command line passed 
through pbs was mpiexec -v -n 4  /usr/local/cth/bin/mpicth i=THEH.Common.in

We compiled PBS with the following configure: --with-default-comm=mpich-gm

Thanks,

Adam Gray