two mpiexec questions on compatability

[Pete Wyckoff]

cpenney at ford.com said:
> Pete Wyckoff wrote:
> >So we use a static
> >mapping of processor# to GM port# and trust that PBS only assigns one
> >task per virtual processor.
> 
> My only comment on this is that I've seen several instances where an 
> application doesn't exit properly and since there is not implicit OS 
> teardown of connections (like with TCP) the other nodes in the process 
> just hang because they assume that the missing client is just being 
> slow.  Mpiexec handles this with -kill, but if someone else is using 
> mpirun, which has --gm-kill, but won't work if the script is killed 
> (like via a 'qdel').
> 
> It would be cool, if mpiexec could just probe for an available port 
> (checking to see if it is busy, etc.).  That would make GM ports more 
> like TCP/IP ports.

Agreed.  But it doesn't scale nicely.  Each compute node process must
know the hostname and myrinet port number of every other process, which
information is usually transmitted in the .gmpi/conf or equivalent file.
But only the localhost knows which ports on its GM device(s) are
available.  The algorithm to do this has to be global, and would go
something like:

    mpiexec: tm_spawn special executable on each node to find a free port
    mpiexec: write ~/.gmpiconf.whatever
    mpiexec: tm_spawn actual application

I just wasn't willing to do that.  If there were a mechanism by which
one Myrinet card could be used to probe port availability on other
nodes, we could use that locally on node#0.

We chastise users who still use mpirun.

The trace shows 7 messages went to the pbs_mom log file.  I'm especially
curious about the one by this subprocess:

> [pid   628] fork()                      = 2741
...
> [pid  2741] connect(4, {sin_family=AF_INET, sin_port=htons(35965), sin_addr=inet_addr("127.0.0.1")}}, 16) = -1 ECONNREFUSED (Connection refused)
> [pid  2741] osf_sigprocmask(0x1, 0x80000, 0x11fffe620) = 0
> [pid  2741] rt_sigaction(SIGCHLD, NULL, {SIG_DFL}, 8, 0) = 0
> [pid  2741] osf_sigprocmask(0x3, 0, 0)  = 524288
> [pid  2741] nanosleep({2, 0}, {2, 0})   = 0
> [pid  2741] connect(4, {sin_family=AF_INET, sin_port=htons(35965), sin_addr=inet_addr("127.0.0.1")}}, 16) = -1 ECONNREFUSED (Connection refused)
> [pid  2741] osf_sigprocmask(0x1, 0x80000, 0x11fffe620) = 0
> [pid  2741] rt_sigaction(SIGCHLD, NULL, {SIG_DFL}, 8, 0) = 0
> [pid  2741] osf_sigprocmask(0x3, 0, 0)  = 524288
> [pid  2741] nanosleep({2, 0}, {2, 0})   = 0
> [pid  2741] ioctl(2, TCGETS, 0x11fffd580) = -1 ENOTTY (Inappropriate ioctl for device)
> [pid  2741] gettimeofday({1022243485, 401566}, NULL) = 0
> [pid  2741] write(3, "05/24/2002 08:31:25;0001;   pbs_"..., 123) = 123

Any idea what was that about?

It might be easier to trace one of the pbs_mom which is not the mother
superior, but will be used to spawn a process for the job.  Unless you
can tell via the logs that the tm_spawn command never reaches the sister
nodes.

		-- Pete