mpiexec + stdio_fork

[Pete Wyckoff]

cdmaest at sandia.gov said on Thu, 26 Jun 2003 16:04 -0600:
> Has anyone seen an error of the following?
> ---
> mpiexec: Error: stdio_fork: need 1031 sockets, only 1024 available in system
> ---
> 
> I can't tell if it's a shell or an mpiexec issue.

Might be both.  Here is the comment in stdio.c that says what this
is about (edited for accuracy):

    /*
     * Ensure we can be connected to this many things at once:
     *   3 aggregated stdio streams to shell or some local output
     *   3 sockets listening for new connections from remote processes
     *   2 * num-processes  per-process connections for stdout/stderr
     *   Some more for stdin:
     *     -nostdin:  0
     *     (default): 1
     *     -allstdin: num_processes
     * On linux-2.4.0-test10 and thereabout, maxfd can be pushed up to
     * 1024*1024, perhaps using a setuid-wrapper program to do:
     *   #include <sys/resource.h>
     *   struct rlimit rlim;
     *   rlim.rlim_cur = rlim.rlim_max = 1024*1024;
     *   setrlimit(RLIMIT_NOFILE, &rlim);
     * That would allow for 350,000 processes with three connected
     * streams each, or 524,000 processs with just out/err.  Default
     * 1024 maxfd only allows 340 process, or 510 processes with just
     * out/err.
     */

You can try running "-nostdout" to get rid of 2*numproc sockets which
feed back stdout and stderr.

You can also try using that code to bump up the limit on the number of
open files in mpiexec.

You can use bash (or other shell) functions to up your limit:  "ulimit -a"
to show "open files", then "ulimit -n 8192" to increase it.  Only root
can increase this value, though, so it might have to go in some system
default place.  Compiled-in linux max is 1e6 files, though, with total
number across the system in /proc/sys/fs/file-max.

Or you can comment out the test that complains about needing 1031 vs
1024 sockets.  I'm not convinced that sysconf() is telling the truth
on all platforms.  Just convert error -> warning, and it will continue
in spite of the warning.  Let me know if you think that should be a
permanent change.

		-- Pete