Mpiexec and mcnpx

[Pete Wyckoff]

bvanhaer at sckcen.be wrote on Sat, 16 Oct 2004 12:05 +0200:
> I did edit the source because the mcnpx binary can't handle the
> arguments mpiexec passes to it. mcnpx also accepts arguments itself
> and can't differentiate betweem mpi arguments and mcnpx program
> arguments. So instead of editing the Fortran code that handles the
> program arguments, I decided to edit the mpiexec code that passes the
> arguments. The changes I made work because I see that the mcnpx
> program starts in pbs with the correct arguments. What happens is that
> the master process gets started, and starts doing its calculations but
> the slave processes don't start, because they can't contact the master
> process.

Some of those arguments turn out to be important to getting the parallel
job started.  If you change them, MPI_Init() won't know how it is
expected to behave regarding communicating with its peer processes.

> You say that MPI_Init() should try to contact mpiexec or mpirun
> somewhere. How would this code look ? Maybe I can change this in the
> mcnpx code that initiates MPI.

Really you don't want to change anything at that level.  What you really
want is for mcnpx not to see these "magic" arguments that are meant only
for MPI_Init.  This is only a problem due to the icky way that the P4
device in MPICH starts up processes; other devices, incl GM, SHMEM, IB,
use environment variables and are much easier to deal with.

In a C program, with a C-provided main(), a code snippet like this:

    main(int argc, char **argv)
    {
	printf("argc = %d\n", argc);
	MPI_Init(&argc, &argv);
	printf("now argc = %d\n", argc);
    }

will show you how MPI_Init() strips off these arguments so that your
program will not have to see them.  It is important to do argument
processing _after_ calling MPI_Init().

In FORTRAN, though, there's a long standing problem in MPICH that
arguments are not stripped like they are for C.  If you happen to
be using the Intel or PGI fortran compiler, I've attached a patch that
will fix this for your MPICH source tree.  You can try to modify it
for other compilers.

You still have to initialize MPI before looking at program arguments, in
mcnpx.  Else your only real option is to edit mcnpx's argument parsing
code to explicitly ignore all the "magic" args needed to start up the
parallel environment.  It obviously already knows about the ones that
mpirun uses, but mpiexec takes advantage of a couple different features
in the MPICH/P4 library and hence more arguments that must be ignored.

		-- Pete
-------------- next part --------------
diff -ruN mpich-1.2.6/src/fortran/src/initf.c mpich/src/fortran/src/initf.c
--- mpich-1.2.6/src/fortran/src/initf.c	2001-12-12 18:36:43.000000000 -0500
+++ mpich/src/fortran/src/initf.c	2004-09-03 17:13:23.000000000 -0400
@@ -117,6 +117,40 @@
 void mpir_getarg_ ( MPI_Fint *, char *, MPI_Fint );
 #endif
 
+#if defined(__PGI)
+
+FORTRAN_API void FORT_CALL mpi_init_(MPI_Fint *ierr)
+{
+    extern int __argc_save;
+    extern char **__argv_save;
+
+    *ierr = MPI_Init(&__argc_save, &__argv_save);
+}
+
+#elif defined(__ICC) && (__ICC >= 800)
+
+FORTRAN_API void FORT_CALL mpi_init_(MPI_Fint *ierr)
+{
+    extern int for__l_argc;
+    extern char **for__a_argv;
+
+    *ierr = MPI_Init(&for__l_argc, &for__a_argv);
+}
+
+#elif defined(__ICC)
+
+/* old versions of intel compiler */
+FORTRAN_API void FORT_CALL mpi_init_(MPI_Fint *ierr)
+{
+    extern int xargc;
+    extern char **xargv;
+
+    *ierr = MPI_Init(&xargc, &xargv);
+}
+
+#else
+/* unknown compiler, just copy argc/argv */
+
 FORTRAN_API void FORT_CALL mpi_init_( MPI_Fint *ierr )
 {
     int  Argc;
@@ -185,4 +219,5 @@
        must initialize all languages */
 }
 
+#endif  /* unknown compiler, copying argc/argv */
 #endif