mpiexec compilation with pgi compilers

[Pete Wyckoff]

Bart.Brashers at mfgenv.com said:
> I finally decided it was time to update my system, so I upgraded to
> mpich-1.2.4, OpenPBS_2_3_16 and compiled and installed them with the
> Portland Group compilers.  I also got mpiexec-0.66, and tried to compile it.
> I used:
> 
>  setenv CC=pgcc 
>  ./configure -with-pbs=/usr/local/lib \
>    --with-pbssrc=/usr/local/OpenPBS_2_3_16 \
>    --with-smp-size=2 --with-default-comm=mpich-p4 \
>    --with-mpi=/usr/local/mpich >&! my.configure.log
> 
> which worked fine.

You can do without the "--with-pbssrc" since that's only for
ancient PBS.  We'll get rid of it someday.  Also read below for how
to change your --with-mpi arg.

> But when I tried to make mpiexec, I immediately get an
> error:
> 
>  pgcc -g -D__attribute__\(x\)= -I/usr/local/OpenPBS_2_3_16/src/include -c
> mpiexec.c
>  PGC-S-0039-Use of undeclared variable __func__ (mpiexec.c: 78)
>  PGC/x86 Linux/x86 4.0-1: compilation completed with severe errors
>  make: *** [mpiexec.o] Error 2
> 
> Ok, so maybe __func__ is something special the gcc understands.  So I
> unsetenv'd CC (so it defaults back to gcc), removed config.cache, and re-ran
> configure.  Now it compiles.

mpiexec is not especially critical for overall performance.  I've been
compiling using good old gcc, but worked around some pgcc issues just
now in the interest of portability.  It works here on x86 with pgcc
4.01.  Give it a try if you like (new version 0.67).

The symbol __func__ is part of the ISO C9X standard.  PGI seems not to
understand it, so I found __FILE__ which at least will help narrow down
the source of error messages a bit.

> Next, I did a "make hello" to test mpiexec.  That produced a bunch of
> errors:
[..]
> Note the reference to pgdbg_stub, the debugging tool that comes with the PG
> compilers.  I suppose this is because I compiled mpich with pgcc, not gcc.
> I really don't want to have to go back and recompile mpich, especially since
> my mpi programs won't work with gcc (only with the pg compilers).

Note that "hello" is just a test program, and you don't have to do this.
But I realize the instructions were a bit misleading, and quite
demanding in how you had your system arranged.  Hence the newest 0.67
mpiexec provides a different configure argument to wash away all
compiler issues and hide them in "mpicc", assuming everyone has one of
those.

In particular, specify on the configure line:

	--with-mpicc=/usr/local/mpich/bin/mpicc

and things will be setup so that the Makefile will respond to "make
hello" with:

    /usr/local/mpich/bin/mpicc -o hello hello.c

or leave the configure option off and trust whatever "mpicc" is found in
your path.

> I went ahead and ran an mpi job with mpiexec anyway, and although qstat says
> it is running ("Elap Time" is incremented) top shows that the mpi process is
> not running anywhere.  

This should not be related to any of the above.  Please try running
mpiexec with one or more "-v" flags to watch what it says it is going
to do and make sure it is right.  It will start the first process,
wait for a response from it, then start all the other processes.  Note
that you must have compiled mpich-p4 using "--with-comm=shared" (to be
made configurable later perhaps).

You might want to use something like "ps xwua" to see all the processes,
just in case there are MPI jobs which are not accumulating any CPU time.

> Any hints would be appreciated.  Otherwise, I'll just have to stick with
> mpirun.

Use whatever works for you, obviously.  I personally don't care.

		-- Pete