ch_p4 / shmem on SMP linux cluster

[Frank Eisenmenger]

Thanks a lot Pete, for your help.

Just for confirmation:
In order to compile the 'shmem' duo "mpich + mpiexec", one 'configure's: --with-comm=shared & --with-device=ch_p4 (mpich) + --enable-p4-shmem (mpiexec),
but for the 'p4' combination, choose --with-comm=ch_p4 & --with-device=ch_p4 (mpich) + --disable-p4-shmem (mpiexec) ?

Sorry, but the first thought which came to my German mind, after reading:
> I'm suggesting you may want to ADD "--with-comm=shared" to that mpich configure line above to use the shmem implementation _inside_ the p4 device implementation..
.. was: have BOTH, --with-comm=p4_ch AND --with-comm=shared in 'configure' ONE version of mpich, and then, compiling TWO versions of mpiexec, by putting either "--disable-p4-shmem" or "--disable-p4-shmem" into the 'configure' line for mpiexec. That variant I excluded after looking at your configuration example.

>> NB: seems to work as well without '--with-smp-size=2' !
>Indeed!  I keep wondering where I have left that in the
documentation---it's an old option that is quietly accepted but ignored.

Found that in one of mpiexec mail archive, not in your documentation.

> What you are seeing with the longer runtime in the mpiexec case is the
two processes fighting for the shared resources on the single node:  the
memory bus is a frequent bottleneck, as might be the single ethernet port.

In order to squeeze the maximum out of our DUAL processor cluster, your hint:

> You can try an mpiexec run with "-l nodes=8:ppn=2" but then say
   mpiexec -pernode -np 8 amber
to get only one process on each of 8 nodes.

gonna be very helpful. I'll try that.

> Okay with you if I bounce this mail from you and my response to
the list?  So others might learn from the archives in the future.

Yes, for sure. It might be also useful to add examples for configuration of mpich with mpiexec to the docu of mpiexec. 

More than that:

Lubos, who issued the recipe in http://structbio.vanderbilt.edu/archives/amber-archive/2004/1080.phtml, has sent scripts for submitting jobs to PBS with mpiexec to me.
I am also compiling some scripts, which make it easier for our people to submit jobs to the cluster. I'll forward those scripts to you, when finished. Might be of interest.

> We solve that mpich bug in a different way, attached.  Should work for
you with the intel fortran compiler.

Sorry, but could you, please, add lines for 'ifort', too, and tell what the patch command should look like.

Regards,
Frank.


Dr. Frank Eisenmenger
Abt. NMR-unterstützte Strukturforschung
Forschungsinstitut für Molekulare Pharmakologie
13125 Berlin, Robert-Rössle-Str. 10
Tel.      +49/0-30-94793-278
FAX     +49/0-30-94793-169
E-Mail  eisenmenger at fmp-berlin.de