ch_p4 / shmem on SMP linux cluster

[Pete Wyckoff]

eisenmenger at fmp-berlin.de wrote on Tue, 08 Feb 2005 20:02 +0100:
> trying to figure out, how to compile mpich-1.26 & mpiexec-0.77 on a 
> Linux-cluster with a master node and 24 compute nodes connected by 
> ethernet (all nodes: Dual Xeon processors, OS: Fedora Core). Both 
> processors on each node share 2 GB of memory.
> 
> Compiled mpich '--with-comm=ch_p4'.
> Therefore configured mpiexec with --disable-p4-shmem 
> --with-default-comm=mpich-p4 & --with-smp-size=2.
> 
> mpiexec tests via ./runtests.pl worked fine.
> 
> Three questions:
> 
> Is this optimal to run jobs on 4, 8 .. processors, i.e. >= 2 nodes ?

I think that's your only option.  Mpich/p4 on ethernet is just fine.
The question to ponder is, should you enable the p4 shared memory
implementation to try to speed up communications inside a single
multiprocessor node?  We use it here; I haven't discovered a consensus
on the matter.

> Does it make sense to install an additional version of mpich/mpiexec 
> with '--with-comm=shared' for running jobs on single nodes (i.e. on 2 
> processors with shared memory) ?

Speaking from a user support angle, I would say no.  There is enough
debate on whether the mpich1/p4/shmem implementation is any faster than
just using mpich/p4/TCP on a node.  (There are two other shared memory
implementations in mpich1 you might consider too, but I definitely won't
recommend considering that.)

If you have a few codes of interest, your best bet would be to run them
against shmem or no-shmem versions of mpich and see which one is faster.
(Let me know if you do---it would be interesting.)

> May I compile ONE version of mpich with both types of comm., and select 
> the appropriate one by using mpiexec's runtime options -mpich-p4-(no)shmem ?

No.  Mpich1 won't let you do that, with or without mpiexec.  It's a
compile-time choice only.  You can complain to their mailing list, but
they'll tell you to update to mpich2.  You can't do what you want with
that code either, as far as I know.

		-- Pete