about sh-mem mechanism

[Troy Baer]

On Wed, 30 Oct 2002, Jose Luis Gordillo Ruiz wrote:
> > >  how can i disable shared mem communications between process in
> > >  a SMP node?
> > >
> > >   by the way, why mpiexec uses shmem by default in SMP nodes? i think
> > >   most people uses sockets. indeed, i think there is no performance
> > >   gain by using shmem vs. sockets (a parallel program is as fast as its
> > >   slowest process).
> >
> > That's interesting to hear.
> >
> > I thought the consensus was that you want to use shmem inside an SMP,
> > but have not done any performance tests myself.  Show me some plots
> > sometime, if you can motivate someone to do the data collection.
> 
>   well, i don't have plots or data,  but think about this:
>   performance in a parallel program is afected by several factors, but
>   2 of the most important are load balancing and comm time. Obviosly,
>   shmem comms are faster than sockets comms in a SMP, but shmem can't be
>   used outside the SMP, so sockets comms are always present and they
>   represent the dominant comm time. It is possible obtain a certain gain
>   by using shmem, but this imply redesign the load balancing of the
>   parallel algorithm. Because clusters of SMP are not (yet) the standard,
>   most parallel programas not take shmem into account. also there is
>   another approach to this, using an hibrid paradimg (OpenMP inside the SMP
>   and MPI-sockets outside) but, again, there is no proof that this is
>   better that pure MPI or MPI-shmem with MPI-sockets.

First off, you're assuming MPICH/ch_p4, but that's not the only option
out there.  MPICH/ch_gm for Myrinet-connected clusters also supports
a hybrid shmem/network configuration, though not using TCP/IP or sockets
for off-node communication.

Second, I think you'll find that most large clusters are clusters of
SMPs.  The only really big (>>100 nodes) cluster of uniproc nodes I
can think of is CPlant.

I've done some tests using MPI vs. hybrid MPI/OpenMP, and everything
I've seen says that the hybrid MPI/OpenMP approach only pays off at
very large processor counts doing something comm-intensive like a
parallel FFT (eg. NAS Parallel Benchmark FT).  For nearest-neighbor
communication patterns, doing shmem transfers can improve performance
because you only pay the big latency hit for going off-node for about
half your total communications.  Shmem helps a *lot* to lower your
average communications latency, in my experience.  (However, I'm also
dealing with MPICH/ch_gm, where going off-node is 2-3x increase in
latency; MPICH/ch_p4 is more like a 10x latency increase.)

	--Troy
-- 
Troy Baer                       email:  troy at osc.edu
Science & Technology Support    phone:  614-292-9701
Ohio Supercomputer Center       web:  http://oscinfo.osc.edu