What does this error mean?

[Derbunovich, Andrey]

Kevin,

Thank you. Actually we never test our library with this mpiexec version.
Did you able to use this mpiexec with earlier versions of the Intel MPI
Library?

Best regards,
Andrey


-----Original Message-----
From: Kevin Van Workum [mailto:vanw at tticluster.com] 
Sent: Wednesday, January 30, 2008 3:30 PM
To: Derbunovich, Andrey
Cc: Thomas Zeiser; Pete Wyckoff; Slavova, Gergana S; Supalov, Alexander;
mpiexec at osc.edu
Subject: Re: What does this error mean?

On 1/30/08, Derbunovich, Andrey <andrey.derbunovich at intel.com> wrote:
> Hi,
>
> Which version of mpiexec do you use?
>
> Best regards,
> Andrey
>

# mpiexec --version
Version 0.82, configure options: '--with-default-comm=mpich2-pmi'
--prefix=/usr/local/mpiexec-0.82' '--with-pbs=/usr/local/pbs'

> On 1/29/08, Thomas Zeiser <thomas.zeiser at rrze.uni-erlangen.de> wrote:
> > Hi Kevin,
> >
> > what is the exact build id of your Intel MPI?
> >
> > You also might experiment with these Intel MPI environment
> > variables (assuming you have a recent Intel MPI 3.1): (export them
> > before calling mpiexec)
> >     I_MPI_USE_DYNAMIC_CONNECTIONS=on
> >     I_MPI_PMI_EXTENSIONS=off
> >     I_MPI_PMI_FAST_STARTUP=off
> > All of them take on/off. The settings given above is what we are
> > using.
> >
> > Regards,
> >
> > thomas
>
> Intel(R) MPI Library 3.1 for Linux*
> Package ID: l_mpi-rt_p_3.1.026
>
> I tried setting all the above to on and off and as shown above, but
> still the same problem.
>
> --
> Kevin
>
> > On Tue, Jan 29, 2008 at 03:28:40PM -0500, Kevin Van Workum wrote:
> > > On 1/29/08, Pete Wyckoff <pw at osc.edu> wrote:
> > > > vanw at tticluster.com wrote on Tue, 29 Jan 2008 14:31 -0500:
> > > > > On 1/29/08, Pete Wyckoff <pw at osc.edu> wrote:
> > > > > > vanw at tticluster.com wrote on Tue, 29 Jan 2008 08:19 -0500:
> > > > > > > Can you please interpret this error for me:
> > > > > > >
> > > > > > > mpiexec: Error: read_keyvals: no '=' found in keyval 4 of
> line: cmd.
> > > > > > >
> > > > > > > I get this error if my command line is either of the
> following:
> > > > > > >
> > > > > > > mpiexec -allstdin -comm=mpich2-pmi -verbose a.out blah
blah
> blah
> > > > > > > mpiexec -allstdin -comm mpich2-pmi -verbose a.out blah
blah
> blah
> > > > > > >
> > > > > > > I should note that a.out was built with Intel MPI.
> > > > > >
> > > > > > That's coming from part of the PMI message exchange between
> starting
> > > > > > processes and mpiexec.  The Intel MPI is somewhat of a rare
> find in
> > > > > > the wild.  If you run with lots of "-v -v -v", it will print
> out all
> > > > > > the details and we may be able to guess why it's unhappy.
> > > > >
> > > > > Here's the error message and some surrounding messages. Let me
> know if
> > > > > you need more of the earlier debug output.
> > > > >
> > > > > mpiexec: handle_pmi: rank 7 spawn 0 kvsname
53001.jman-spawn-0.
> > > > > mpiexec: read_keyvals: keyval 0 key cmd val get_my_kvsname.
> > > > > mpiexec: handle_pmi: cmd=get_my_kvsname.
> > > > > mpiexec: do_child: poll got 1.
> > > > > mpiexec: handle_pmi: rank 0 spawn 0 kvsname
53001.jman-spawn-0.
> > > > > mpiexec: read_keyvals: keyval 0 key cmd val put.
> > > > > mpiexec: read_keyvals: keyval 1 key kvsname val
> 53001.jman-spawn-0.
> > > > > mpiexec: read_keyvals: keyval 2 key key val DAPL_PROVIDER.
> > > > > mpiexec: read_keyvals: keyval 3 key value val <NULL.
> > > > > mpiexec: Error: read_keyvals: no '=' found in keyval 4 of
line:
> cmd.
> > > > > mpiexec: process_obit_event: evt 10 task 0 on n143104 stat 13.
> > > > > mpiexec: stdio_msg_parent_read: pipe closed.
> > > > > mpiexec: kill_stdio: sent SIGTERM, waiting on 25806.
> > > > > mpiexec: Warning: tasks 0-7 exited with status 13.
> > > >
> > > > (Adding some Intel MPI folks to CC.)
> > > >
> > > > I should have asked for one more "-v" so we could see the actual
> PMI
> > > > line sent by the Intel MPI library.  But we can mostly
reconstruct
> > > > it from the above.
> > > >
> > > > Rank 0 tries to do a PMI put, to store a value into its
> dictionary.
> > > > The line from task 0 probably looks like this:
> > > >
> > > > cmd=put kvsname=53001.jman-spawn-0 key=DAPL_PROVIDER value=<NULL
> cmd
> > > >
> > > > Not sure if anything follows.
> > > >
> > > > Is DAPL_PROVIDER a key anyone recognizes?  Any ideas how it
could
> > > > work out to be "<NULL"?  Maybe some misconfiguration on Kevin's
> > > > system?
> > > >
> > > >                 -- Pete
> > > >
> > >
> > > With one more '-v' I got this for PMI line:
> > >
> > > mpiexec: read_keyvals: read 73 chars: cmd=put
> > > kvsname=53002.jman-spawn-0 key=DAPL_PROVIDER value=<NULL string>
> > >
> > > --
> > > Kevin
> >
>
>
> --
> Kevin Van Workum, Ph.D.
> Tsunamic Technologies Inc.
> Vice President
> www.clusterondemand.com
> ONLINE COMPUTER CLUSTERS
> --------------------------------------------------------------------
> Closed Joint Stock Company Intel A/O
> Registered legal address: 125252, Moscow, Russian Federation,
> Chapayevsky Per, 14.
>
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
>
>


-- 
Kevin Van Workum, Ph.D.
Tsunamic Technologies Inc.
Vice President
www.clusterondemand.com
ONLINE COMPUTER CLUSTERS