Re: [Beowulf] mpi slow pairs

2014-08-29 Thread Michael Di Domenico
On Fri, Aug 29, 2014 at 3:26 PM, Håkon Bugge wrote: > Hmm, are all pairs going through the spine? If not, look up the parking-lot > problem. Håkon i believe all the pairs do pass through a spine. i'm not familiar with the "parking-lot problem", i'll google it, but suspect a bazillion hits will c

Re: [Beowulf] mpi slow pairs

2014-08-29 Thread Greg Lindahl
On Fri, Aug 29, 2014 at 08:49:57AM -0700, Greg Lindahl wrote: > Huh, Intel (PathScale/QLogic) has shipped a NxN debugging program for > more than a decade. The first vendor I recall shipping such a program > was Microway. I guess it takes a while for good practices to spread > throughout our commu

Re: [Beowulf] mpi slow pairs

2014-08-29 Thread Michael Di Domenico
On Fri, Aug 29, 2014 at 11:38 AM, John Hearns wrote: >> Also have you run ibdiagnet to see if anything is flagged up? > > i've run a multitude of ib diags on the machines, but nothing is popping out > as wrong. what's weird is that it's only certain pairing of machines not any > one machine in

Re: [Beowulf] mpi slow pairs

2014-08-29 Thread Greg Lindahl
On Fri, Aug 29, 2014 at 11:30:09AM -0400, Michael Di Domenico wrote: > > Also have you run ibdiagnet to see if anything is flagged up? > > i've run a multitude of ib diags on the machines, but nothing is > popping out as wrong. what's weird is that it's only certain pairing > of machines not an

Re: [Beowulf] mpi slow pairs

2014-08-29 Thread Michael Di Domenico
On Fri, Aug 29, 2014 at 9:32 AM, John Hearns wrote: > I would say the usual tool for that pair-wise comparison is Intel IBM > https://software.intel.com/en-us/articles/intel-mpi-benchmarks > I hope I have got your requirement correct! John, Close, but not exact. IMB will test ranks, but will no

[Beowulf] mpi slow pairs

2014-08-29 Thread Michael Di Domenico
does anyone have any mpi code that checks for slow pairs between mpi ranks. and no i don't me a micro benchmark that i can run a bunch of times and tabulate the results. i mean a program i can submit with a bunch of ranks and have it test the throughput between each and spit out a result for all