[Stgt-devel] Different throughput numbers on SLES 10 and RHEL 5.1

[Robin Humble]

On Tue, Feb 12, 2008 at 12:02:05PM +0100, Bart Van Assche wrote:
>On Feb 12, 2008 10:59 AM, Erez Zilber <erezz at voltaire.com> wrote:
>> I'm using the latest stgt release (tgt-20071227) and I see different
>> throughput numbers for READ commands on SLES 10  & RHEL 5.1. I'm using
>> stgt with iSCSI over iSER.
>Which numbers does ib_write_bw report between initiator and target for
>the two different Linux distributions ?

are you loading mthca with
  options ib_mthca tune_pci=1
so that lspci -vvv says:
  MaxReadReq 4096 bytes
for the IB card?
some distros seem to have this on by default, and others don't.
with the normal default of MaxReadReq 128(?) total IB bandwidth is
about halved.

you could look at lspci -vvv between the 2 distros and check.

as Bart says you should prob be able to see that difference with the ib
bw utils too, but in my testing I used netpipe's NPmpi with OpenMPI for
bandwidth tests.

looking at /proc/interrupts between the 2 distros might also tell you
something. I haven't tried pinning interrupts to cores and don't know
what the distros do by default. possibly msi_x=1 as another option to
ib_mthca might help too.

cheers,
robin