[Stgt-devel] Different throughput numbers on SLES 10 and RHEL 5.1

[Erez Zilber]

Bart Van Assche wrote:

> On Feb 12, 2008 12:17 PM, Robin Humble <robin.humble+stgt at anu.edu.au> wrote:
>   
>> are you loading mthca with
>>   options ib_mthca tune_pci=1
>> so that lspci -vvv says:
>>   MaxReadReq 4096 bytes
>>     
>
> An even better way than specifying tune_pci=1 is to perform a BIOS
> upgrade. On an Intel S5000PAL this increased MaxReadReq from 128 to
> 512 bytes, which is enough to obtain maximal throughput on an SDR 4x
> IB network.
>
> Bart Van Assche.
>   

I'm adding Mike Christie. Mike - as the maintainer of stgt in RHEL 5,
maybe you can help.

I don't think that IB is the problem here. I ran ib_rdma_bw on both
distros and got ~1350 MB/sec. Note that although there's a big
difference between SLES & RedHat, we're talking about 400 MB/sec which
is much lower than IB bandwidth.

Another test that I ran that proves that IB is not the problem here - I
ran another iSCSI over iSER target (i.e. not stgt) on RHEL 5, and I got
400 MB/sec when running sgp_dd from the initiator. I guess that there's
something bad with stgt on RHEL 5...

So, here's the current status for the sgp_dd test:

    * Running open-iscsi over iSER against stgt that runs on SLES 10:
      READ/WRITE - 400 MB/sec
    * Running open-iscsi over iSER against stgt that runs on RHEL 5:
      READ - 240 MB/sec, WRITE - 400 MB/sec
    * Running open-iscsi over iSER against our own iSCSI over iSER
      target that runs on RHEL 5: READ/WRITE - 400 MB/sec
    * Running sgp_dd directly from the target machine: READ/WRITE - 400
      MB/sec