[Stgt-devel] Strange throughput results with tgtd and iSCSI

[Pete Wyckoff]

bart.vanassche at gmail.com wrote on Thu, 07 Feb 2008 17:04 +0100:
> While running performance tests to compare various iSCSI target
> implementations I noticed that tgtd processes data writes slower via
> iSCSI for a block transfer size of 1 MB than for 512 KB. Although such
> large block transfer sizes are irrelevant when a filesystem is mounted
> on top of an iSCSI target, I decided to report this anyway.
> 
> Kernel version: 2.6.23.14
> STGT version (target): 20071227. The target was set up such that a RAM
> disk of 2GB in size was exported via iSCSI.
> open-iscsi version (initiator): 2.0.865-1
> Performance related settings on the iSCSI initiator:
> node.session.iscsi.FirstBurstLength = 262144,
> node.session.iscsi.MaxBurstLength = 16776192,
> node.conn[0].iscsi.MaxRecvDataSegmentLength = 131072 and
> node.conn[0].tcp.window_size = 524288.
> Test commands that were run on the initiator:
> dd if=/dev/zero of=/dev/sde bs=512K oflag=direct
> dd if=/dev/zero of=/dev/sde bs=1M oflag=direct
> 
> Results on Ethernet:
> For a block size of 512 KB: write throughput of 43 MB/s.
> For a block size of 1 MB: write throughput of 15 MB/s.
> 
> Results on with IPoIB:
> For a block size of 512 KB: write throughput of 95 MB/s.
> For a block size of 1 MB: write throughput of 26 MB/s.

Nice report.  I'll try to take a look at the ethernet numbers first.

Please show me the output of "iscsiadm -m session -P 3".  It has the
actual negotiated connection params, which will differ from what you
quote from the iscsid.conf above.  You may find it handy to do "rm
-rf /var/lib/iscsi/*/*" to destroy all iscsi state if you find the
negotiated values are not what you expect.

My nodes only have 2G.  When I build a tmpfs, put in a 2G file,
export that, these tests essentially measure swap performance of the
system.  I get 12 MB/s for writes at both block sizes.  Running
"vmstat 2" on the target shows it is busy swapping.

Redoing the tests at 1G keeps it all in RAM for me.  Then I get 95
MB/s at 512 kB and 88 MB/s at 1MB, so confirming your performance
disparity, but not finding numbers nearly so bad.

Here's the good bits from iscsiadm -m session -P 3 here:

    Negotiated iSCSI params:
    ************************
    HeaderDigest: None
    DataDigest: None
    MaxRecvDataSegmentLength: 131072
    MaxXmitDataSegmentLength: 131072
    FirstBurstLength: 131072
    MaxBurstLength: 16776192
    ImmediateData: Yes
    InitialR2T: Yes
    MaxOutstandingR2T: 1

and the script I use to try to replicate your seutp:

    #!/bin/bash
    ./tgtadm --lld iscsi --mode target --op new --tid 1 --targetname $(hostname)
    ./tgtadm --lld iscsi --mode target --op bind --tid 1 --initiator-address ALL
    ./tgtadm --lld iscsi --mode logicalunit --op new --tid 1 --lun 1 --backing-store /tmp/ram/tgt-ram-1g-pw
    ./tgtadm --lld iscsi --mode logicalunit --op update --tid 1 --lun 1 --name=scsi_sn --value=$(hostname)
    ./tgtadm --lld iscsi --mode target --op update --tid 1 --name MaxRecvDataSegmentLength --value 131072
    ./tgtadm --lld iscsi --mode target --op update --tid 1 --name MaxXmitDataSegmentLength --value 131072
    ./tgtadm --lld iscsi --mode target --op update --tid 1 --name FirstBurstLength --value 131072
    ./tgtadm --lld iscsi --mode target --op update --tid 1 --name MaxBurstLength --value 16776192

Show me what you have and I'll try to replicate.  The 88 vs 95 is
enough in itself to cause alarm, which we can try to understand, but
I'd also like to make sure that you can get numbers in this
neighborhood at least.

		-- Pete