[Stgt-devel] iSER tgtd performance

[Pete Wyckoff]

nejc.porenta at gmail.com wrote on Tue, 09 Oct 2007 10:46 +0200:
> I am using OSC git tgt tree for testing iSER with virtualization. My
> research focus in on I/O performance in virtualization tehniques
> (OpenVZ, VMWare server, Xen).
> 
> I am using CentOS5 + OFED1.2.5 where it is possible (on target and
> OpenVZ) with Dell 1950 servers with 4GB ram with Mellanox single port
> memfree infiniband cards.
> 
> On target node i usually setup my iSCSI target like this:
> dd if=/dev/zero bs=1k count=1 seek=1048575 of=/dev/shm/tid1lun1
> ./tgtadm --lld iscsi --mode target --op new --tid 1 --targetname node1.nejc.net
> ./tgtadm --lld iscsi --mode target --op bind --tid 1 --initiator-address ALL
> ./tgtadm --lld iscsi --mode logicalunit --op new --tid 1 --lun 1
> --backing-store /dev/shm/tid1lun1
> 
> And the performance from client using ib_iser, measured with disktest,
> is around 1030MB/s which is quite impressive.
> 
> But the performance drops significantly when I increase the backing
> store size to 3.5GB. Usually I get around 400MB/s. This
> happens with tmpfs or with /dev/ram0.

Apparently you have DDR IB cards.  1030 MB/s is quite amazing.
We've only seen 920-ish for reads with SDR, 850-ish for writes.

The tests you are running.  Disktest does big linear reads/writes to
its iscsi block device?  If you stay within the first 1GB for both
test setups you describe, I would expect the same performance.  But
if you wander up closer to the 3.5 GB area for the larger ram disk,
the Linux VM will probably start pushing some of the older pages out
to swap.  It likes to keep some amount of memory free.

If you just want to torture the network, you can edit bs_sync.c to
stub out the pread64 and pwrite64 calls, and just set ret to
cmd->len.  Of course, hopefully disktest doesn't ever look at the
read results.  :)  Maybe you could use scsi_debug on the target with
"fake_rw" to do the same thing.

> My init scripts on target and client are available here:
> http://www.nejc.net/diploma/tgtd/

Scripts look fine.  Is it necessary to turn off HeaderDigest on the
initiator side?  I had hoped that iscsiadm would realize it was
asked to use iser and turn off HeaderDigest and DataDigest on its
own, but never got around to testing that.

		-- Pete