[Stgt-devel] Tuning iSER for performance

[Erez Zilber]

Bart Van Assche wrote:
> On Feb 12, 2008 11:06 AM, Erez Zilber <erezz at voltaire.com> wrote:
>   
>>> The results I obtained with STGT over iSER on an SDR network are as
>>> follows (direct I/O with  xdd):
>>> * 7.4 MB/s for reading data in blocks of 512 bytes and 8.9 MB/s for writing.
>>> * 363 MB/s for reading data in blocks of 65536 bytes and 346 MB/s for writing.
>>> * 890 MB/s for reading data in blocks of 10 MB and 900 MB/s for writing.
>>>
>>> Please keep in mind that when running a filesystem on top of iSCSI
>>> that all transfers happen in units of 64 KB or less.
>>>       
>> Can you send more details about your configuration?
>>
>>     * stgt version
>>     * initiator version
>>     * Which distro do you use for the initiator/target?
>>     * Which benchmark do you use? What are the parameters?
>>     * What are the results if you run the same benchmark directly from
>>       the target machine (i.e. no iSCSI & iSER involved)?
>>     * Are you using the default config for the initiator/target?
>>
>> Here's my config:
>>
>>     * target:
>>           o tgt-20071227 (actually, I use the RPM that I uploaded to OpenFabrics, which is almost identical to tgt-20071227)
>>           o SLES 10
>>           o default stgt config
>>     * initiator:
>>           o OFED 1.3 rc4
>>           o SLES 10 sp1
>>           o default open-iscsi config
>>     
>
> My config was as follows (same software versions on initiator and target):
> * Ubuntu 7.10 server.
> * Linux kernel 2.6.23.14 + SCST kernel patches (SVN revision 253).
> * OFED 1.2.5.5 userspace components, compiled from the OFED source distribution.
> * tgt-20071227 + TCP_NODELAY patch
> * Ubuntu's open-iscsi (version 2.0.865-1).
> * xdd version xdd65.013007.
> * Benchmark (with e.g. disk=/dev/sde and i in the range 9 .. 30): xdd
> -verbose -processlock -dio -op read -targets 1 $disk -reqsize 1
> -blocksize $((2**i)) -mbytes 2048 -passes 3;
> xdd -verbose -processlock -dio -op write -targets 1 $disk -reqsize 1
> -blocksize $((2**i)) -mbytes 2048 -passes 3
>   

I got these numbers:

                     T  Q       Bytes      Ops    Time      Rate     
IOPS   Latency     %CPU  OP_Type    ReqSize
TARGET   PASS0001    0  1    2147483648   4194304   315.311    
6.811     13302.12    0.0001     0.00   read         512
TARGET   PASS0002    0  1    2147483648   4194304   315.902    
6.798     13277.24    0.0001     0.00   read         512
TARGET   PASS0003    0  1    2147483648   4194304   315.539    
6.806     13292.49    0.0001     0.00   read         512
TARGET   Average     0  1    6442450944   12582912   946.752    
6.805     13290.61    0.0001     0.00   read         512
         Combined    1  1    6442450944   12582912   946.752    
6.805     13290.61    0.0001     0.00   read         512

from running the following cmd:

bin/xdd.linux -verbose -processlock -dio -op read -targets 1 /dev/sdc
-reqsize 1 -blocksize $((2**9)) -mbytes 2048 -passes 3

I'm not familiar enough with xdd, so I don't know how to tune it. Can
you try the following commands (assuming that /dev/sg4 is your sg device)?

READ: sgp_dd if=/dev/sg4 of=/dev/null bs=512 bpt=1024 count=20480000
thr=8 time=1
WRITE: sgp_dd of=/dev/sg4 if=/dev/zero bs=512 bpt=1024 count=20480000
thr=8 time=1

I ran that on a 4Gb LUN and got 400 MB/sec (wire speed).
 
> * Target setup:
> modprobe ib_uverbs
> modprobe rdma_ucm
> /bin/mkdir -p /dev/infiniband
> /bin/mknod /dev/infiniband/uverbs0 c $(cat
> /sys/class/infiniband_verbs/uverbs0/dev | sed 's/:/ /g')
> /bin/mknod /dev/infiniband/rdma_cm c $(cat /sys/class/misc/rdma_cm/dev
> | sed 's/:/ /g')
> tgtd
> tgtadm --lld iscsi --op new --mode target --tid 1 -T
> iqn.2007-05.com.example:storage.disk2.sys1.xyz
> tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /dev/ram0
> tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL
>   

Instead of running that manually, you can try to use the tgt-setup-lun
script that we added. It creates a targets, assigns LUNs & initiators.

> * Initiator setup:
> iscsi_target_ip=192.168.102.10
> rm -rf /etc/iscsi/nodes /etc/iscsi/send_targets
> iscsiadm -m discovery -t sendtargets -p ${iscsi_target_ip}
> iscsiadm --mode node --targetname
> iqn.2007-05.com.example:storage.disk2.sys1.xyz \
>   --portal ${iscsi_target_ip}:3260 --op update -n node.transport_name -v iser
> iscsiadm --mode node --targetname
> iqn.2007-05.com.example:storage.disk2.sys1.xyz \
>   --portal ${iscsi_target_ip}:3260 --op update -n
> "node.conn[0].iscsi.HeaderDigest" -v None
>   

You can use the iscsi_discovery script instead. It does all of the above
automatically. If you install OFED with iSER support, it is included in
the open-iscsi package.

> iscsiadm -m node -T iqn.2007-05.com.example:storage.disk2.sys1.xyz -p
> ${iscsi_target_ip} --login
> * Results for a local run of the benchmark (!! using buffered I/O
> instead of direct I/O !! because /dev/ram0 does not support direct
> I/O):
> - 354 MB/s for reading data in blocks of 512 bytes and 303 MB/s for writing.
> - 2514 MB/s for reading data in blocks of 65536 bytes and 2582 MB/s for writing.
> - 1400 MB/s for reading data in blocks of 10 MB and 1690 MB/s for writing.
>
> Bart Van Assche.
>