[Stgt-devel] Tuning iSER for performance

[Bart Van Assche]

On Feb 12, 2008 11:06 AM, Erez Zilber <erezz at voltaire.com> wrote:
> > The results I obtained with STGT over iSER on an SDR network are as
> > follows (direct I/O with  xdd):
> > * 7.4 MB/s for reading data in blocks of 512 bytes and 8.9 MB/s for writing.
> > * 363 MB/s for reading data in blocks of 65536 bytes and 346 MB/s for writing.
> > * 890 MB/s for reading data in blocks of 10 MB and 900 MB/s for writing.
> >
> > Please keep in mind that when running a filesystem on top of iSCSI
> > that all transfers happen in units of 64 KB or less.
>
> Can you send more details about your configuration?
>
>     * stgt version
>     * initiator version
>     * Which distro do you use for the initiator/target?
>     * Which benchmark do you use? What are the parameters?
>     * What are the results if you run the same benchmark directly from
>       the target machine (i.e. no iSCSI & iSER involved)?
>     * Are you using the default config for the initiator/target?
>
> Here's my config:
>
>     * target:
>           o tgt-20071227 (actually, I use the RPM that I uploaded to OpenFabrics, which is almost identical to tgt-20071227)
>           o SLES 10
>           o default stgt config
>     * initiator:
>           o OFED 1.3 rc4
>           o SLES 10 sp1
>           o default open-iscsi config

My config was as follows (same software versions on initiator and target):
* Ubuntu 7.10 server.
* Linux kernel 2.6.23.14 + SCST kernel patches (SVN revision 253).
* OFED 1.2.5.5 userspace components, compiled from the OFED source distribution.
* tgt-20071227 + TCP_NODELAY patch
* Ubuntu's open-iscsi (version 2.0.865-1).
* xdd version xdd65.013007.
* Benchmark (with e.g. disk=/dev/sde and i in the range 9 .. 30): xdd
-verbose -processlock -dio -op read -targets 1 $disk -reqsize 1
-blocksize $((2**i)) -mbytes 2048 -passes 3;
xdd -verbose -processlock -dio -op write -targets 1 $disk -reqsize 1
-blocksize $((2**i)) -mbytes 2048 -passes 3
* Target setup:
modprobe ib_uverbs
modprobe rdma_ucm
/bin/mkdir -p /dev/infiniband
/bin/mknod /dev/infiniband/uverbs0 c $(cat
/sys/class/infiniband_verbs/uverbs0/dev | sed 's/:/ /g')
/bin/mknod /dev/infiniband/rdma_cm c $(cat /sys/class/misc/rdma_cm/dev
| sed 's/:/ /g')
tgtd
tgtadm --lld iscsi --op new --mode target --tid 1 -T
iqn.2007-05.com.example:storage.disk2.sys1.xyz
tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /dev/ram0
tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL
* Initiator setup:
iscsi_target_ip=192.168.102.10
rm -rf /etc/iscsi/nodes /etc/iscsi/send_targets
iscsiadm -m discovery -t sendtargets -p ${iscsi_target_ip}
iscsiadm --mode node --targetname
iqn.2007-05.com.example:storage.disk2.sys1.xyz \
  --portal ${iscsi_target_ip}:3260 --op update -n node.transport_name -v iser
iscsiadm --mode node --targetname
iqn.2007-05.com.example:storage.disk2.sys1.xyz \
  --portal ${iscsi_target_ip}:3260 --op update -n
"node.conn[0].iscsi.HeaderDigest" -v None
iscsiadm -m node -T iqn.2007-05.com.example:storage.disk2.sys1.xyz -p
${iscsi_target_ip} --login
* Results for a local run of the benchmark (!! using buffered I/O
instead of direct I/O !! because /dev/ram0 does not support direct
I/O):
- 354 MB/s for reading data in blocks of 512 bytes and 303 MB/s for writing.
- 2514 MB/s for reading data in blocks of 65536 bytes and 2582 MB/s for writing.
- 1400 MB/s for reading data in blocks of 10 MB and 1690 MB/s for writing.

Bart Van Assche.