[Stgt-devel] iSER tgtd performance

[Jernej Porenta]

On 10/10/07, Pete Wyckoff <pw at osc.edu> wrote:
>
> Apparently you have DDR IB cards.  1030 MB/s is quite amazing.
> We've only seen 920-ish for reads with SDR, 850-ish for writes.
>
> The tests you are running.  Disktest does big linear reads/writes to
> its iscsi block device?  If you stay within the first 1GB for both
> test setups you describe, I would expect the same performance.  But
> if you wander up closer to the 3.5 GB area for the larger ram disk,
> the Linux VM will probably start pushing some of the older pages out
> to swap.  It likes to keep some amount of memory free.
>

Yes I have DDR IB cards (and I figured out that that might be the issue).

I made some additional tests with disktest
(http://www.open-iscsi.org/bits/disktest.tar.gz). I have added 4GB RAM
to target machine so now it has 8GB and created several ramdisks and
testing bandwidth throughput with several clients on same client
machine. Here are the results:
- 2GB target: 2 clients - each 280MB/s, 3 clients - each 215MB/s
- 1GB target: 1 client - 900MB/s, 2 clients - each 350MB/s, 3 clients 238MB/s

each client had different LUN target.

I have also made some tests with target size 512MB but the results
were beyond 2.5GB/s so I believe that there must be something wrong
with disktest or Linux buffers screw up the test.

I am planning to do some testing with IOzone or bonnie++, so results
might follow...

I use disktest like that:
./disktest -PT -T120 -pr -h1 -K1 -B256K -IB /dev/sdb

I also tried testing target from 2 different machines:
- 3GB target: 1 client - 440MB/s, 2 clients - each 420MB/s
- 2GB target: 1 client - 470MB/s, 2 clients - each 470MB/s
- 1GB target 1 client - 900MB/s, 2 clients - each 850MB/s

I assume that there might be some IO buffers somewhere, so I assume
that normal speed would be around 450MB/s. Same results Robin got:
https://lists.berlios.de/pipermail/stgt-devel/2007-October/001063.html

>
> > My init scripts on target and client are available here:
> > http://www.nejc.net/diploma/tgtd/
>
> Scripts look fine.  Is it necessary to turn off HeaderDigest on the
> initiator side?  I had hoped that iscsiadm would realize it was
> asked to use iser and turn off HeaderDigest and DataDigest on its
> own, but never got around to testing that.

I had some troubles with HeaderDigest at the beginning so I turned it
to none and it stayed in my scripts. I tried it now and it is not
working without it

I have also tried tgtd with iSER with TCP connection over IPoIB but
disktest reported read errors. From the begining everything went just
fine, but afterwards the read requests started to fail and the
performance dropped significantly, so I tried iSCSI enterprise target
software for TCP and it worked well, so I believe that there might be
a bug in tgtd software but I am unable to reproduce it, because if I
turn on the debug option, the performance drops because of 100% CPU
usage on the target. And believe it or not, everything works if CPU
usage on target is 100%, so there are some issues with highspeed tgtd
IPoIB target :)

Again thanks for your help...

cheers from Slovenia, Jernej