[Stgt-devel] iSER

[Robin Humble]

On Thu, Oct 04, 2007 at 01:20:35PM -0400, Pete Wyckoff wrote:
>pw at osc.edu wrote on Sun, 09 Sep 2007 14:12 -0400:
>> robin.humble+stgt at anu.edu.au wrote on Sun, 09 Sep 2007 11:30 -0400:
>> > with the 2.6.22.6 kernel and iSER I couldn't find any corruption
>> > issues using dd to /dev/sdc. however (as reported previously) if I put
>> > an ext3 filesystem on the iSER device and then dd to a file in the ext3
>> > filsystem then pretty much immediately I get:
>> >   Sep  9 21:46:22 x11 kernel: EXT3-fs error (device sdc): ext3_new_block: Allocating block in system zone - blocks from 196611, length 1
>> >   Sep  9 21:46:22 x11 kernel: EXT3-fs error (device sdc): ext3_new_block: Allocating block in system zone - blocks from 196612, length 1
>> >   Sep  9 21:46:22 x11 kernel: EXT3-fs error (device sdc): ext3_new_block: Allocating block in system zone - blocks from 196613, length 1
>> >   ...
>> > I get the same type of errors with 2.6.23-rc5 too.

eventually (with Pete and Erez's offline help) I managed to reproduce
this without a filesystem by using sufficiently large and preferably
random lmdd's. eg.
  lmdd of=internal ipat=1 if=/dev/sdc bs=800000 count=2000 rand=6000m mismatch=10
however, that's kinda irrelevant 'cos...

>Maybe this is fixed.  I did find one possible case where the Send
>result may have gone out before the final RDMA write, in the case
>when the target is starved for RDMA slots.  But I never saw the
>problem myself, so can't say for sure.

... yes! that has fixed the problem.

now lmdd to the block device, or ext3 with multiple bonnie++'s or
iozone multithreaded etc. all work as expected.

well done finding the bug! :-)

performance is good. backing store is a 7G file on a ramfs, 2.6.22.6
kernels with OFED 1.2.5 modules and userland, dual dual core Xeon
2.66Ghz, CentOS5 x86_64, initiator with mem=512M, DDR IB.

~650MB/s writes and 280MB/s reads with eg.
  dd if=/dev/zero of=/dev/sdc bs=1M count=6000
 
bonnie++
 1.03 ------Sequential Output------ --Sequential Input- --Random-
      -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
 Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
   4G 79119  99 464995 98 171640 28 70948  81 291538 23 +++++ +++
   6G 79061  99 459926 93 165442 27 70987  80 291262 23 15929  41

so ~450 MiB/s writes and 280 MiB/s reads.
or bonnie++ in semaphore mode:
 2-way (-s 2g) min aggregate write/read of ~480/380 MiB/s
 4-way (-s 1g) min aggregate write/read of ~525/570 MiB/s

and about the same from iozone:
min throughput * threads from a single client doing iozone 1,2,4-way is:
 iozone -s 4g -r 256k -i 0 -i 1 -t 1
        write   =  481869.34 KB/sec
        read    =  293940.09 KB/sec
 iozone -s 2g -r 256k -i 0 -i 1 -t 2
        write   =  252719.98 KB/sec   * 2 = 505440 KB/s
        read    =  196478.64 KB/sec   * 2 = 392957 KB/s
 iozone -s 1g -r 256k -i 0 -i 1 -t 4
        write   =  132325.09 KB/sec   * 4 = 529300 KB/s
        read    =  136420.22 KB/sec   * 4 = 545681 KB/s

cheers,
robin