[Stgt-devel] iSER

[Robin Humble]

On Tue, Sep 04, 2007 at 04:19:12PM -0400, Pete Wyckoff wrote:
>I haven't done much testing of throughput vs packet size lately, and
>in fact, ripping out the unsignaled completions may have slowed
>things down some.  But they were a mess as far as tracking RDMA
>resources went.  Would love some help with tuning if you are so
>inclined.

the good news:

I'm not seeing any of the iSER resource starvation/flow control
problems I was seeing before. with latest git://git.osc.edu/tgt that
looks to be fixed.

so, trying out a few speed tests ->

bypassing the page cache (and readahead?) with O_DIRECT:
 eg. dd if=/dev/zero of=/dev/sdc bs=1k count=8000 oflag=direct
   bs   write MB/s read MB/s
  10M     1200      520
   1M      790      460
 200k      480      350
   4k       40       34
   1k       11        9
large writes look fabulous, but reads seem to be limited by something
other than IB bandwidth.

in the more usual usage case via the page cache:
 eg. dd if=/dev/zero of=/dev/sdc bs=1k count=8000000 
   bs   write MB/s read MB/s
  10M     1100      260
   1M     1100      270
   4k      960      270
   1k       30      240
so maybe extra copies to/from page cache are getting in the way of the
read bandwidth and are lowering it by a factor of 2.
I'm guessing the good small block read performance here is due to
readahead, and the mostly better writes are from aggregation.

setup is DDR IB, 2.6.22.6 kernel, centos5 x86_64 with ofed 1.2
userland, server with 7G ramfs backing store, single client with
mem=512M.

the bad news:

if I make an ext3 filesystem on the iSER device and then dd a few G to
a file in the filesystem then I very quickly get an ext3 shutdown.
  Sep  5 23:06:03 x11 kernel: EXT3-fs error (device sdc): ext3_new_block: Allocating block in system zone - blocks from 393216, length 1
  Sep  5 23:06:03 x11 kernel: Aborting journal on device sdc.
  Sep  5 23:06:03 x11 kernel: ext3_abort called.
  Sep  5 23:06:03 x11 kernel: EXT3-fs error (device sdc): ext3_journal_start_sb: Detected aborted journal
  Sep  5 23:06:03 x11 kernel: Remounting filesystem read-only
  Sep  5 23:06:03 x11 kernel: EXT3-fs error (device sdc): ext3_free_blocks: Freeing blocks in system zones - Block = 393216, count = 1
  Sep  5 23:06:03 x11 kernel: EXT3-fs error (device sdc) in ext3_free_blocks_sb: Journal has aborted
  Sep  5 23:06:03 x11 kernel: __journal_remove_journal_head: freeing b_committed_data

if I leave out the iSER setup steps and use IPoIB instead then it works ok.
I tried with a centos kernel and ofed1.1, and also 2.6.22.6 and ofed1.2.

the above error message is from the centos kernel. 2.6.22.6 generates these:
  Sep  5 23:48:33 x11 kernel: EXT3-fs error (device sdc): ext3_new_block: Allocating block in system zone - blocks from 164279, length 1
  Sep  5 23:48:33 x11 kernel: EXT3-fs error (device sdc): ext3_new_block: Allocating block in system zone - blocks from 164280, length 1
  Sep  5 23:48:33 x11 kernel: EXT3-fs error (device sdc): ext3_new_block: Allocating block in system zone - blocks from 164281, length 1
  Sep  5 23:48:33 x11 kernel: EXT3-fs error (device sdc): ext3_new_block: Allocating block in system zone - blocks from 164282, length 1
  Sep  5 23:48:33 x11 kernel: EXT3-fs error (device sdc): ext3_new_block: Allocating block in system zone - blocks from 164283, length 1
  Sep  5 23:48:33 x11 kernel: EXT3-fs error (device sdc): ext3_new_block: Allocating block in system zone - blocks from 164284, length 1
  Sep  5 23:48:33 x11 kernel: EXT3-fs error (device sdc): ext3_new_block: Allocating block in system zone - blocks from 164285, length 1
  ... and many more

which looks a bit like a miscommunication of the size/geometry of the
device? or maybe corruption? :-/ the dd's were all smaller (few G) than
the size of the filesystem (6+G) so it wasn't running out of space.

cheers,
robin