[Stgt-devel] read << write

[Pete Wyckoff]

robin.humble+stgt at anu.edu.au wrote on Thu, 06 Dec 2007 03:40 -0500:
> On Thu, Dec 06, 2007 at 07:54:53AM +0900, FUJITA Tomonori wrote:
> >On Wed, 5 Dec 2007 08:56:09 -0500 Robin Humble <robin.humble+stgt at anu.edu.au> wrote:
> >> I'm finding that reads are a lot slower than writes when I have a real
> >> file or device behind tgt instead of a ramdisk. is this expected?
> >> 
> >> iSER reads backed by a file on lustre or a md raid0 device seem to be at
> >> most ~100MB/s which is 4 or 5 times slower than writes:
> >Can you try a block-level benchmark like disktest to avoid file
> >systems effects?
> 
> I'm afraid I don't have enough experience with disktest to know when
> it's lying to me and/or when I'm driving it foolishly.
> 
> how about just large dd's?
> 
>                           write    read (MB/s)
>  iSER + /dev/md0           333     110
>  iSER + file on lustre     552     207
>  iSER + ramdisk            905     410
> 
>  local /dev/md0            313     330
>  local file on lustre      705     473
>  local ramdisk            1600    2900
> 
> which are eg.
>   dd if=/dev/zero of=/dev/sdc bs=1M count=10000
> where 10G is >> (512M ram on initiator + 512M ram on target) so
> buffering should be small.

Interesting, but I can not guess why.  My results to ram disk for
the default tgt and linux client config show reads at line rate (920
MB/s) but writes somewhat slower (850 MB/s).  It is important to get
multiple outstanding commands from the client point of view, and
perhaps dd is not doing that, nor is bonnie?

You might wander around /sys to make sure that iscsi is keeping
enough commands in flight (128 is default, plenty).  And try things
like sg_dd, sgp_dd, and disktest as Tomo suggests.

One of the big effects we noticed with speeds like this is that
caching effects on the target become significant.  Yes, normal L2
processor cache compared to memory speeds show up in a big way.

Here's some numbers and details on ramdisk performance from a
talk I gave at SNAPI, if it helps you with insight:

    http://www.osc.edu/~pw/talks/iser-snapi07/talk.pdf

You are lucky/cursed to have such good-performing disks.  Will be
interested to see if you find much.  Other tools of interest may be
blktrace and oprofile on the target.

		-- Pete