[Stgt-devel] read << write

[FUJITA Tomonori]

On Wed, 5 Dec 2007 08:56:09 -0500
Robin Humble <robin.humble+stgt at anu.edu.au> wrote:

> Hi,
> 
> I'm finding that reads are a lot slower than writes when I have a real
> file or device behind tgt instead of a ramdisk. is this expected?
> 
> iSER reads backed by a file on lustre or a md raid0 device seem to be at
> most ~100MB/s which is 4 or 5 times slower than writes:
> 
> bonnie++
>  Version  1.03    ------Sequential Output------ --Sequential Input- --Random-
>                   -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
>  Machine     Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
>  iSER/ramdisk  4G 79699  98 458229 96 260343 38 87650  98 474552 32 +++++ +++
>  iSER/md0      4G 79479  98 341628 68  96554 13 86280  96  64672  4 788.0   0
>  iSER/lustre   4G 78691  97 484972 90 131885 18 82747  93  98331  6 236.8   0
>                              ^^^                            ^^^
>  IPoIB/lustre  4G 76653  96 225736 50  71555  9 67401  90  91115  4 241.6   0 
>  GigE/lustre   4G 75556  96 115951 26  46772  5 64280  84  93495  5 247.3   0
>  native lustre 4G 69852  98 367647 87  62090 38 75893  93 333886 98 272.0   1
>  native md0                 309000                        335000
> 
> where:
>  iSER/ramdisk  - iSER to a 7G ramdisk
>  iSER/md0      - iSER to a software raid0 device
>  iSER/lustre   - iSER to a file on a Lustre fs client (also using IB)
>  IPoIB/lustre  - iSCSI over IP-over-IB to a file on a Lustre fs
>  GigE/lustre   - iSCSI over gigE to a file on a Lustre fs
>  native lustre - to native lustre
>  native md0    - dd speeds to /dev/md0
> 
> the trend of reads being limited to ~100MB/s seems to continue with
> normal iSCSI IPoIB and with TCP over gigE, which is odd.
> 
> perhaps I didn't see this before because my filesystems were too slow...??

Can you try a block-level benchmark like disktest to avoid file
systems effects?