[stgt] Worker Threads

Mon Sep 26 22:32:12 CEST 2011

Hi Greg,

I am no networking guru, but this is what I see on our 10gb network, from host to switch to host:

64 bytes from 192.168.230.95: icmp_seq=1 ttl=63 time=0.197 ms
64 bytes from 192.168.230.95: icmp_seq=2 ttl=63 time=0.154 ms
64 bytes from 192.168.230.95: icmp_seq=3 ttl=63 time=0.173 ms
64 bytes from 192.168.230.95: icmp_seq=4 ttl=63 time=0.162 ms
64 bytes from 192.168.230.95: icmp_seq=5 ttl=63 time=0.166 ms

So either both of us have faulty 10gbe networks or both are fine :)

Hope this helps

PS. Regarding the random I/O performance, one thing I noticed is that when increasing the block size on tgtd.conf, the sequential i/o went up but the random i/o went down. Actually, out-of-the-box settings work very well for us and give us a good balance between seq. and random.

jab

-----Original Message-----
From: GProcunier at symcor.com [mailto:GProcunier at symcor.com] 
Sent: Monday, September 26, 2011 1:20 PM
To: Bennett, Jeffrey
Cc: agrover at redhat.com; FUJITA Tomonori; stgt at vger.kernel.org; stgt-owner at vger.kernel.org
Subject: RE: Worker Threads

Jeffrey,

Our network is 10Gbit end to end not 1Gbit as you mentioned earlier.

Regarding latency, I personally feel something is off with what the UCS network cards are rated for and what Linux ends up showing:

[root at storage02 mnt]# ping 172.16.1.100
PING 172.16.1.100 (172.16.1.100) 56(84) bytes of data.
64 bytes from 172.16.1.100: icmp_seq=1 ttl=64 time=0.185 ms
64 bytes from 172.16.1.100: icmp_seq=2 ttl=64 time=0.156 ms
64 bytes from 172.16.1.100: icmp_seq=3 ttl=64 time=0.158 ms
64 bytes from 172.16.1.100: icmp_seq=4 ttl=64 time=0.154 ms
64 bytes from 172.16.1.100: icmp_seq=5 ttl=64 time=0.151 ms
64 bytes from 172.16.1.100: icmp_seq=6 ttl=64 time=0.153 ms ^C
--- 172.16.1.100 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5429ms rtt min/avg/max/mdev = 0.151/0.159/0.185/0.017 ms

(More relevant) 

[root at storage02 ~]# netperf -H 172.16.1.100 -t TCP_SENDFILE  -F 100M -C -c -l 60  -- -m 16M -s 16M -S 16M TCP SENDFILE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.1.100
(172.16.1.100) port 0 AF_INET
Recv   Send    Send                          Utilization       Service 
Demand
Socket Socket  Message  Elapsed              Send     Recv     Send Recv
Size   Size    Size     Time     Throughput  local    remote   local 
remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB us/KB

33554432 33554432 16777216    60.03      9259.27   1.14     3.27     0.242 
  0.695 

[root at storage02 ~]# modinfo enic
filename: 
/lib/modules/2.6.32-131.0.15.el6.x86_64/kernel/drivers/net/enic/enic.ko
version:        2.1.1.13
license:        GPL
author:         Scott Feldman <scofeldm at cisco.com>
description:    Cisco VIC Ethernet NIC Driver
srcversion:     EB6B26B5EEA5A1C378A1BD6
alias:          pci:v00001137d00000044sv*sd*bc*sc*i*
alias:          pci:v00001137d00000043sv*sd*bc*sc*i*
depends: 
vermagic:       2.6.32-131.0.15.el6.x86_64 SMP mod_unload modversions 

Reading the product specs for this card, it is rated for 10-12 us, yet we are seeing latency of 180us+, thoughts ?  Does anyone else on a 10Gbit local network experience this kind of latency ?

Is there a general rule of thumb to determine how latency is going to affect IO with iSCSI ?

--

Greg Procunier
UNIX Administrator III - Enterprise Servers and Storage
1 Robert Speck Parkway, Suite 400, Mississauga, Ontario L4Z 4E7
Office: 416-673-3320
Mobile: 647-895-2977
Email: gprocunier at symcor.com

"Bennett, Jeffrey" <jab at sdsc.edu> wrote on 09/26/2011 04:11:15 PM:

> From: "Bennett, Jeffrey" <jab at sdsc.edu>
> To: "GProcunier at symcor.com" <GProcunier at symcor.com>, FUJITA Tomonori 
> <fujita.tomonori at lab.ntt.co.jp>
> Cc: "agrover at redhat.com" <agrover at redhat.com>, 
> "fujita.tomonori at lab.ntt.co.jp" <fujita.tomonori at lab.ntt.co.jp>, 
> "stgt at vger.kernel.org" <stgt at vger.kernel.org>, "stgt- 
> owner at vger.kernel.org" <stgt-owner at vger.kernel.org>
> Date: 09/26/2011 04:11 PM
> Subject: RE: Worker Threads
> 
> Thanks Greg, you did a great job documenting all of your testing.
> 
> We have noticed a great improvement in random performance (4kb
> blocksize) over iSER when using 128 worker threads in comparison with 
> 4 worker threads. I don't have the numbers in front of me but 
> something around 200.000 random read IOPS and 4GB/sec is what we see 
> in our system using 16 flash drives and QDR IB.
> 
> It seems you are using 1Gbit network so maybe in your particular case, 
> given the latency of Ethernet-based network, increasing the worker 
> threads does not help with random i/o, even when using a ramdisk.
> 
> jab
> 
> -----Original Message-----
> From: GProcunier at symcor.com [mailto:GProcunier at symcor.com]
> Sent: Monday, September 26, 2011 1:02 PM
> To: FUJITA Tomonori
> Cc: agrover at redhat.com; fujita.tomonori at lab.ntt.co.jp; Bennett, 
> Jeffrey; stgt at vger.kernel.org; stgt-owner at vger.kernel.org
> Subject: Re: Worker Threads
> 
> Few additional things.
> 
> 1) my apologies, lotus notes is the devil and did all sorts of 
> terrible things to the formatting of initial post when viewed on 
> http://lists.wpkg.org/pipermail/stgt/2011-September/004782.html
> 
> 2) I have yet to see anyone post a detailed setup / results showing
> 1000+ MB/s throughput between open-iSCSI and stgt, so hopefully this
> helps people.
> 
> 3) I am surprised the iops are so low for small block tests, kind of 
> makes you wonder about the validity of the splash page of http:// 
> www.open-iscsi.org/ which boasts 50k+ iops with small block sizes.
> Using a ramdisk eliminates the storage as a contention point so really 
> all that is left is the initiator and the target.
> 
> 4) tgt-admin needs some serious updating/work/rewrite, missing many 
> flags / cant add nullio devices via it etc etc.  My patch for bsflags 
> is the tip of the iceberg as far as problems with that tool.
> 
> Cheers,
> 
> 
> --
> 
> Greg Procunier
> UNIX Administrator III - Enterprise Servers and Storage
> 1 Robert Speck Parkway, Suite 400, Mississauga, Ontario L4Z 4E7
> Office: 416-673-3320
> Mobile: 647-895-2977
> Email: gprocunier at symcor.com
> 
> 
> 
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 10.0.1410 / Virus Database: 1520/3920 - Release Date: 
> 09/26/11

-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 10.0.1410 / Virus Database: 1520/3920 - Release Date: 09/26/11
--
To unsubscribe from this list: send the line "unsubscribe stgt" in
the body of a message to majordomo at vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html