[stgt] Worker Threads

Mon Sep 26 22:20:22 CEST 2011

Jeffrey,

Our network is 10Gbit end to end not 1Gbit as you mentioned earlier.

Regarding latency, I personally feel something is off with what the UCS 
network cards are rated for and what Linux ends up showing:

[root at storage02 mnt]# ping 172.16.1.100
PING 172.16.1.100 (172.16.1.100) 56(84) bytes of data.
64 bytes from 172.16.1.100: icmp_seq=1 ttl=64 time=0.185 ms
64 bytes from 172.16.1.100: icmp_seq=2 ttl=64 time=0.156 ms
64 bytes from 172.16.1.100: icmp_seq=3 ttl=64 time=0.158 ms
64 bytes from 172.16.1.100: icmp_seq=4 ttl=64 time=0.154 ms
64 bytes from 172.16.1.100: icmp_seq=5 ttl=64 time=0.151 ms
64 bytes from 172.16.1.100: icmp_seq=6 ttl=64 time=0.153 ms
^C
--- 172.16.1.100 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5429ms
rtt min/avg/max/mdev = 0.151/0.159/0.185/0.017 ms

(More relevant) 

[root at storage02 ~]# netperf -H 172.16.1.100 -t TCP_SENDFILE  -F 100M -C -c 
-l 60  -- -m 16M -s 16M -S 16M
TCP SENDFILE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.1.100 
(172.16.1.100) port 0 AF_INET
Recv   Send    Send                          Utilization       Service 
Demand
Socket Socket  Message  Elapsed              Send     Recv     Send Recv
Size   Size    Size     Time     Throughput  local    remote   local 
remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB us/KB

33554432 33554432 16777216    60.03      9259.27   1.14     3.27     0.242 
  0.695 

[root at storage02 ~]# modinfo enic
filename: 
/lib/modules/2.6.32-131.0.15.el6.x86_64/kernel/drivers/net/enic/enic.ko
version:        2.1.1.13
license:        GPL
author:         Scott Feldman <scofeldm at cisco.com>
description:    Cisco VIC Ethernet NIC Driver
srcversion:     EB6B26B5EEA5A1C378A1BD6
alias:          pci:v00001137d00000044sv*sd*bc*sc*i*
alias:          pci:v00001137d00000043sv*sd*bc*sc*i*
depends: 
vermagic:       2.6.32-131.0.15.el6.x86_64 SMP mod_unload modversions 

Reading the product specs for this card, it is rated for 10-12 us, yet we 
are seeing latency of 180us+, thoughts ?  Does anyone else on a 10Gbit 
local network experience this kind of latency ?

Is there a general rule of thumb to determine how latency is going to 
affect IO with iSCSI ?

--

Greg Procunier
UNIX Administrator III - Enterprise Servers and Storage
1 Robert Speck Parkway, Suite 400, Mississauga, Ontario L4Z 4E7
Office: 416-673-3320 
Mobile: 647-895-2977
Email: gprocunier at symcor.com

"Bennett, Jeffrey" <jab at sdsc.edu> wrote on 09/26/2011 04:11:15 PM:

> From: "Bennett, Jeffrey" <jab at sdsc.edu>
> To: "GProcunier at symcor.com" <GProcunier at symcor.com>, FUJITA Tomonori
> <fujita.tomonori at lab.ntt.co.jp>
> Cc: "agrover at redhat.com" <agrover at redhat.com>, 
> "fujita.tomonori at lab.ntt.co.jp" <fujita.tomonori at lab.ntt.co.jp>, 
> "stgt at vger.kernel.org" <stgt at vger.kernel.org>, "stgt-
> owner at vger.kernel.org" <stgt-owner at vger.kernel.org>
> Date: 09/26/2011 04:11 PM
> Subject: RE: Worker Threads
> 
> Thanks Greg, you did a great job documenting all of your testing.
> 
> We have noticed a great improvement in random performance (4kb 
> blocksize) over iSER when using 128 worker threads in comparison 
> with 4 worker threads. I don't have the numbers in front of me but 
> something around 200.000 random read IOPS and 4GB/sec is what we see
> in our system using 16 flash drives and QDR IB.
> 
> It seems you are using 1Gbit network so maybe in your particular 
> case, given the latency of Ethernet-based network, increasing the 
> worker threads does not help with random i/o, even when using a ramdisk.
> 
> jab
> 
> -----Original Message-----
> From: GProcunier at symcor.com [mailto:GProcunier at symcor.com] 
> Sent: Monday, September 26, 2011 1:02 PM
> To: FUJITA Tomonori
> Cc: agrover at redhat.com; fujita.tomonori at lab.ntt.co.jp; Bennett, 
> Jeffrey; stgt at vger.kernel.org; stgt-owner at vger.kernel.org
> Subject: Re: Worker Threads
> 
> Few additional things.
> 
> 1) my apologies, lotus notes is the devil and did all sorts of 
> terrible things to the formatting of initial post when viewed on 
> http://lists.wpkg.org/pipermail/stgt/2011-September/004782.html
> 
> 2) I have yet to see anyone post a detailed setup / results showing 
> 1000+ MB/s throughput between open-iSCSI and stgt, so hopefully this
> helps people.
> 
> 3) I am surprised the iops are so low for small block tests, kind of
> makes you wonder about the validity of the splash page of http://
> www.open-iscsi.org/ which boasts 50k+ iops with small block sizes. 
> Using a ramdisk eliminates the storage as a contention point so 
> really all that is left is the initiator and the target.
> 
> 4) tgt-admin needs some serious updating/work/rewrite, missing many 
> flags / cant add nullio devices via it etc etc.  My patch for 
> bsflags is the tip of the iceberg as far as problems with that tool.
> 
> Cheers,
> 
> 
> --
> 
> Greg Procunier
> UNIX Administrator III - Enterprise Servers and Storage
> 1 Robert Speck Parkway, Suite 400, Mississauga, Ontario L4Z 4E7
> Office: 416-673-3320
> Mobile: 647-895-2977
> Email: gprocunier at symcor.com
> 
> 
> 
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 10.0.1410 / Virus Database: 1520/3920 - Release Date: 09/26/11

-------------- next part --------------

CONFIDENTIALITY WARNING 
This communication, including any attachments, is for the exclusive use of addressee and may contain proprietary and/or confidential information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this communication and destroy all copies.

AVERTISSEMENT RELATIF ? LA CONFIDENTIALIT? 
Ce message, ainsi que les pi?ces qui y sont jointes, est destin? ? l?usage exclusif de la personne ? laquelle il s?adresse et peut contenir de l?information personnelle ou confidentielle. Si le lecteur de ce message n?en est pas le destinataire, nous l?avisons par la pr?sente que toute diffusion, distribution, reproduction ou utilisation de son contenu est strictement interdite. Veuillez avertir sur-le-champ l?exp?diteur par retour de courrier ?lectronique et supprimez ce message ainsi que toutes les pi?ces jointes.