[stgt] Worker Threads
jab at sdsc.edu
Mon Sep 26 22:32:12 CEST 2011
I am no networking guru, but this is what I see on our 10gb network, from host to switch to host:
64 bytes from 192.168.230.95: icmp_seq=1 ttl=63 time=0.197 ms
64 bytes from 192.168.230.95: icmp_seq=2 ttl=63 time=0.154 ms
64 bytes from 192.168.230.95: icmp_seq=3 ttl=63 time=0.173 ms
64 bytes from 192.168.230.95: icmp_seq=4 ttl=63 time=0.162 ms
64 bytes from 192.168.230.95: icmp_seq=5 ttl=63 time=0.166 ms
So either both of us have faulty 10gbe networks or both are fine :)
Hope this helps
PS. Regarding the random I/O performance, one thing I noticed is that when increasing the block size on tgtd.conf, the sequential i/o went up but the random i/o went down. Actually, out-of-the-box settings work very well for us and give us a good balance between seq. and random.
From: GProcunier at symcor.com [mailto:GProcunier at symcor.com]
Sent: Monday, September 26, 2011 1:20 PM
To: Bennett, Jeffrey
Cc: agrover at redhat.com; FUJITA Tomonori; stgt at vger.kernel.org; stgt-owner at vger.kernel.org
Subject: RE: Worker Threads
Our network is 10Gbit end to end not 1Gbit as you mentioned earlier.
Regarding latency, I personally feel something is off with what the UCS network cards are rated for and what Linux ends up showing:
[root at storage02 mnt]# ping 172.16.1.100
PING 172.16.1.100 (172.16.1.100) 56(84) bytes of data.
64 bytes from 172.16.1.100: icmp_seq=1 ttl=64 time=0.185 ms
64 bytes from 172.16.1.100: icmp_seq=2 ttl=64 time=0.156 ms
64 bytes from 172.16.1.100: icmp_seq=3 ttl=64 time=0.158 ms
64 bytes from 172.16.1.100: icmp_seq=4 ttl=64 time=0.154 ms
64 bytes from 172.16.1.100: icmp_seq=5 ttl=64 time=0.151 ms
64 bytes from 172.16.1.100: icmp_seq=6 ttl=64 time=0.153 ms ^C
--- 172.16.1.100 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5429ms rtt min/avg/max/mdev = 0.151/0.159/0.185/0.017 ms
[root at storage02 ~]# netperf -H 172.16.1.100 -t TCP_SENDFILE -F 100M -C -c -l 60 -- -m 16M -s 16M -S 16M TCP SENDFILE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.1.100
(172.16.1.100) port 0 AF_INET
Recv Send Send Utilization Service
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local
bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB
33554432 33554432 16777216 60.03 9259.27 1.14 3.27 0.242
[root at storage02 ~]# modinfo enic
author: Scott Feldman <scofeldm at cisco.com>
description: Cisco VIC Ethernet NIC Driver
vermagic: 2.6.32-131.0.15.el6.x86_64 SMP mod_unload modversions
Reading the product specs for this card, it is rated for 10-12 us, yet we are seeing latency of 180us+, thoughts ? Does anyone else on a 10Gbit local network experience this kind of latency ?
Is there a general rule of thumb to determine how latency is going to affect IO with iSCSI ?
UNIX Administrator III - Enterprise Servers and Storage
1 Robert Speck Parkway, Suite 400, Mississauga, Ontario L4Z 4E7
Email: gprocunier at symcor.com
"Bennett, Jeffrey" <jab at sdsc.edu> wrote on 09/26/2011 04:11:15 PM:
> From: "Bennett, Jeffrey" <jab at sdsc.edu>
> To: "GProcunier at symcor.com" <GProcunier at symcor.com>, FUJITA Tomonori
> <fujita.tomonori at lab.ntt.co.jp>
> Cc: "agrover at redhat.com" <agrover at redhat.com>,
> "fujita.tomonori at lab.ntt.co.jp" <fujita.tomonori at lab.ntt.co.jp>,
> "stgt at vger.kernel.org" <stgt at vger.kernel.org>, "stgt-
> owner at vger.kernel.org" <stgt-owner at vger.kernel.org>
> Date: 09/26/2011 04:11 PM
> Subject: RE: Worker Threads
> Thanks Greg, you did a great job documenting all of your testing.
> We have noticed a great improvement in random performance (4kb
> blocksize) over iSER when using 128 worker threads in comparison with
> 4 worker threads. I don't have the numbers in front of me but
> something around 200.000 random read IOPS and 4GB/sec is what we see
> in our system using 16 flash drives and QDR IB.
> It seems you are using 1Gbit network so maybe in your particular case,
> given the latency of Ethernet-based network, increasing the worker
> threads does not help with random i/o, even when using a ramdisk.
> -----Original Message-----
> From: GProcunier at symcor.com [mailto:GProcunier at symcor.com]
> Sent: Monday, September 26, 2011 1:02 PM
> To: FUJITA Tomonori
> Cc: agrover at redhat.com; fujita.tomonori at lab.ntt.co.jp; Bennett,
> Jeffrey; stgt at vger.kernel.org; stgt-owner at vger.kernel.org
> Subject: Re: Worker Threads
> Few additional things.
> 1) my apologies, lotus notes is the devil and did all sorts of
> terrible things to the formatting of initial post when viewed on
> 2) I have yet to see anyone post a detailed setup / results showing
> 1000+ MB/s throughput between open-iSCSI and stgt, so hopefully this
> helps people.
> 3) I am surprised the iops are so low for small block tests, kind of
> makes you wonder about the validity of the splash page of http://
> www.open-iscsi.org/ which boasts 50k+ iops with small block sizes.
> Using a ramdisk eliminates the storage as a contention point so really
> all that is left is the initiator and the target.
> 4) tgt-admin needs some serious updating/work/rewrite, missing many
> flags / cant add nullio devices via it etc etc. My patch for bsflags
> is the tip of the iceberg as far as problems with that tool.
> Greg Procunier
> UNIX Administrator III - Enterprise Servers and Storage
> 1 Robert Speck Parkway, Suite 400, Mississauga, Ontario L4Z 4E7
> Office: 416-673-3320
> Mobile: 647-895-2977
> Email: gprocunier at symcor.com
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 10.0.1410 / Virus Database: 1520/3920 - Release Date:
No virus found in this message.
Checked by AVG - www.avg.com
Version: 10.0.1410 / Virus Database: 1520/3920 - Release Date: 09/26/11
To unsubscribe from this list: send the line "unsubscribe stgt" in
the body of a message to majordomo at vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
More information about the stgt