Hi Greg, I am no networking guru, but this is what I see on our 10gb network, from host to switch to host: 64 bytes from 192.168.230.95: icmp_seq=1 ttl=63 time=0.197 ms 64 bytes from 192.168.230.95: icmp_seq=2 ttl=63 time=0.154 ms 64 bytes from 192.168.230.95: icmp_seq=3 ttl=63 time=0.173 ms 64 bytes from 192.168.230.95: icmp_seq=4 ttl=63 time=0.162 ms 64 bytes from 192.168.230.95: icmp_seq=5 ttl=63 time=0.166 ms So either both of us have faulty 10gbe networks or both are fine :) Hope this helps PS. Regarding the random I/O performance, one thing I noticed is that when increasing the block size on tgtd.conf, the sequential i/o went up but the random i/o went down. Actually, out-of-the-box settings work very well for us and give us a good balance between seq. and random. jab -----Original Message----- From: GProcunier at symcor.com [mailto:GProcunier at symcor.com] Sent: Monday, September 26, 2011 1:20 PM To: Bennett, Jeffrey Cc: agrover at redhat.com; FUJITA Tomonori; stgt at vger.kernel.org; stgt-owner at vger.kernel.org Subject: RE: Worker Threads Jeffrey, Our network is 10Gbit end to end not 1Gbit as you mentioned earlier. Regarding latency, I personally feel something is off with what the UCS network cards are rated for and what Linux ends up showing: [root at storage02 mnt]# ping 172.16.1.100 PING 172.16.1.100 (172.16.1.100) 56(84) bytes of data. 64 bytes from 172.16.1.100: icmp_seq=1 ttl=64 time=0.185 ms 64 bytes from 172.16.1.100: icmp_seq=2 ttl=64 time=0.156 ms 64 bytes from 172.16.1.100: icmp_seq=3 ttl=64 time=0.158 ms 64 bytes from 172.16.1.100: icmp_seq=4 ttl=64 time=0.154 ms 64 bytes from 172.16.1.100: icmp_seq=5 ttl=64 time=0.151 ms 64 bytes from 172.16.1.100: icmp_seq=6 ttl=64 time=0.153 ms ^C --- 172.16.1.100 ping statistics --- 6 packets transmitted, 6 received, 0% packet loss, time 5429ms rtt min/avg/max/mdev = 0.151/0.159/0.185/0.017 ms (More relevant) [root at storage02 ~]# netperf -H 172.16.1.100 -t TCP_SENDFILE -F 100M -C -c -l 60 -- -m 16M -s 16M -S 16M TCP SENDFILE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.1.100 (172.16.1.100) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 33554432 33554432 16777216 60.03 9259.27 1.14 3.27 0.242 0.695 [root at storage02 ~]# modinfo enic filename: /lib/modules/2.6.32-131.0.15.el6.x86_64/kernel/drivers/net/enic/enic.ko version: 2.1.1.13 license: GPL author: Scott Feldman <scofeldm at cisco.com> description: Cisco VIC Ethernet NIC Driver srcversion: EB6B26B5EEA5A1C378A1BD6 alias: pci:v00001137d00000044sv*sd*bc*sc*i* alias: pci:v00001137d00000043sv*sd*bc*sc*i* depends: vermagic: 2.6.32-131.0.15.el6.x86_64 SMP mod_unload modversions Reading the product specs for this card, it is rated for 10-12 us, yet we are seeing latency of 180us+, thoughts ? Does anyone else on a 10Gbit local network experience this kind of latency ? Is there a general rule of thumb to determine how latency is going to affect IO with iSCSI ? -- Greg Procunier UNIX Administrator III - Enterprise Servers and Storage 1 Robert Speck Parkway, Suite 400, Mississauga, Ontario L4Z 4E7 Office: 416-673-3320 Mobile: 647-895-2977 Email: gprocunier at symcor.com "Bennett, Jeffrey" <jab at sdsc.edu> wrote on 09/26/2011 04:11:15 PM: > From: "Bennett, Jeffrey" <jab at sdsc.edu> > To: "GProcunier at symcor.com" <GProcunier at symcor.com>, FUJITA Tomonori > <fujita.tomonori at lab.ntt.co.jp> > Cc: "agrover at redhat.com" <agrover at redhat.com>, > "fujita.tomonori at lab.ntt.co.jp" <fujita.tomonori at lab.ntt.co.jp>, > "stgt at vger.kernel.org" <stgt at vger.kernel.org>, "stgt- > owner at vger.kernel.org" <stgt-owner at vger.kernel.org> > Date: 09/26/2011 04:11 PM > Subject: RE: Worker Threads > > Thanks Greg, you did a great job documenting all of your testing. > > We have noticed a great improvement in random performance (4kb > blocksize) over iSER when using 128 worker threads in comparison with > 4 worker threads. I don't have the numbers in front of me but > something around 200.000 random read IOPS and 4GB/sec is what we see > in our system using 16 flash drives and QDR IB. > > It seems you are using 1Gbit network so maybe in your particular case, > given the latency of Ethernet-based network, increasing the worker > threads does not help with random i/o, even when using a ramdisk. > > jab > > -----Original Message----- > From: GProcunier at symcor.com [mailto:GProcunier at symcor.com] > Sent: Monday, September 26, 2011 1:02 PM > To: FUJITA Tomonori > Cc: agrover at redhat.com; fujita.tomonori at lab.ntt.co.jp; Bennett, > Jeffrey; stgt at vger.kernel.org; stgt-owner at vger.kernel.org > Subject: Re: Worker Threads > > Few additional things. > > 1) my apologies, lotus notes is the devil and did all sorts of > terrible things to the formatting of initial post when viewed on > http://lists.wpkg.org/pipermail/stgt/2011-September/004782.html > > 2) I have yet to see anyone post a detailed setup / results showing > 1000+ MB/s throughput between open-iSCSI and stgt, so hopefully this > helps people. > > 3) I am surprised the iops are so low for small block tests, kind of > makes you wonder about the validity of the splash page of http:// > www.open-iscsi.org/ which boasts 50k+ iops with small block sizes. > Using a ramdisk eliminates the storage as a contention point so really > all that is left is the initiator and the target. > > 4) tgt-admin needs some serious updating/work/rewrite, missing many > flags / cant add nullio devices via it etc etc. My patch for bsflags > is the tip of the iceberg as far as problems with that tool. > > Cheers, > > > -- > > Greg Procunier > UNIX Administrator III - Enterprise Servers and Storage > 1 Robert Speck Parkway, Suite 400, Mississauga, Ontario L4Z 4E7 > Office: 416-673-3320 > Mobile: 647-895-2977 > Email: gprocunier at symcor.com > > > > ----- > No virus found in this message. > Checked by AVG - www.avg.com > Version: 10.0.1410 / Virus Database: 1520/3920 - Release Date: > 09/26/11 ----- No virus found in this message. Checked by AVG - www.avg.com Version: 10.0.1410 / Virus Database: 1520/3920 - Release Date: 09/26/11 -- To unsubscribe from this list: send the line "unsubscribe stgt" in the body of a message to majordomo at vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html |