Jeffrey, Our network is 10Gbit end to end not 1Gbit as you mentioned earlier. Regarding latency, I personally feel something is off with what the UCS network cards are rated for and what Linux ends up showing: [root at storage02 mnt]# ping 172.16.1.100 PING 172.16.1.100 (172.16.1.100) 56(84) bytes of data. 64 bytes from 172.16.1.100: icmp_seq=1 ttl=64 time=0.185 ms 64 bytes from 172.16.1.100: icmp_seq=2 ttl=64 time=0.156 ms 64 bytes from 172.16.1.100: icmp_seq=3 ttl=64 time=0.158 ms 64 bytes from 172.16.1.100: icmp_seq=4 ttl=64 time=0.154 ms 64 bytes from 172.16.1.100: icmp_seq=5 ttl=64 time=0.151 ms 64 bytes from 172.16.1.100: icmp_seq=6 ttl=64 time=0.153 ms ^C --- 172.16.1.100 ping statistics --- 6 packets transmitted, 6 received, 0% packet loss, time 5429ms rtt min/avg/max/mdev = 0.151/0.159/0.185/0.017 ms (More relevant) [root at storage02 ~]# netperf -H 172.16.1.100 -t TCP_SENDFILE -F 100M -C -c -l 60 -- -m 16M -s 16M -S 16M TCP SENDFILE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.1.100 (172.16.1.100) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 33554432 33554432 16777216 60.03 9259.27 1.14 3.27 0.242 0.695 [root at storage02 ~]# modinfo enic filename: /lib/modules/2.6.32-131.0.15.el6.x86_64/kernel/drivers/net/enic/enic.ko version: 2.1.1.13 license: GPL author: Scott Feldman <scofeldm at cisco.com> description: Cisco VIC Ethernet NIC Driver srcversion: EB6B26B5EEA5A1C378A1BD6 alias: pci:v00001137d00000044sv*sd*bc*sc*i* alias: pci:v00001137d00000043sv*sd*bc*sc*i* depends: vermagic: 2.6.32-131.0.15.el6.x86_64 SMP mod_unload modversions Reading the product specs for this card, it is rated for 10-12 us, yet we are seeing latency of 180us+, thoughts ? Does anyone else on a 10Gbit local network experience this kind of latency ? Is there a general rule of thumb to determine how latency is going to affect IO with iSCSI ? -- Greg Procunier UNIX Administrator III - Enterprise Servers and Storage 1 Robert Speck Parkway, Suite 400, Mississauga, Ontario L4Z 4E7 Office: 416-673-3320 Mobile: 647-895-2977 Email: gprocunier at symcor.com "Bennett, Jeffrey" <jab at sdsc.edu> wrote on 09/26/2011 04:11:15 PM: > From: "Bennett, Jeffrey" <jab at sdsc.edu> > To: "GProcunier at symcor.com" <GProcunier at symcor.com>, FUJITA Tomonori > <fujita.tomonori at lab.ntt.co.jp> > Cc: "agrover at redhat.com" <agrover at redhat.com>, > "fujita.tomonori at lab.ntt.co.jp" <fujita.tomonori at lab.ntt.co.jp>, > "stgt at vger.kernel.org" <stgt at vger.kernel.org>, "stgt- > owner at vger.kernel.org" <stgt-owner at vger.kernel.org> > Date: 09/26/2011 04:11 PM > Subject: RE: Worker Threads > > Thanks Greg, you did a great job documenting all of your testing. > > We have noticed a great improvement in random performance (4kb > blocksize) over iSER when using 128 worker threads in comparison > with 4 worker threads. I don't have the numbers in front of me but > something around 200.000 random read IOPS and 4GB/sec is what we see > in our system using 16 flash drives and QDR IB. > > It seems you are using 1Gbit network so maybe in your particular > case, given the latency of Ethernet-based network, increasing the > worker threads does not help with random i/o, even when using a ramdisk. > > jab > > -----Original Message----- > From: GProcunier at symcor.com [mailto:GProcunier at symcor.com] > Sent: Monday, September 26, 2011 1:02 PM > To: FUJITA Tomonori > Cc: agrover at redhat.com; fujita.tomonori at lab.ntt.co.jp; Bennett, > Jeffrey; stgt at vger.kernel.org; stgt-owner at vger.kernel.org > Subject: Re: Worker Threads > > Few additional things. > > 1) my apologies, lotus notes is the devil and did all sorts of > terrible things to the formatting of initial post when viewed on > http://lists.wpkg.org/pipermail/stgt/2011-September/004782.html > > 2) I have yet to see anyone post a detailed setup / results showing > 1000+ MB/s throughput between open-iSCSI and stgt, so hopefully this > helps people. > > 3) I am surprised the iops are so low for small block tests, kind of > makes you wonder about the validity of the splash page of http:// > www.open-iscsi.org/ which boasts 50k+ iops with small block sizes. > Using a ramdisk eliminates the storage as a contention point so > really all that is left is the initiator and the target. > > 4) tgt-admin needs some serious updating/work/rewrite, missing many > flags / cant add nullio devices via it etc etc. My patch for > bsflags is the tip of the iceberg as far as problems with that tool. > > Cheers, > > > -- > > Greg Procunier > UNIX Administrator III - Enterprise Servers and Storage > 1 Robert Speck Parkway, Suite 400, Mississauga, Ontario L4Z 4E7 > Office: 416-673-3320 > Mobile: 647-895-2977 > Email: gprocunier at symcor.com > > > > ----- > No virus found in this message. > Checked by AVG - www.avg.com > Version: 10.0.1410 / Virus Database: 1520/3920 - Release Date: 09/26/11 -------------- next part -------------- CONFIDENTIALITY WARNING This communication, including any attachments, is for the exclusive use of addressee and may contain proprietary and/or confidential information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this communication and destroy all copies. AVERTISSEMENT RELATIF ? LA CONFIDENTIALIT? Ce message, ainsi que les pi?ces qui y sont jointes, est destin? ? l?usage exclusif de la personne ? laquelle il s?adresse et peut contenir de l?information personnelle ou confidentielle. Si le lecteur de ce message n?en est pas le destinataire, nous l?avisons par la pr?sente que toute diffusion, distribution, reproduction ou utilisation de son contenu est strictement interdite. Veuillez avertir sur-le-champ l?exp?diteur par retour de courrier ?lectronique et supprimez ce message ainsi que toutes les pi?ces jointes. |