Executive summary: nr_iothread patch does not appear to meaningfully
increase performance
My environment:
Storage01/Storage02 servers:
Cisco UCS Server Blades with 192 GB of memory and 24 core Intel(R)
Xeon(R) CPU X5670 @ 2.93GHz running RHEL6.1
kernel 2.6.32-131.0.15.el6.x86_64
UCS service profiles have a 9000 byte MTU vNIC associated with
each machine.
Are set in the platinum traffic class of the UCS guarantee of 62%
backplane bandwidth.
tgt-1.0.19 + nr_iothread patch
both servers connect to each other through a Cisco 7k nexus
switch.
Both servers have a dedicated vNIC with 10GBit backplane access.
iozone was used for testing and modified to allow record sizes up
to 64MB (default is 16MB)
the test disk was formatted with ext4 (mkfs.ext4 -E
stride=16,stripe-width=64 /dev/mapper/mpathd)
the test disk was mounted on the client with -o nobarrier option
Overview of changes and testing:
Storage01 creates a 50GB ramdisk on boot and exports it as an iSCSI target
for clients to test with:
-- Excerpt from /etc/init.d/ramdisk
start() {
[ "$EUID" != "0" ] && exit 4
[ "$NETWORKING" = "no" ] && exit 1
# Start daemons.
echo -n $"Starting ramdisk: "
mount -t ramfs none /exports/ram0
dd if=/dev/zero of=/exports/ram0/ramdisk bs=1G
count=50
sync
losetup /dev/loop0 /exports/ram0/ramdisk
pvcreate /dev/loop0
vgcreate -c n ramdisk /dev/loop0
lvcreate -n ramdisk -l 100%VG ramdisk
RETVAL=$?
echo
return $RETVAL
}
Storage01 exports the ramdisk to several channels:
-- /etc/tgt/targets.conf
# Storage Cluster Node 01 Target Config
default-driver iscsi
# these params never actually work
MaxConnections 1
InitialR2T No
ImmediateData Yes
MaxRecvDataSegmentLength 16776192
MaxXmitDataSegmentLength 16776192
MaxBurstLength 16776192
FirstBurstLength 16776192
MaxOutstandingR2T 16
DataDigest None
HeaderDigest None
<target iqn.1970-01.lab.symcor.symcor-nas:perf.ramdisk>
incominguser perf test1234
backing-store /dev/ramdisk/ramdisk
write-cache off
bs-flags direct:sync
</target>
-- SNIP
Performance parameters are never auto-negotiated properly (ever) so we
hardcode them in target.c
--- tgt-1.0.19/usr/iscsi/target.c 2011-08-02
02:01:28.000000000 -0400
+++ tgt-1.0.19.new/usr/iscsi/target.c 2011-09-09
21:04:53.897306138 -0400
@@ -431,15 +431,15 @@
int tid = t->tid;
struct iscsi_target *target;
struct param default_tgt_session_param[] = {
- [ISCSI_PARAM_MAX_RECV_DLENGTH] = {0, 8192},
+ [ISCSI_PARAM_MAX_RECV_DLENGTH] = {0, 16776192},
[ISCSI_PARAM_MAX_XMIT_DLENGTH] =
{0, 8192}, /* do not edit */
[ISCSI_PARAM_HDRDGST_EN] = {0,
DIGEST_NONE},
[ISCSI_PARAM_DATADGST_EN] = {0,
DIGEST_NONE},
- [ISCSI_PARAM_INITIAL_R2T_EN] = {0, 1},
- [ISCSI_PARAM_MAX_R2T] = {0, 1},
+ [ISCSI_PARAM_INITIAL_R2T_EN] = {0, 0},
+ [ISCSI_PARAM_MAX_R2T] = {0, 16},
[ISCSI_PARAM_IMM_DATA_EN] = {0,
1},
- [ISCSI_PARAM_FIRST_BURST] = {0, 65536},
- [ISCSI_PARAM_MAX_BURST] = {0, 262144},
+ [ISCSI_PARAM_FIRST_BURST] = {0, 16776192},
+ [ISCSI_PARAM_MAX_BURST] = {0, 16776192},
[ISCSI_PARAM_PDU_INORDER_EN] = {0,
1},
[ISCSI_PARAM_DATASEQ_INORDER_EN] =
{0, 1},
[ISCSI_PARAM_ERL] = {0, 0},
@@ -449,10 +449,10 @@
[ISCSI_PARAM_DEFAULTTIME2RETAIN] =
{0, 20},
[ISCSI_PARAM_OFMARKINT] = {0,
2048},
[ISCSI_PARAM_IFMARKINT] = {0,
2048},
- [ISCSI_PARAM_MAXCONNECTIONS] = {0, 1},
+ [ISCSI_PARAM_MAXCONNECTIONS] = {0, 4},
[ISCSI_PARAM_RDMA_EXTENSIONS] =
{0, 1},
- [ISCSI_PARAM_TARGET_RDSL] = {0, 262144},
- [ISCSI_PARAM_INITIATOR_RDSL] = {0, 262144},
+ [ISCSI_PARAM_TARGET_RDSL] = {0, 16776192},
+ [ISCSI_PARAM_INITIATOR_RDSL] = {0, 16776192},
[ISCSI_PARAM_MAX_OUTST_PDU] = {0,
0}, /* not in open-iscsi */
};
--SNIP
tgt-admin is badly out of sync with options available from the command
line, most notably bsflags
--- tgt-1.0.19/scripts/tgt-admin 2011-08-02
02:01:28.000000000 -0400
+++ /usr/sbin/tgt-admin 2011-09-20 10:53:11.479141197 -0400
@@ -284,10 +284,6 @@
$data_key = make_key($target_options_ref, "lun",
"allow-in-use");
}
- if (defined
$target_options{"controller_tid"}) {
- $next_tid =
$target_options{"controller_tid"};
- }
-
if (not defined $target_options{"driver"}) {
$target_options{"driver"} = $default_driver;
}
@@ -463,6 +459,7 @@
my @exec_commands;
my $device_type;
my $bs_type;
+ my $bs_flags;
my %luns;
my @added_luns;
# Find out LUNs which are
"reserved" in the config file
@@ -513,6 +510,10 @@
$bs_type = $result;
$params_added{$store_option} = 1;
}
+ if ($store_option
eq "bs-flags") {
+ $bs_flags
= $result;
+ $params_added{$store_option} = 1;
+ }
if ($store_option eq "mode_page") {
unless (ref($result) eq 'ARRAY') {
$result = [
$result ];
@@ -626,6 +627,11 @@
check_if_hash_array($$target_options_ref{"bs-type"}, "bs-type");
$bs_type = $$target_options_ref{"bs-type"};
}
+ # bs-flags
+ if ($params_added{"bs-flags"} ne 1) {
+ check_if_hash_array($$target_options_ref{"bs-flags"},
"bs-flags");
+ $bs_flags =
$$target_options_ref{"bs-flags"};
+ }
} else {
print "If you got
here, this means your config file is not supported.\n";
print "Please
report it to stgt mailing list and attach your config files.\n";
@@ -634,7 +640,8 @@
# Execute commands for a given LUN
if (length $device_type) {
$device_type = "--device-type $device_type" };
if (length $bs_type) { $bs_type =
"--bstype $bs_type" };
- execute("tgtadm -C $control_port --lld $driver
--op new --mode logicalunit --tid $next_tid --lun $lun -b $backing_store
$device_type $bs_type");
+ if (length $bs_flags) { $bs_flags = "--bsoflags
$bs_flags" };
+ execute("tgtadm -C $control_port --lld $driver
--op new --mode logicalunit --tid $next_tid --lun $lun -b $backing_store
$device_type $bs_type $bs_flags");
# Commands should be executed in
order
my @execute_last;
--SNIP
tgtd startup
-- Excerpt from modified /etc/init.d/tgtd
start()
{
# Check for and source configuration file
otherwise use defaults above
[ -f /etc/sysconfig/$exec ] && .
/etc/sysconfig/$exec
[ -x /usr/sbin/$exec ] || exit 5
echo -n $"Starting $prog: "
# Note /dev/null redirection to silence rdma not
found messages
NR_IOTHREAD=128
CHANNELS=10
BIND_PREFIX="172.16.1"
BIND_OFFSET=100
BIND_SUM=0
i=0
while [[ $i -le $CHANNELS ]]
do
let BIND_SUM="$BIND_OFFSET + $i"
daemon $exec -C $i -t $NR_IOTHREAD --iscsi
portal=$BIND_PREFIX.$BIND_SUM >/dev/null 2>&1
let i="$i + 1"
done
retval=$?
if [ $retval -ne 0 ] ; then
failure
echo
return $retval
fi
success
echo
touch $lockfile
# Put tgtd into "offline" state until all the
targets are configured.
# We don't want initiators to (re)connect and fail
the connection
# if it's not ready.
tgtadm --op update --mode sys --name State -v
offline
# Configure the targets.
i=0
while [[ $i -le $CHANNELS ]]
do
tgt-admin -C $i -e --force -c
/etc/tgt/targets.conf
let i="$i + 1"
done
# Put tgtd into "ready" state.
tgtadm --op update --mode sys --name State -v
ready
return 0
}
-- SNIP
Open-ISCSI client config:
Storage02
/var/lib/iscsi/iqn.1970-01.lab.symcor.symcor-nas:perf.ramdisk/*/default
# BEGIN RECORD 2.0-872
node.name = iqn.1970-01.lab.symcor.symcor-nas:perf.ramdisk
node.tpgt = 1
node.startup = manual
iface.iscsi_ifacename = default
iface.transport_name = tcp
node.discovery_address = 172.16.1.100
node.discovery_port = 3260
node.discovery_type = send_targets
node.session.initial_cmdsn = 0
node.session.initial_login_retry_max = 4
node.session.xmit_thread_priority = -20
node.session.cmds_max = 128
node.session.queue_depth = 128
node.session.auth.authmethod = None
node.session.timeo.replacement_timeout = 5
node.session.err_timeo.abort_timeout = 15
node.session.err_timeo.lu_reset_timeout = 30
node.session.err_timeo.tgt_reset_timeout = 30
node.session.err_timeo.host_reset_timeout = 60
node.session.iscsi.FastAbort = No
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 16776192
node.session.iscsi.MaxBurstLength = 16776192
node.session.iscsi.DefaultTime2Retain = 0
node.session.iscsi.DefaultTime2Wait = 2
node.session.iscsi.MaxConnections = 1
node.session.iscsi.MaxOutstandingR2T = 1
node.session.iscsi.ERL = 0
node.conn[0].address = 172.16.1.100
node.conn[0].port = 3260
node.conn[0].startup = manual
node.conn[0].tcp.window_size = 16776192
node.conn[0].tcp.type_of_service = 0
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.auth_timeout = 45
node.conn[0].timeo.noop_out_interval = 10
node.conn[0].timeo.noop_out_timeout = 15
node.conn[0].iscsi.MaxXmitDataSegmentLength = 16776192
node.conn[0].iscsi.MaxRecvDataSegmentLength = 16776192
node.conn[0].iscsi.HeaderDigest = None
node.conn[0].iscsi.IFMarker = No
node.conn[0].iscsi.OFMarker = No
# END RECORD
node.session.auth.authmethod = CHAP
node.session.auth.username = perf
node.session.auth.password = test1234
--SNIP
Network Tuning for 10GigE (based on best practices @ fasterdata.es.net)
# increase TCP max buffer size setable using setsockopt()
# 16 MB with a few parallel streams is recommended for most 10G
paths
# 32 MB might be needed for some very long end-to-end 10G or 40G
paths
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
# increase Linux autotuning TCP buffer limits
# min, default, and max number of bytes to use
# (only change the 3rd value, and make it 16 MB or more)
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432
# recommended to increase this for 10G NICS
net.core.netdev_max_backlog = 300000
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0
Device ioqueue tuning
-- /root/bin/nrq
#!/bin/bash
[[ $# == 0 ]] && echo "$0: <device> <nr_requests>" && exit 1
echo "noop" > /sys/block/$1/queue/scheduler
echo $2 > /sys/block/$1/queue/nr_requests
for i in /sys/block/$1/slaves/*
do
echo "noop" >$i/queue/scheduler
echo $2 > $i/queue/nr_requests
done
-- SNIP
Invoked as: nrq dm-11 128
# multipath.conf - recommended defaults in the userspace multipath are out
of sync
# with the progress of dm-multipath >= 1.2.0 which is based on an
request-based approach.
# essentially rr_min_io should be 1 almost all of the time. The majority
of "defaults"
# packaged in userspace tools became obsolete with 1.2.x
defaults {
user_friendly_names yes
path_selector "round-robin 0"
path_grouping_policy multibus
rr_weight "uniform"
rr_min_io = 1
path_checker directio
failback immediate
no_path_retry 6
prio "const"
fast_io_fail_tmo 3
dev_loss_tmo 6
max_fds max
features "0"
}
blacklist {
devnode
"^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
devnode "^dcssblk[0-9]*"
device {
vendor "DGC"
product "LUNZ"
}
device {
vendor "IBM"
product "S/390.*"
}
# don't count normal SATA devices as multipaths
device {
vendor "ATA"
}
# don't count 3ware devices as multipaths
device {
vendor "3ware"
}
device {
vendor "AMCC"
}
# nor highpoint devices
device {
vendor "HPT"
}
#wwid "20080930-1"
#wwid "20080930-1"
#wwid "360a980006466585a676f65693043556e"
#wwid "360a980006466585a676f65693043556e"
device {
vendor Cisco
product Virtual_CD_DVD
}
#wwid "*"
}
blacklist_exceptions {
wwid "360a980006466585a676f656931675068"
devnode "^sd[a-z]"
devnode "^sd[a-z][a-z]"
}
multipaths {
multipath {
uid 0
alias mpathb
gid 0
wwid
"360a980006466585a676f656931675068"
mode 0600
}
}
devices {
device {
vendor "IET"
product "VIRTUAL-DISK"
path_grouping_policy multibus
getuid_callout "/lib/udev/scsi_id
--whitelisted --device=/dev/%n"
path_selector "round-robin 0"
path_checker directio
features "0"
hardware_handler "0"
prio const
failback immediate
rr_weight uniform
rr_min_io 1
}
device {
vendor "NETAPP"
product "LUN"
path_grouping_policy multi_bus
getuid_callout "/lib/udev/scsi_id
--whitelisted --device=/dev/%n"
path_selector "round-robin 0"
path_checker directio
features "0"
hardware_handler "0"
prio const
failback immediate
rr_weight uniform
rr_min_io 1
}
}
--
Test 1 (4 channels nr_threads = 16)
root 10625 1 1 14:26 ? 00:00:10 tgtd -C 0 -t 16 --iscsi
portal=172.16.1.100
root 10626 10625 0 14:26 ? 00:00:00 tgtd -C 0 -t 16 --iscsi
portal=172.16.1.100
root 10629 1 1 14:26 ? 00:00:10 tgtd -C 1 -t 16 --iscsi
portal=172.16.1.101
root 10630 10629 0 14:26 ? 00:00:00 tgtd -C 1 -t 16 --iscsi
portal=172.16.1.101
root 10633 1 1 14:26 ? 00:00:10 tgtd -C 2 -t 16 --iscsi
portal=172.16.1.102
root 10634 10633 0 14:26 ? 00:00:00 tgtd -C 2 -t 16 --iscsi
portal=172.16.1.102
root 10637 1 1 14:26 ? 00:00:10 tgtd -C 3 -t 16 --iscsi
portal=172.16.1.103
root 10638 10637 0 14:26 ? 00:00:00 tgtd -C 3 -t 16 --iscsi
portal=172.16.1.103
Target 4: iqn.1970-01.lab.symcor.symcor-nas:perf.ramdisk
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00040000
SCSI SN: beaf40
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1
Type: disk
SCSI ID: IET 00040001
SCSI SN: beaf41
Size: 53683 MB, Block size: 512
Online: Yes
Removable media: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/ramdisk/ramdisk
Backing store flags: sync:direct
Account information:
ACL information:
ALL
[root at storage02 mnt]# /opt/iozone/bin/iozone-bigrec -I -az -s 64M
Iozone: Performance Test of File I/O
Version $Revision: 3.397 $
Compiled for 64 bit mode.
Build: linux-AMD64
Contributors:William Norcott, Don Capps, Isom Crawford, Kirby
Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain
CYR,
Randy Dunlap, Mark Montague, Dan Million, Gavin
Brebner,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave
Boone,
Erik Habbinga, Kris Strecker, Walter Wong, Joshua
Root,
Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren
Sawyer.
Ben England.
Run began: Mon Sep 26 11:11:02 2011
O_DIRECT feature enabled
Auto Mode
Cross over of record size disabled.
File size set to 65536 KB
Command line used: /opt/iozone/bin/iozone-bigrec -I -az -s 64M
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random
bkwd record stride
KB reclen write rewrite read reread read write
read rewrite read fwrite frewrite fread freread
65536 4 20887 22830 26637 26883 27194 22778
24991 22924 24573 1552615 2169435 4259770 2941197
65536 8 39821 43416 46859 46950 46355 42409
46629 43274 47314 1695722 2513564 4729504 5011465
65536 16 75551 79134 82536 83684 77902 81319
81903 83944 86898 2017736 2682604 4864251 5183167
65536 32 133316 138218 145389 152287 148942 144161
147717 149165 144204 1924634 2861320 5286438 5411748
65536 64 219197 233459 215835 203843 209681 227099
199867 246088 215980 2159617 2833506 5241179 5464030
65536 128 390131 352822 286832 284263 289042 356753
286414 366141 272927 2056548 2790190 5173801 5486934
65536 256 467726 471964 342877 326157 322137 455019
339264 482936 337871 2081701 2769444 5212852 5484635
65536 512 532587 550773 376129 374921 374177 553602
371911 584157 367282 1773785 2653497 5241579 5465442
65536 1024 642836 645731 576104 551004 582184 676912
562772 640612 496951 1628300 2320510 5336419 5480152
65536 2048 743576 727999 688831 663520 652794 724266
731650 779504 707403 1611376 2109985 5329073 5538007
65536 4096 858610 860378 836474 846521 822653 890276
848527 897666 923758 1599728 2047876 5077561 5366528
65536 8192 999862 992866 1001454 954278 974586 994853
1000335 958578 993781 1580468 1942149 3660606 3800708
65536 16384 1065039 1078511 1038341 1043533 1038357 1066733
1032875 1073709 1036141 1677314 2098499 3483929 3515116
65536 32768 1099376 1107606 1045112 1040168 1056674 1096153
1040881 1102113 1030084 1902232 2752803 3492295 3510985
65536 65536 1128974 1127769 1069431 1077167 1077678 1120999
1091032 1123538 1077345 1843793 2355709 3500656 3518130
iozone test complete.
[root at storage02 mnt]# /opt/iozone/bin/iozone-bigrec -I -az -s 64M -O
Iozone: Performance Test of File I/O
Version $Revision: 3.397 $
Compiled for 64 bit mode.
Build: linux-AMD64
Contributors:William Norcott, Don Capps, Isom Crawford, Kirby
Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain
CYR,
Randy Dunlap, Mark Montague, Dan Million, Gavin
Brebner,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave
Boone,
Erik Habbinga, Kris Strecker, Walter Wong, Joshua
Root,
Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren
Sawyer.
Ben England.
Run began: Mon Sep 26 11:48:25 2011
O_DIRECT feature enabled
Auto Mode
Cross over of record size disabled.
File size set to 65536 KB
OPS Mode. Output is in operations per second.
Command line used: /opt/iozone/bin/iozone-bigrec -I -az -s 64M -O
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random
bkwd record stride
KB reclen write rewrite read reread read write
read rewrite read fwrite frewrite fread freread
65536 4 4783 5243 5215 5188 5054 5322
5487 5446 5487 393042 478002 1076841 1112203
65536 8 4850 5154 4733 4823 4740 5165
4973 5310 5055 203578 290640 595264 630629
65536 16 4357 4835 5066 4647 4691 4877
5570 5095 5619 123440 128626 303023 317617
65536 32 4204 4461 4724 4209 4147 4200
4558 4565 4584 59212 88322 158477 167704
65536 64 3429 3678 3310 3300 3263 3443
3417 3677 3316 28389 44137 83748 85740
65536 128 2609 2647 2126 2149 2174 2613
2145 2712 2147 16317 22431 42012 43010
65536 256 1775 1813 1254 1268 1302 1812
1268 1933 1267 7075 10755 20081 21260
65536 512 989 998 683 684 691 1004
681 1059 685 3826 5295 10199 10617
65536 1024 656 639 515 540 547 643
532 684 549 1700 2114 5182 5376
65536 2048 376 392 327 349 338 384
347 418 351 757 1045 2457 2688
65536 4096 217 222 219 202 198 210
195 213 207 386 455 1060 1330
65536 8192 118 114 119 120 121 120
118 122 119 189 231 446 455
65536 16384 64 64 63 63 63 65
63 64 63 92 126 212 213
65536 32768 33 33 31 31 31 33
31 33 31 63 82 99 106
65536 65536 16 17 16 16 16 17
16 17 15 28 35 53 53
iozone test complete.
Test 2 (4 channels nr_threads = 128)
root 12308 1 0 14:42 ? 00:00:00 tgtd -C 0 -t 128 --iscsi
portal=172.16.1.100
root 12309 12308 0 14:42 ? 00:00:00 tgtd -C 0 -t 128 --iscsi
portal=172.16.1.100
root 12312 1 0 14:42 ? 00:00:00 tgtd -C 1 -t 128 --iscsi
portal=172.16.1.101
root 12313 12312 0 14:42 ? 00:00:00 tgtd -C 1 -t 128 --iscsi
portal=172.16.1.101
root 12316 1 0 14:42 ? 00:00:00 tgtd -C 2 -t 128 --iscsi
portal=172.16.1.102
root 12317 12316 0 14:42 ? 00:00:00 tgtd -C 2 -t 128 --iscsi
portal=172.16.1.102
root 12320 1 0 14:42 ? 00:00:00 tgtd -C 3 -t 128 --iscsi
portal=172.16.1.103
root 12321 12320 0 14:42 ? 00:00:00 tgtd -C 3 -t 128 --iscsi
portal=172.16.1.103
[root at storage02 mnt]# /opt/iozone/bin/iozone-bigrec -I -az -s 64M
Iozone: Performance Test of File I/O
Version $Revision: 3.397 $
Compiled for 64 bit mode.
Build: linux-AMD64
Contributors:William Norcott, Don Capps, Isom Crawford, Kirby
Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain
CYR,
Randy Dunlap, Mark Montague, Dan Million, Gavin
Brebner,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave
Boone,
Erik Habbinga, Kris Strecker, Walter Wong, Joshua
Root,
Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren
Sawyer.
Ben England.
Run began: Mon Sep 26 11:16:35 2011
O_DIRECT feature enabled
Auto Mode
Cross over of record size disabled.
File size set to 65536 KB
Command line used: /opt/iozone/bin/iozone-bigrec -I -az -s 64M
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random
bkwd record stride
KB reclen write rewrite read reread read write
read rewrite read fwrite frewrite fread freread
65536 4 20171 21882 22658 20949 19833 21040
22840 21613 22327 1503183 2192571 4269165 4422685
65536 8 36730 39735 42587 42140 37950 39680
37827 40414 41992 1677447 2520132 4765583 5026126
65536 16 70505 72805 79017 78979 77526 74277
79173 74523 78706 1855129 2730110 4818462 5148991
65536 32 123053 129425 131189 134596 130959 124538
130335 132018 127121 2096819 2827764 5276696 5361713
65536 64 202942 212126 196814 196591 193274 210769
189747 215359 194899 2025856 2952476 5354297 5492526
65536 128 288316 304427 247440 243774 248244 307274
250843 319013 248989 1954551 2754403 5289795 5516888
65536 256 388961 388695 297016 292229 298474 394579
282797 434317 310877 1771202 2754762 5282679 5419324
65536 512 453746 456907 325908 324517 324470 446902
321917 486121 328697 2059445 2751756 5289083 5452216
65536 1024 676416 710470 588732 567873 600405 661424
567484 734469 554585 1825972 2292448 5257218 5423815
65536 2048 794576 786464 743999 699789 685529 738151
702610 792932 650338 1676250 2100648 5238682 5404619
65536 4096 829159 858688 836367 858814 913471 974382
804725 902390 854234 1633088 2012846 5110413 5352942
65536 8192 1045028 1053261 996184 980887 983417 990266
996971 1024063 1000120 1570822 1934019 3645363 3724693
65536 16384 1081111 1087213 1065555 1016771 1048388 1080347
1057577 1093593 1053071 1653074 2079182 3404111 3406600
65536 32768 1109408 1120428 1035505 1037832 1042903 1118723
1071094 1117617 1035501 1934985 2552236 3225176 3265366
65536 65536 1127811 1133251 1041145 1054684 1048744 1127677
1037408 1133289 1047019 1789676 2230467 3319789 3321474
iozone test complete.
[root at storage02 mnt]# /opt/iozone/bin/iozone-bigrec -I -az -s 64M -O
Iozone: Performance Test of File I/O
Version $Revision: 3.397 $
Compiled for 64 bit mode.
Build: linux-AMD64
Contributors:William Norcott, Don Capps, Isom Crawford, Kirby
Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain
CYR,
Randy Dunlap, Mark Montague, Dan Million, Gavin
Brebner,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave
Boone,
Erik Habbinga, Kris Strecker, Walter Wong, Joshua
Root,
Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren
Sawyer.
Ben England.
Run began: Mon Sep 26 11:43:21 2011
O_DIRECT feature enabled
Auto Mode
Cross over of record size disabled.
File size set to 65536 KB
OPS Mode. Output is in operations per second.
Command line used: /opt/iozone/bin/iozone-bigrec -I -az -s 64M -O
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random
bkwd record stride
KB reclen write rewrite read reread read write
read rewrite read fwrite frewrite fread freread
65536 4 4384 4826 5247 4893 4769 4831
4827 4928 4898 378436 540779 1066314 1099005
65536 8 4461 4788 4651 4622 4444 4714
4401 4845 4614 205519 293367 599617 624009
65536 16 4238 4470 4337 4521 4304 4552
4757 4713 4597 114317 165642 287097 318064
65536 32 4077 4155 4004 3776 3899 4081
3878 4104 3897 63409 85683 159661 166140
65536 64 3302 3423 3203 3257 3142 3489
3095 3655 3255 34168 44192 81153 85690
65536 128 2356 2425 1929 1936 1935 2389
1953 2445 1959 14165 22253 41403 43029
65536 256 1605 1599 1140 1184 1162 1605
1176 1710 1160 8272 10921 20730 21320
65536 512 865 874 643 640 628 876
631 938 632 3839 5374 10341 10728
65536 1024 575 579 493 497 508 576
450 660 483 1741 2249 5115 5378
65536 2048 354 359 336 344 343 394
363 392 349 782 985 2569 2669
65536 4096 219 215 196 203 217 232
205 222 226 389 473 1254 1319
65536 8192 124 126 103 117 119 126
120 128 121 192 239 450 461
65536 16384 66 67 64 64 63 66
63 66 63 102 125 210 213
65536 32768 31 34 32 31 32 34
31 34 31 64 82 105 106
65536 65536 17 17 15 15 15 17
16 17 15 28 34 52 53
iozone test complete.
--
Greg Procunier
UNIX Administrator III - Enterprise Servers and Storage
1 Robert Speck Parkway, Suite 400, Mississauga, Ontario L4Z 4E7
Office: 416-673-3320
Mobile: 647-895-2977
Email: gprocunier at symcor.com
From: FUJITA Tomonori <fujita.tomonori at lab.ntt.co.jp>
To: agrover at redhat.com
Cc: fujita.tomonori at lab.ntt.co.jp, jab at sdsc.edu, stgt at vger.kernel.org
Date: 09/25/2011 08:11 PM
Subject: Re: Worker Threads
Sent by: stgt-owner at vger.kernel.org
On Wed, 21 Sep 2011 14:29:28 -0700
Andy Grover <agrover at redhat.com> wrote:
> On 07/27/2011 07:03 PM, FUJITA Tomonori wrote:
> > On Wed, 27 Jul 2011 13:03:52 -0700
> > "Bennett, Jeffrey" <jab at sdsc.edu> wrote:
> >
> >> Is there any "side effect" by doing this change?
> >
> > NR_WORKER_THREADS is the number of threads that perform I/Os. So you
> > run more threads. You consume more memory.
> >
> >> Is there any plan
> >> to make this a configuration option that can be changed with the
> >> tgt-adm command by the user?
> >
> > tgtd should adjust NR_WORKER_THREADS dynamically since the optimal
> > value depends on your workload, hardware, etc. Well, until we
> > implement that feature, I think that we can support the boot option to
> > specify the number of I/O threads.
> >
> > The following patch works for you? You can specify the number in the
> > following way:
>
> Hello Tomo-san,
>
> I haven't tried it but this looks pretty handy to me! Will you be
> applying it?
Ok, let's merge this if someone actually tests it and confirm that it
works.
--
To unsubscribe from this list: send the line "unsubscribe stgt" in
the body of a message to majordomo at vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
-------------- next part --------------
CONFIDENTIALITY WARNING
This communication, including any attachments, is for the exclusive use of addressee and may contain proprietary and/or confidential information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this communication and destroy all copies.
AVERTISSEMENT RELATIF ? LA CONFIDENTIALIT?
Ce message, ainsi que les pi?ces qui y sont jointes, est destin? ? l?usage exclusif de la personne ? laquelle il s?adresse et peut contenir de l?information personnelle ou confidentielle. Si le lecteur de ce message n?en est pas le destinataire, nous l?avisons par la pr?sente que toute diffusion, distribution, reproduction ou utilisation de son contenu est strictement interdite. Veuillez avertir sur-le-champ l?exp?diteur par retour de courrier ?lectronique et supprimez ce message ainsi que toutes les pi?ces jointes.
|
|