[stgt] Worker Threads

GProcunier at symcor.com GProcunier at symcor.com
Mon Sep 26 21:28:45 CEST 2011


Executive summary: nr_iothread patch does not appear to meaningfully 
increase performance

My environment:

Storage01/Storage02 servers:

        Cisco UCS Server Blades with 192 GB of memory and 24 core Intel(R) 
Xeon(R) CPU X5670  @ 2.93GHz running RHEL6.1
        kernel 2.6.32-131.0.15.el6.x86_64
        UCS service profiles have a 9000 byte MTU vNIC associated with 
each machine.
        Are set in the platinum traffic class of the UCS guarantee of 62% 
backplane bandwidth.
        tgt-1.0.19 + nr_iothread patch
        both servers connect to each other through a Cisco 7k nexus 
switch.
        Both servers have a dedicated vNIC with 10GBit backplane access.
        iozone was used for testing and modified to allow record sizes up 
to 64MB (default is 16MB)
        the test disk was formatted with ext4 (mkfs.ext4 -E 
stride=16,stripe-width=64 /dev/mapper/mpathd)
        the test disk was mounted on the client with -o nobarrier option

Overview of changes and testing:

Storage01 creates a 50GB ramdisk on boot and exports it as an iSCSI target 
for clients to test with:

        -- Excerpt from /etc/init.d/ramdisk
        start() {
                        [ "$EUID" != "0" ] && exit 4
                        [ "$NETWORKING" = "no" ] && exit 1

                        # Start daemons.
                        echo -n $"Starting ramdisk: "
                        mount -t ramfs none /exports/ram0
                        dd if=/dev/zero of=/exports/ram0/ramdisk bs=1G 
count=50
                        sync
                        losetup /dev/loop0 /exports/ram0/ramdisk
                        pvcreate /dev/loop0
                        vgcreate -c n ramdisk /dev/loop0
                        lvcreate -n ramdisk -l 100%VG ramdisk
                        RETVAL=$?
                        echo
                        return $RETVAL
        }

Storage01 exports the ramdisk to several channels:

        -- /etc/tgt/targets.conf
        # Storage Cluster Node 01 Target Config

        default-driver iscsi

        # these params never actually work
        MaxConnections 1
        InitialR2T No
        ImmediateData Yes
        MaxRecvDataSegmentLength 16776192
        MaxXmitDataSegmentLength 16776192
        MaxBurstLength 16776192
        FirstBurstLength 16776192
        MaxOutstandingR2T 16
        DataDigest None
        HeaderDigest None

        <target iqn.1970-01.lab.symcor.symcor-nas:perf.ramdisk>
          incominguser perf test1234
          backing-store /dev/ramdisk/ramdisk
          write-cache off
          bs-flags direct:sync
        </target>

        -- SNIP

Performance parameters are never auto-negotiated properly (ever) so we 
hardcode them in target.c

        --- tgt-1.0.19/usr/iscsi/target.c       2011-08-02 
02:01:28.000000000 -0400
        +++ tgt-1.0.19.new/usr/iscsi/target.c   2011-09-09 
21:04:53.897306138 -0400
        @@ -431,15 +431,15 @@
                        int tid = t->tid;
                        struct iscsi_target *target;
                        struct param default_tgt_session_param[] = {
        -               [ISCSI_PARAM_MAX_RECV_DLENGTH] = {0, 8192},
        +               [ISCSI_PARAM_MAX_RECV_DLENGTH] = {0, 16776192},
                                        [ISCSI_PARAM_MAX_XMIT_DLENGTH] = 
{0, 8192},  /* do not edit */
                                        [ISCSI_PARAM_HDRDGST_EN] = {0, 
DIGEST_NONE},
                                        [ISCSI_PARAM_DATADGST_EN] = {0, 
DIGEST_NONE},
        -               [ISCSI_PARAM_INITIAL_R2T_EN] = {0, 1},
        -               [ISCSI_PARAM_MAX_R2T] = {0, 1},
        +               [ISCSI_PARAM_INITIAL_R2T_EN] = {0, 0},
        +               [ISCSI_PARAM_MAX_R2T] = {0, 16},
                                        [ISCSI_PARAM_IMM_DATA_EN] = {0, 
1},
        -               [ISCSI_PARAM_FIRST_BURST] = {0, 65536},
        -               [ISCSI_PARAM_MAX_BURST] = {0, 262144},
        +               [ISCSI_PARAM_FIRST_BURST] = {0, 16776192},
        +               [ISCSI_PARAM_MAX_BURST] = {0, 16776192},
                                        [ISCSI_PARAM_PDU_INORDER_EN] = {0, 
1},
                                        [ISCSI_PARAM_DATASEQ_INORDER_EN] = 
{0, 1},
                                        [ISCSI_PARAM_ERL] = {0, 0},
        @@ -449,10 +449,10 @@
                                        [ISCSI_PARAM_DEFAULTTIME2RETAIN] = 
{0, 20},
                                        [ISCSI_PARAM_OFMARKINT] = {0, 
2048},
                                        [ISCSI_PARAM_IFMARKINT] = {0, 
2048},
        -               [ISCSI_PARAM_MAXCONNECTIONS] = {0, 1},
        +               [ISCSI_PARAM_MAXCONNECTIONS] = {0, 4},
                                        [ISCSI_PARAM_RDMA_EXTENSIONS] = 
{0, 1},
        -               [ISCSI_PARAM_TARGET_RDSL] = {0, 262144},
        -               [ISCSI_PARAM_INITIATOR_RDSL] = {0, 262144},
        +               [ISCSI_PARAM_TARGET_RDSL] = {0, 16776192},
        +               [ISCSI_PARAM_INITIATOR_RDSL] = {0, 16776192},
                                        [ISCSI_PARAM_MAX_OUTST_PDU] =  {0, 
0},  /* not in open-iscsi */
                        };
--SNIP

tgt-admin is badly out of sync with options available from the command 
line, most notably bsflags

        --- tgt-1.0.19/scripts/tgt-admin        2011-08-02 
02:01:28.000000000 -0400
        +++ /usr/sbin/tgt-admin 2011-09-20 10:53:11.479141197 -0400
        @@ -284,10 +284,6 @@
                        $data_key = make_key($target_options_ref, "lun", 
"allow-in-use");
        }
 
        -                                       if (defined 
$target_options{"controller_tid"}) {
        -                                               $next_tid = 
$target_options{"controller_tid"};
        -                                       }
        - 
        if (not defined $target_options{"driver"}) {
                        $target_options{"driver"} = $default_driver;
        }
        @@ -463,6 +459,7 @@
                                        my @exec_commands;
                                        my $device_type;
                                        my $bs_type;
        +               my $bs_flags;
                                        my %luns;
                                        my @added_luns;
                                        # Find out LUNs which are 
"reserved" in the config file
        @@ -513,6 +510,10 @@
                                        $bs_type = $result;
                                        $params_added{$store_option} = 1;
                        }
        +                                               if ($store_option 
eq "bs-flags") {
        +                                                       $bs_flags 
= $result;
        + $params_added{$store_option} = 1;
        +                                               }
                        if ($store_option eq "mode_page") {
                                        unless (ref($result) eq 'ARRAY') {
                                                        $result = [ 
$result ];
        @@ -626,6 +627,11 @@
 check_if_hash_array($$target_options_ref{"bs-type"}, "bs-type");
 $bs_type = $$target_options_ref{"bs-type"};
                                                        }
        +                       # bs-flags
        +                       if ($params_added{"bs-flags"} ne 1) {
        + check_if_hash_array($$target_options_ref{"bs-flags"}, 
"bs-flags");
        +                               $bs_flags = 
$$target_options_ref{"bs-flags"};
        +                       }
                                        } else {
                                                        print "If you got 
here, this means your config file is not supported.\n";
                                                        print "Please 
report it to stgt mailing list and attach your config files.\n";
        @@ -634,7 +640,8 @@
                                        # Execute commands for a given LUN
                                        if (length $device_type) { 
$device_type = "--device-type $device_type" };
                                        if (length $bs_type) { $bs_type = 
"--bstype $bs_type" };
        -               execute("tgtadm -C $control_port --lld $driver 
--op new --mode logicalunit --tid $next_tid --lun $lun -b $backing_store 
$device_type $bs_type");
        +               if (length $bs_flags) { $bs_flags = "--bsoflags 
$bs_flags" };
        +               execute("tgtadm -C $control_port --lld $driver 
--op new --mode logicalunit --tid $next_tid --lun $lun -b $backing_store 
$device_type $bs_type $bs_flags");
 
                                        # Commands should be executed in 
order
                                        my @execute_last;
        --SNIP

tgtd startup

        -- Excerpt from modified /etc/init.d/tgtd

        start()
        {
                        # Check for and source configuration file 
otherwise use defaults above
                        [ -f /etc/sysconfig/$exec ] && . 
/etc/sysconfig/$exec

                        [ -x /usr/sbin/$exec ] || exit 5
                        echo -n $"Starting $prog: "
                        # Note /dev/null redirection to silence rdma not 
found messages
                        NR_IOTHREAD=128
                        CHANNELS=10
                        BIND_PREFIX="172.16.1"
                        BIND_OFFSET=100
                        BIND_SUM=0
                        i=0
                        while [[ $i -le $CHANNELS ]]
                        do
                          let BIND_SUM="$BIND_OFFSET + $i" 
                          daemon $exec -C $i -t $NR_IOTHREAD --iscsi 
portal=$BIND_PREFIX.$BIND_SUM >/dev/null 2>&1
                          let i="$i + 1"
                        done
                        retval=$?
                        if [ $retval -ne 0 ] ; then
                                        failure
                                        echo
                                        return $retval
                        fi

                        success
                        echo
                        touch $lockfile

                        # Put tgtd into "offline" state until all the 
targets are configured.
                        # We don't want initiators to (re)connect and fail 
the connection
                        # if it's not ready.
                        tgtadm --op update --mode sys --name State -v 
offline
                        # Configure the targets.
                        i=0
                        while [[ $i -le $CHANNELS ]]
                        do
                          tgt-admin -C $i -e --force -c 
/etc/tgt/targets.conf
                          let i="$i + 1"
                        done
                        # Put tgtd into "ready" state.
                        tgtadm --op update --mode sys --name State -v 
ready

                        return 0
        }
        -- SNIP
Open-ISCSI client config:
Storage02 
/var/lib/iscsi/iqn.1970-01.lab.symcor.symcor-nas:perf.ramdisk/*/default

        # BEGIN RECORD 2.0-872
        node.name = iqn.1970-01.lab.symcor.symcor-nas:perf.ramdisk
        node.tpgt = 1
        node.startup = manual
        iface.iscsi_ifacename = default
        iface.transport_name = tcp
        node.discovery_address = 172.16.1.100
        node.discovery_port = 3260
        node.discovery_type = send_targets
        node.session.initial_cmdsn = 0
        node.session.initial_login_retry_max = 4
        node.session.xmit_thread_priority = -20
        node.session.cmds_max = 128
        node.session.queue_depth = 128
        node.session.auth.authmethod = None
        node.session.timeo.replacement_timeout = 5
        node.session.err_timeo.abort_timeout = 15
        node.session.err_timeo.lu_reset_timeout = 30
        node.session.err_timeo.tgt_reset_timeout = 30
        node.session.err_timeo.host_reset_timeout = 60
        node.session.iscsi.FastAbort = No
        node.session.iscsi.InitialR2T = No
        node.session.iscsi.ImmediateData = Yes
        node.session.iscsi.FirstBurstLength = 16776192
        node.session.iscsi.MaxBurstLength = 16776192
        node.session.iscsi.DefaultTime2Retain = 0
        node.session.iscsi.DefaultTime2Wait = 2
        node.session.iscsi.MaxConnections = 1
        node.session.iscsi.MaxOutstandingR2T = 1
        node.session.iscsi.ERL = 0
        node.conn[0].address = 172.16.1.100
        node.conn[0].port = 3260
        node.conn[0].startup = manual
        node.conn[0].tcp.window_size = 16776192
        node.conn[0].tcp.type_of_service = 0
        node.conn[0].timeo.logout_timeout = 15
        node.conn[0].timeo.login_timeout = 15
        node.conn[0].timeo.auth_timeout = 45
        node.conn[0].timeo.noop_out_interval = 10
        node.conn[0].timeo.noop_out_timeout = 15
        node.conn[0].iscsi.MaxXmitDataSegmentLength = 16776192
        node.conn[0].iscsi.MaxRecvDataSegmentLength = 16776192
        node.conn[0].iscsi.HeaderDigest = None
        node.conn[0].iscsi.IFMarker = No
        node.conn[0].iscsi.OFMarker = No
        # END RECORD
        node.session.auth.authmethod = CHAP
        node.session.auth.username = perf
        node.session.auth.password = test1234
--SNIP
 

Network Tuning for 10GigE (based on best practices @ fasterdata.es.net)

        # increase TCP max buffer size setable using setsockopt()
        # 16 MB with a few parallel streams is recommended for most 10G 
paths
        # 32 MB might be needed for some very long end-to-end 10G or 40G 
paths
        net.core.rmem_max = 33554432
        net.core.wmem_max = 33554432
        # increase Linux autotuning TCP buffer limits 
        # min, default, and max number of bytes to use
        # (only change the 3rd value, and make it 16 MB or more)
        net.ipv4.tcp_rmem = 4096 87380 33554432
        net.ipv4.tcp_wmem = 4096 65536 33554432
        # recommended to increase this for 10G NICS
        net.core.netdev_max_backlog = 300000
        net.ipv4.tcp_sack = 0
        net.ipv4.tcp_timestamps = 0

Device ioqueue tuning

        -- /root/bin/nrq
        #!/bin/bash

        [[ $# == 0 ]] && echo "$0: <device> <nr_requests>" && exit 1
        echo "noop" > /sys/block/$1/queue/scheduler
        echo $2 > /sys/block/$1/queue/nr_requests
        for i in /sys/block/$1/slaves/*
        do
          echo "noop" >$i/queue/scheduler
          echo $2 > $i/queue/nr_requests
        done

        -- SNIP

Invoked as: nrq dm-11 128

# multipath.conf - recommended defaults in the userspace multipath are out 
of sync
# with the progress of dm-multipath >= 1.2.0 which is based on an 
request-based approach.
# essentially rr_min_io should be 1 almost all of the time.  The majority 
of "defaults" 
# packaged in userspace tools became obsolete with 1.2.x

        defaults {
                        user_friendly_names yes
                        path_selector "round-robin 0"
                        path_grouping_policy multibus
                        rr_weight "uniform"
                        rr_min_io = 1
                        path_checker directio
                        failback immediate
                        no_path_retry 6
                        prio "const"
                        fast_io_fail_tmo 3 
                        dev_loss_tmo 6
                        max_fds max
                        features "0"
        }
        blacklist {
                        devnode 
"^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
                        devnode "^hd[a-z]"
                        devnode "^dcssblk[0-9]*"
                        device {
                                        vendor "DGC"
                                        product "LUNZ"
                        }
                        device {
                                        vendor "IBM"
                                        product "S/390.*"
                        }
                        # don't count normal SATA devices as multipaths
                        device {
                                        vendor  "ATA"
                        }
                        # don't count 3ware devices as multipaths
                        device {
                                        vendor  "3ware"
                        }
                        device {
                                        vendor  "AMCC"
                        }
                        # nor highpoint devices
                        device {
                                        vendor  "HPT"
                        }
                        #wwid "20080930-1"
                        #wwid "20080930-1"
                        #wwid "360a980006466585a676f65693043556e"
                        #wwid "360a980006466585a676f65693043556e"
                        device {
                                        vendor Cisco
                                        product Virtual_CD_DVD
                        }
                        #wwid "*"
        }
        blacklist_exceptions {
                        wwid "360a980006466585a676f656931675068"
                        devnode "^sd[a-z]"
                        devnode "^sd[a-z][a-z]"
        }
        multipaths {
                        multipath {
                                        uid 0
                                        alias mpathb
                                        gid 0
                                        wwid 
"360a980006466585a676f656931675068"
                                        mode 0600
                        }
        }

        devices {
                        device {
                                        vendor "IET"
                                        product "VIRTUAL-DISK"
                                        path_grouping_policy multibus
                                        getuid_callout "/lib/udev/scsi_id 
--whitelisted --device=/dev/%n"
                                        path_selector "round-robin 0"
                                        path_checker directio
                                        features "0"
                                        hardware_handler "0"
                                        prio const
                                        failback immediate
                                        rr_weight uniform
                                        rr_min_io 1
                        }
                        device {
                                        vendor "NETAPP"
                                        product "LUN"
                                        path_grouping_policy multi_bus
                                        getuid_callout "/lib/udev/scsi_id 
--whitelisted --device=/dev/%n"
                                        path_selector "round-robin 0"
                                        path_checker directio
                                        features "0"
                                        hardware_handler "0"
                                        prio const
                                        failback immediate
                                        rr_weight uniform
                                        rr_min_io 1
                        }
        }


 
 
--
Test 1 (4 channels nr_threads = 16)

root     10625     1  1 14:26 ?        00:00:10 tgtd -C 0 -t 16 --iscsi 
portal=172.16.1.100
root     10626 10625  0 14:26 ?        00:00:00 tgtd -C 0 -t 16 --iscsi 
portal=172.16.1.100
root     10629     1  1 14:26 ?        00:00:10 tgtd -C 1 -t 16 --iscsi 
portal=172.16.1.101
root     10630 10629  0 14:26 ?        00:00:00 tgtd -C 1 -t 16 --iscsi 
portal=172.16.1.101
root     10633     1  1 14:26 ?        00:00:10 tgtd -C 2 -t 16 --iscsi 
portal=172.16.1.102
root     10634 10633  0 14:26 ?        00:00:00 tgtd -C 2 -t 16 --iscsi 
portal=172.16.1.102
root     10637     1  1 14:26 ?        00:00:10 tgtd -C 3 -t 16 --iscsi 
portal=172.16.1.103
root     10638 10637  0 14:26 ?        00:00:00 tgtd -C 3 -t 16 --iscsi 
portal=172.16.1.103

Target 4: iqn.1970-01.lab.symcor.symcor-nas:perf.ramdisk
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00040000
            SCSI SN: beaf40
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Readonly: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
        LUN: 1
            Type: disk
            SCSI ID: IET     00040001
            SCSI SN: beaf41
            Size: 53683 MB, Block size: 512
            Online: Yes
            Removable media: No
            Readonly: No
            Backing store type: rdwr
            Backing store path: /dev/ramdisk/ramdisk
            Backing store flags: sync:direct
    Account information:
    ACL information:
        ALL

[root at storage02 mnt]# /opt/iozone/bin/iozone-bigrec -I -az -s 64M
        Iozone: Performance Test of File I/O
                Version $Revision: 3.397 $
                Compiled for 64 bit mode.
                Build: linux-AMD64 

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby 
Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain 
CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin 
Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave 
Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua 
Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren 
Sawyer.
                     Ben England.

        Run began: Mon Sep 26 11:11:02 2011

        O_DIRECT feature enabled
        Auto Mode
        Cross over of record size disabled.
        File size set to 65536 KB
        Command line used: /opt/iozone/bin/iozone-bigrec -I -az -s 64M
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                            random  random 
   bkwd   record   stride 
              KB  reclen   write rewrite    read    reread    read   write 
   read  rewrite     read   fwrite frewrite   fread  freread
           65536       4   20887   22830    26637    26883   27194   22778 
  24991    22924    24573  1552615  2169435 4259770  2941197
           65536       8   39821   43416    46859    46950   46355   42409 
  46629    43274    47314  1695722  2513564 4729504  5011465
           65536      16   75551   79134    82536    83684   77902   81319 
  81903    83944    86898  2017736  2682604 4864251  5183167
           65536      32  133316  138218   145389   152287  148942  144161 
 147717   149165   144204  1924634  2861320 5286438  5411748
           65536      64  219197  233459   215835   203843  209681  227099 
 199867   246088   215980  2159617  2833506 5241179  5464030
           65536     128  390131  352822   286832   284263  289042  356753 
 286414   366141   272927  2056548  2790190 5173801  5486934
           65536     256  467726  471964   342877   326157  322137  455019 
 339264   482936   337871  2081701  2769444 5212852  5484635
           65536     512  532587  550773   376129   374921  374177  553602 
 371911   584157   367282  1773785  2653497 5241579  5465442
           65536    1024  642836  645731   576104   551004  582184  676912 
 562772   640612   496951  1628300  2320510 5336419  5480152
           65536    2048  743576  727999   688831   663520  652794  724266 
 731650   779504   707403  1611376  2109985 5329073  5538007
           65536    4096  858610  860378   836474   846521  822653  890276 
 848527   897666   923758  1599728  2047876 5077561  5366528
           65536    8192  999862  992866  1001454   954278  974586  994853 
1000335   958578   993781  1580468  1942149 3660606  3800708
           65536   16384 1065039 1078511  1038341  1043533 1038357 1066733 
1032875  1073709  1036141  1677314  2098499 3483929  3515116
           65536   32768 1099376 1107606  1045112  1040168 1056674 1096153 
1040881  1102113  1030084  1902232  2752803 3492295  3510985
           65536   65536 1128974 1127769  1069431  1077167 1077678 1120999 
1091032  1123538  1077345  1843793  2355709 3500656  3518130

iozone test complete.

[root at storage02 mnt]# /opt/iozone/bin/iozone-bigrec -I -az -s 64M -O
        Iozone: Performance Test of File I/O
                Version $Revision: 3.397 $
                Compiled for 64 bit mode.
                Build: linux-AMD64 

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby 
Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain 
CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin 
Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave 
Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua 
Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren 
Sawyer.
                     Ben England.

        Run began: Mon Sep 26 11:48:25 2011

        O_DIRECT feature enabled
        Auto Mode
        Cross over of record size disabled.
        File size set to 65536 KB
        OPS Mode. Output is in operations per second.
        Command line used: /opt/iozone/bin/iozone-bigrec -I -az -s 64M -O
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                            random  random 
   bkwd   record   stride 
              KB  reclen   write rewrite    read    reread    read   write 
   read  rewrite     read   fwrite frewrite   fread  freread
           65536       4    4783    5243     5215     5188    5054    5322 
   5487     5446     5487   393042   478002 1076841  1112203
           65536       8    4850    5154     4733     4823    4740    5165 
   4973     5310     5055   203578   290640  595264   630629
           65536      16    4357    4835     5066     4647    4691    4877 
   5570     5095     5619   123440   128626  303023   317617
           65536      32    4204    4461     4724     4209    4147    4200 
   4558     4565     4584    59212    88322  158477   167704
           65536      64    3429    3678     3310     3300    3263    3443 
   3417     3677     3316    28389    44137   83748    85740
           65536     128    2609    2647     2126     2149    2174    2613 
   2145     2712     2147    16317    22431   42012    43010
           65536     256    1775    1813     1254     1268    1302    1812 
   1268     1933     1267     7075    10755   20081    21260
           65536     512     989     998      683      684     691    1004 
    681     1059      685     3826     5295   10199    10617
           65536    1024     656     639      515      540     547     643 
    532      684      549     1700     2114    5182     5376
           65536    2048     376     392      327      349     338     384 
    347      418      351      757     1045    2457     2688
           65536    4096     217     222      219      202     198     210 
    195      213      207      386      455    1060     1330
           65536    8192     118     114      119      120     121     120 
    118      122      119      189      231     446      455
           65536   16384      64      64       63       63      63      65 
     63       64       63       92      126     212      213
           65536   32768      33      33       31       31      31      33 
     31       33       31       63       82      99      106
           65536   65536      16      17       16       16      16      17 
     16       17       15       28       35      53       53

iozone test complete.


Test 2 (4 channels nr_threads = 128)

root     12308     1  0 14:42 ?        00:00:00 tgtd -C 0 -t 128 --iscsi 
portal=172.16.1.100
root     12309 12308  0 14:42 ?        00:00:00 tgtd -C 0 -t 128 --iscsi 
portal=172.16.1.100
root     12312     1  0 14:42 ?        00:00:00 tgtd -C 1 -t 128 --iscsi 
portal=172.16.1.101
root     12313 12312  0 14:42 ?        00:00:00 tgtd -C 1 -t 128 --iscsi 
portal=172.16.1.101
root     12316     1  0 14:42 ?        00:00:00 tgtd -C 2 -t 128 --iscsi 
portal=172.16.1.102
root     12317 12316  0 14:42 ?        00:00:00 tgtd -C 2 -t 128 --iscsi 
portal=172.16.1.102
root     12320     1  0 14:42 ?        00:00:00 tgtd -C 3 -t 128 --iscsi 
portal=172.16.1.103
root     12321 12320  0 14:42 ?        00:00:00 tgtd -C 3 -t 128 --iscsi 
portal=172.16.1.103


[root at storage02 mnt]# /opt/iozone/bin/iozone-bigrec -I -az -s 64M
        Iozone: Performance Test of File I/O
                Version $Revision: 3.397 $
                Compiled for 64 bit mode.
                Build: linux-AMD64 

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby 
Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain 
CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin 
Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave 
Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua 
Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren 
Sawyer.
                     Ben England.

        Run began: Mon Sep 26 11:16:35 2011

        O_DIRECT feature enabled
        Auto Mode
        Cross over of record size disabled.
        File size set to 65536 KB
        Command line used: /opt/iozone/bin/iozone-bigrec -I -az -s 64M
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                            random  random 
   bkwd   record   stride 
              KB  reclen   write rewrite    read    reread    read   write 
   read  rewrite     read   fwrite frewrite   fread  freread
           65536       4   20171   21882    22658    20949   19833   21040 
  22840    21613    22327  1503183  2192571 4269165  4422685
           65536       8   36730   39735    42587    42140   37950   39680 
  37827    40414    41992  1677447  2520132 4765583  5026126
           65536      16   70505   72805    79017    78979   77526   74277 
  79173    74523    78706  1855129  2730110 4818462  5148991
           65536      32  123053  129425   131189   134596  130959  124538 
 130335   132018   127121  2096819  2827764 5276696  5361713
           65536      64  202942  212126   196814   196591  193274  210769 
 189747   215359   194899  2025856  2952476 5354297  5492526
           65536     128  288316  304427   247440   243774  248244  307274 
 250843   319013   248989  1954551  2754403 5289795  5516888
           65536     256  388961  388695   297016   292229  298474  394579 
 282797   434317   310877  1771202  2754762 5282679  5419324
           65536     512  453746  456907   325908   324517  324470  446902 
 321917   486121   328697  2059445  2751756 5289083  5452216
           65536    1024  676416  710470   588732   567873  600405  661424 
 567484   734469   554585  1825972  2292448 5257218  5423815
           65536    2048  794576  786464   743999   699789  685529  738151 
 702610   792932   650338  1676250  2100648 5238682  5404619
           65536    4096  829159  858688   836367   858814  913471  974382 
 804725   902390   854234  1633088  2012846 5110413  5352942
           65536    8192 1045028 1053261   996184   980887  983417  990266 
 996971  1024063  1000120  1570822  1934019 3645363  3724693
           65536   16384 1081111 1087213  1065555  1016771 1048388 1080347 
1057577  1093593  1053071  1653074  2079182 3404111  3406600
           65536   32768 1109408 1120428  1035505  1037832 1042903 1118723 
1071094  1117617  1035501  1934985  2552236 3225176  3265366
           65536   65536 1127811 1133251  1041145  1054684 1048744 1127677 
1037408  1133289  1047019  1789676  2230467 3319789  3321474

iozone test complete.

[root at storage02 mnt]# /opt/iozone/bin/iozone-bigrec -I -az -s 64M -O
        Iozone: Performance Test of File I/O
                Version $Revision: 3.397 $
                Compiled for 64 bit mode.
                Build: linux-AMD64 

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby 
Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain 
CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin 
Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave 
Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua 
Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren 
Sawyer.
                     Ben England.

        Run began: Mon Sep 26 11:43:21 2011

        O_DIRECT feature enabled
        Auto Mode
        Cross over of record size disabled.
        File size set to 65536 KB
        OPS Mode. Output is in operations per second.
        Command line used: /opt/iozone/bin/iozone-bigrec -I -az -s 64M -O
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                            random  random 
   bkwd   record   stride 
              KB  reclen   write rewrite    read    reread    read   write 
   read  rewrite     read   fwrite frewrite   fread  freread
           65536       4    4384    4826     5247     4893    4769    4831 
   4827     4928     4898   378436   540779 1066314  1099005
           65536       8    4461    4788     4651     4622    4444    4714 
   4401     4845     4614   205519   293367  599617   624009
           65536      16    4238    4470     4337     4521    4304    4552 
   4757     4713     4597   114317   165642  287097   318064
           65536      32    4077    4155     4004     3776    3899    4081 
   3878     4104     3897    63409    85683  159661   166140
           65536      64    3302    3423     3203     3257    3142    3489 
   3095     3655     3255    34168    44192   81153    85690
           65536     128    2356    2425     1929     1936    1935    2389 
   1953     2445     1959    14165    22253   41403    43029
           65536     256    1605    1599     1140     1184    1162    1605 
   1176     1710     1160     8272    10921   20730    21320
           65536     512     865     874      643      640     628     876 
    631      938      632     3839     5374   10341    10728
           65536    1024     575     579      493      497     508     576 
    450      660      483     1741     2249    5115     5378
           65536    2048     354     359      336      344     343     394 
    363      392      349      782      985    2569     2669
           65536    4096     219     215      196      203     217     232 
    205      222      226      389      473    1254     1319
           65536    8192     124     126      103      117     119     126 
    120      128      121      192      239     450      461
           65536   16384      66      67       64       64      63      66 
     63       66       63      102      125     210      213
           65536   32768      31      34       32       31      32      34 
     31       34       31       64       82     105      106
           65536   65536      17      17       15       15      15      17 
     16       17       15       28       34      52       53

iozone test complete.

--

Greg Procunier
UNIX Administrator III - Enterprise Servers and Storage
1 Robert Speck Parkway, Suite 400, Mississauga, Ontario L4Z 4E7
Office: 416-673-3320 
Mobile: 647-895-2977
Email: gprocunier at symcor.com



From:   FUJITA Tomonori <fujita.tomonori at lab.ntt.co.jp>
To:     agrover at redhat.com
Cc:     fujita.tomonori at lab.ntt.co.jp, jab at sdsc.edu, stgt at vger.kernel.org
Date:   09/25/2011 08:11 PM
Subject:        Re: Worker Threads
Sent by:        stgt-owner at vger.kernel.org



On Wed, 21 Sep 2011 14:29:28 -0700
Andy Grover <agrover at redhat.com> wrote:

> On 07/27/2011 07:03 PM, FUJITA Tomonori wrote:
> > On Wed, 27 Jul 2011 13:03:52 -0700
> > "Bennett, Jeffrey" <jab at sdsc.edu> wrote:
> > 
> >> Is there any "side effect" by doing this change?
> > 
> > NR_WORKER_THREADS is the number of threads that perform I/Os. So you
> > run more threads. You consume more memory.
> > 
> >> Is there any plan
> >> to make this a configuration option that can be changed with the
> >> tgt-adm command by the user?
> > 
> > tgtd should adjust NR_WORKER_THREADS dynamically since the optimal
> > value depends on your workload, hardware, etc. Well, until we
> > implement that feature, I think that we can support the boot option to
> > specify the number of I/O threads.
> > 
> > The following patch works for you? You can specify the number in the
> > following way:
> 
> Hello Tomo-san,
> 
> I haven't tried it but this looks pretty handy to me! Will you be
> applying it?

Ok, let's merge this if someone actually tests it and confirm that it
works.
--
To unsubscribe from this list: send the line "unsubscribe stgt" in
the body of a message to majordomo at vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


-------------- next part --------------


CONFIDENTIALITY WARNING 
This communication, including any attachments, is for the exclusive use of addressee and may contain proprietary and/or confidential information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this communication and destroy all copies.

AVERTISSEMENT RELATIF ? LA CONFIDENTIALIT? 
Ce message, ainsi que les pi?ces qui y sont jointes, est destin? ? l?usage exclusif de la personne ? laquelle il s?adresse et peut contenir de l?information personnelle ou confidentielle. Si le lecteur de ce message n?en est pas le destinataire, nous l?avisons par la pr?sente que toute diffusion, distribution, reproduction ou utilisation de son contenu est strictement interdite. Veuillez avertir sur-le-champ l?exp?diteur par retour de courrier ?lectronique et supprimez ce message ainsi que toutes les pi?ces jointes.


More information about the stgt mailing list