[stgt] Multipath iSCSI storage

[GProcunier at symcor.com]

Hello,

I am trying to configure multipath for a client over iSCSI using the stgt 
package in Red Hat Enterprise Linux 6.1 (tgt-1.0.14-2).

Here is a very rough sketch of what I am trying to do here: 
http://i51.tinypic.com/6jir1v.jpg

I have a two node Red Hat Cluster where both nodes share a LUN off our 
SAN.  I then encapsulate the shared LUN and create a volume group with the 
cluster bit set so CLVMD will track metadata changes between cluster 
members.  I finally carve up a 64GB volume out of the volume group and 
export it as an iSCSI target on both cluster members (targets.conf below).

I log into the iSCSI service on both cluster nodes with the client and 
enumerate the targets as /dev/sda and /dev/sdb.

I then start the multipath service on my client with an active/passive 
configuration.  It detects that /dev/sda and /dev/sdb are paths to the 
same disk which in turn creates a pseudo disk called /dev/mpatha.

I am able to format /dev/mpatha and perform work on it.

If I change the multipath configuration to active/active where IO is 
uniformly sent down each iSCSI target at the same time I end up corrupting 
/dev/mpatha.

Why does this occur when my client communicates with iSCSI targets via 
stgt and not when my client communicates with SAN storage directly using 
the same native multipath software (dm-multipath)?

Could this problem be a result of IO being split between paths not making 
it to the disk in order/correctly because stgt is using bufered IO rather 
than direct IO?

Any help would be GREATLY appreciated.

Cheers,


-- SNIP /etc/tgt/targets.conf --
default-driver iscsi

<target iqn.1970-01.com.example:netclient-0.disk>
  incominguser netclient ed86543cf2a15600c93a7c52c0f61b2bae710230
  backing-store /dev/cvg00/netclient-0
  write-cache off
</target>
-- SNIP /etc/tgt/targets.conf --

On the client I have the following Active/Passive multipath configuration:

-- SNIP /etc/multipath.conf --
defaults {
        user_friendly_names yes
}
blacklist {
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
        devnode "^hd[a-z]"
        devnode "^dcssblk[0-9]*"
        device {
                vendor "DGC"
                product "LUNZ"
        }
        device {
                vendor "IBM"
                product "S/390.*"
        }
        # don't count normal SATA devices as multipaths
        device {
                vendor  "ATA"
        }
        # don't count 3ware devices as multipaths
        device {
                vendor  "3ware"
        }
        device {
                vendor  "AMCC"
        }
        # nor highpoint devices
        device {
                vendor  "HPT"
        }
        device {
                vendor QEMU
                product QEMU_DVD-ROM
        }
        wwid "*"
}
blacklist_exceptions {
        wwid "1IET     00010001"
}
multipaths {
        multipath {
                uid 0
                alias mpatha
                gid 0
                wwid "1IET     00010001"
                mode 0600
        }
}
-- SNIP /etc/multipath.conf --
--

Greg Procunier
UNIX Administrator III - Enterprise Servers and Storage
1 Robert Speck Parkway, Suite 400, Mississauga, Ontario L4Z 4E7
Office: 416-673-3320 
Mobile: 647-895-2977
Email: gprocunier at symcor.com
-------------- next part --------------


CONFIDENTIALITY WARNING 
This communication, including any attachments, is for the exclusive use of addressee and may contain proprietary and/or confidential information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this communication and destroy all copies.

AVERTISSEMENT RELATIF ? LA CONFIDENTIALIT? 
Ce message, ainsi que les pi?ces qui y sont jointes, est destin? ? l?usage exclusif de la personne ? laquelle il s?adresse et peut contenir de l?information personnelle ou confidentielle. Si le lecteur de ce message n?en est pas le destinataire, nous l?avisons par la pr?sente que toute diffusion, distribution, reproduction ou utilisation de son contenu est strictement interdite. Veuillez avertir sur-le-champ l?exp?diteur par retour de courrier ?lectronique et supprimez ce message ainsi que toutes les pi?ces jointes.