[Stgt-devel] Multiple tgtd exporting same device.

[Simone Gotti]

On Sun, 2008-06-29 at 21:46 +0900, FUJITA Tomonori wrote:
> From: Simone Gotti <simone.gotti at gmail.com>
> Subject: [Stgt-devel] Multiple tgtd exporting same device.
> Date: Thu, 26 Jun 2008 23:13:27 +0200
> 
> > Hi all,
> > 
> > to increase the availability (and also to use in future some features
> > like cluster mirror with RedHat Cluster Suite when it'll be available on
> > RHEL5 or in the upstream kernel) of a iscsi target server I'd like to
> 
> I'm not sure how this configuration can increase the availability.

I'll try to explain it in a clearer way.

                        ? ____________
                        |            |
                        | ? Storage   |
                        ?|____________|
                          /        \
                         /          \
               ___________          ?___________
              | Target 1  |        | Target 2  |
              |   tgtd    |        |   tgtd    |
              |___________|        |??___________|
                 | |                 /  /
?                 | |   _____________/  /
?                 | |  /  _____________/
                 | | /  /
?              ? ___________          ?___________
              | CLient 1  |  ...   | Client X  |
              |   tgtd    |        |           |
              |?___________|        |?___________|



The storage exports the same devices via FC (also multipathed), SCSI
Cable ecc...

Then 2 (or more) Servers exports in an active/active way the same device
via iSCSI and multipathing on the client is used to see the same device
(here via 4 paths).

?What I'd like to do is to minimize the recovery time (and maybe, but
this is not my primary reason the increase throughput) when a machine
dies (or similar problems).

Another solution would be an active/passive management where tgtd runs
only on one machine and it's relocated on the other one using the
cluster. But this can fail (take too much time) if node fencing is
failing for every reason etc... 

(The reasons I'm thinking about this solution are various: Need that
many clients can access the same device via iscsi (like using oVirt for
virtualization deployment) and trying to reuse an existent non iscsi
storage etc...)

> 
> 
> > export the same block devices from 2 or more machines (connected to a
> > common storage). Then the iscsi initiators on other client machines will
> > use dm-multipath di balance I/O and to survive the lost of one or more
> > scsi target machines.
> 
> You don't need multiple tgt daemons to use dm-multipath on the
> initiator side.

I know I can access a single tgtd over 2 different network interfaces/IP
and get multipath over them.
But my idea was to have multiple machines exporting the same device in
an active/active way, so a tgtd running for every machine.

> 
> 
> > I'm already doing something similar with gnbd and now I'd like to do the
> > same with stgt via iscsi.
> 
> You runs multiple tgt daemons on one machine and they export the same
> file to initiators?
> 
> If so, you can't do that easily because SCSI protocol is not about
> only READ and WRITE commands. For example, how can tgt daemons handle
> RESERVE command?

?In fact I didn't thought about the RESERVE command (?for the moment I
won't need SCSI reservation). Gnbd uses it's own fencing to avoid the
access to the exported devices instead of scsi reservation but requires
the gnbd clients to be part of the same cluster of the gnbd servers).

Probably a solution would be that the N tgtd should communicate each
other the reservation state.
?
BTW. does/will tgtd support SCSI3 persistent reservation? Is/will the
state be saved on local file system? Or how is it implemented?

Thanks!
Bye!