[sheepdog-users] qemu/KVM+sheepdog HA problem when one sheep leaves the cluster

[Zhaohui Yang]

We have an openstack cluster whose compute nodes have sheepdog installed
and configured the "standard" way as sheepdog documentation described. That
is, the qemu/KVM virtual machines uses sheepdog VDI by attaching to a the
local sheepdog gateway through the qemu block driver.

When one sheep decided it has lost connection with the rest of cluster
nodes (caused by software problem), all VMs on the same node instantly
"lost" their VDI and cannot work. In this regard the storage layer is not
fully HA. We would expect a better situation where the VMs still work fine
as long as sheeps on other nodes are functioning and network is up.

We found a "kind of " solution to this in the following article, where the
sheepdog storage cluster is deployed separately from the compute cluster,
connected via a switch, through a iSCSI interface exposed by tgtd sitting
before sheepdog. The client has to use special iSCSI multipath-tool to be
able to fail over to another tgtd upon sheepdog node failure.
http://events.linuxfoundation.org/sites/events/files/slides/COJ2015_Sheepdog_20150604.pdf

However this solution add 2 more layers to the complexity - the iSCSI
multipath-tool on client side and tgtd on the server side. Also the
performance will degrade as everything has to go through the network and
iSCSI simulation. We surely don't want to go this direction if there are
simpler solutions we are not aware of - e.g. patches to sheepdog or the
qemu block driver that perform automatic failovers.

Please point a link to us if such solutions exists, or share your idea how
to avoid "VM losing VDI when sheep on same machine leaves cluster". The
version of sheepdog we are using is 0.9.0.


Thanks and Regards,

Yang, Zhaohui
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wpkg.org/pipermail/sheepdog-users/attachments/20150814/95c14c7d/attachment.html>