[Sheepdog] [PATCH v3 1/7] sheep: add transient failure detection

[HaiTing Yao]

On Thu, May 3, 2012 at 3:37 AM, MORITA Kazutaka
<morita.kazutaka at gmail.com>wrote:

> At Wed,  2 May 2012 15:12:49 +0800,
> yaohaiting.wujue at gmail.com wrote:
> >
> > From: HaiTing Yao <wujue.yht at taobao.com>
> >
> > Sometimes we need node can be back in a while.
> >
> > When we need this:
> >
> > 1, restart sheepdog daemon for ugrade or other purpose
> >
> > 2, the corosync driver lose its token for a short while
>
> This is a corosync specific problem, and should be handled by changing
> parameters in corosync.conf, I think.
>

For cluster storage, storage system should deal with temporary node or
network failures. It can not assume the cluster is always stable. Changing
parameter of corosync can not eliminate the temporay node failue because of
some protocol reasons. I am not sure zookeeper and other drivers have same
problems, but zookeeper also has the timeout that zookeeper server can not
commnunicate with the node. I think it alos can not avoid the problem on
some conditions.

I tried to implement the similar solution with Amzon Dynamo for temporary
node or network failures. Perhaps I  should keep the hinted handoff of
failed node on the VM hosted node, so I reused the object cache to keep
the hinted handoff. With the cache, the I/O will not be blocked.


>
> So I think the main benefit of this patchset is to allow us to restart
> sheep daemons without changing node membership, but what's the reason
> you want to avoid temporal membership changes?  Sheepdog blocks write
> I/Os when it cannot create full replicas, so basically we should
> remove the failed nodes from node membership ASAP.
>

Restarting the daemon will lead to two times of data recovery. If we
upgrade the cluster with much data, the lazy repair is useful.

When we format the cluster, we can specify the temorary failure detection
on/off. When it is on, there is an optional lazy reparr for eager repair.

Thanks
Haiti

>
> Thanks,
>
> Kazutaka
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wpkg.org/pipermail/sheepdog/attachments/20120503/95e9f5da/attachment-0003.html>