[Sheepdog] [PATCH v2 6/6] sheep: transient failure - deal with I/O to failed node

[yaohaiting.wujue at gmail.com]

From: HaiTing Yao <wujue.yht at taobao.com>

If the VDI has cache, there is no wrong with users' I/O because users'
I/O reach cache and return. When flush the cache, just return success
and wait for the next flush.

If th VDI has not cache, the users' I/O will be blocked because of the
retry. The block ends when the node comes back or the node be removed
from cluster when templeave time is up.

Signed-off-by: HaiTing Yao <wujue.yht at taobao.com>
---
 sheep/group.c        |   11 +++++++++++
 sheep/object_cache.c |    4 ++--
 sheep/sheep_priv.h   |    2 ++
 3 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/sheep/group.c b/sheep/group.c
index 69bdc82..e1a1197 100644
--- a/sheep/group.c
+++ b/sheep/group.c
@@ -326,6 +326,17 @@ static struct sd_node *find_entry_epoch(struct sd_node *entry,
 	return NULL;
 }
 
+int cluster_have_failed_node(void)
+{
+	int i = 0;
+
+	for (i = 0; i < sys->nr_nodes; i++)
+		if (sys->nodes[i].status != NODE_STATUS_NORMAL)
+			return 1;
+
+	return 0;
+}
+
 static int cluster_sanity_check(struct sd_node *entries,
 			     int nr_entries, uint64_t ctime, uint32_t epoch)
 {
diff --git a/sheep/object_cache.c b/sheep/object_cache.c
index df9ab49..bd1e0e0 100644
--- a/sheep/object_cache.c
+++ b/sheep/object_cache.c
@@ -466,8 +466,8 @@ int object_cache_push(struct object_cache *oc)
 	struct object_cache_entry *entry, *t;
 	int ret = SD_RES_SUCCESS;
 
-	if (node_in_recovery())
-		/* We don't do flushing in recovery */
+	if (node_in_recovery() || cluster_have_failed_node())
+		/* We don't do flushing in recovery or with failed node */
 		return SD_RES_SUCCESS;
 
 	list_for_each_entry_safe(entry, t, &oc->dirty_list, list) {
diff --git a/sheep/sheep_priv.h b/sheep/sheep_priv.h
index e18c696..848f19e 100644
--- a/sheep/sheep_priv.h
+++ b/sheep/sheep_priv.h
@@ -302,6 +302,8 @@ void resume_recovery_work(void);
 int is_recoverying_oid(uint64_t oid);
 int node_in_recovery(void);
 
+int cluster_have_failed_node(void);
+
 int write_object(struct sd_vnode *e,
 		 int vnodes, int zones, uint32_t node_version,
 		 uint64_t oid, char *data, unsigned int datalen,
-- 
1.7.1