[sheepdog] [PATCH] sheep: implement a correct detection of object cache pushing completion

[Hitoshi Mitake]

Current implementation for pushing object cache would have a bug. In
object_cache_push(), the variable oc->push_count is incremented in the
loop of queuing work. This can produce a non deterministic bug when
a sequence of scheduling like below happens:

1. object_cache_push() creates a work and queue it to
   sys->oc_push_wqueue (oc->push_count == 1)
2. Before queuing next work, a worker thread of oc_push_wqueue wakes
   up and pushes the object
3. After completion of pushing, the worker thread decrements
   oc->push_count atomically. The thread calls eventfd_write() because
   oc->push_count is 0.
4. The thread which calls object_cache_push() detects an invalid
   completion after its queuing iteration.

This sequence can produce an invalid early finish of flush request
from VMs, especially when a number of cached object is large.

Cc: Luca Lazzeroni <luca at gvnet.it>
Signed-off-by: Hitoshi Mitake <mitake.hitoshi at lab.ntt.co.jp>
---
 sheep/object_cache.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/sheep/object_cache.c b/sheep/object_cache.c
index 1bd2a59..27d544d 100644
--- a/sheep/object_cache.c
+++ b/sheep/object_cache.c
@@ -932,11 +932,12 @@ static int object_cache_push(struct object_cache *oc)
 		unlock_cache(oc);
 		return SD_RES_SUCCESS;
 	}
+
+	uatomic_set(&oc->push_count, uatomic_read(&oc->dirty_count));
 	list_for_each_entry_safe(entry, t, &oc->dirty_head, dirty_list) {
 		struct push_work *pw;
 
 		get_cache_entry(entry);
-		uatomic_inc(&oc->push_count);
 		pw = xzalloc(sizeof(struct push_work));
 		pw->work.fn = do_push_object;
 		pw->work.done = push_object_done;
-- 
1.7.10.4