[sheepdog] [PATCH] sheep: implement a correct detection of object cache pushing completion

[Ing. Luca Lazzeroni - Trend Servizi Srl]

Il 24/07/2013 12:46, Hitoshi Mitake ha scritto:
> Current implementation for pushing object cache would have a bug. In
> object_cache_push(), the variable oc->push_count is incremented in the
> loop of queuing work. This can produce a non deterministic bug when
> a sequence of scheduling like below happens:

Hi Hitoshi,
I've tried to apply the patch as you suggested, but it isn't applyable 
to sheepdog-stable since:

object_cache.c:888:1: error: ‘struct object_cache’ has no member named 
‘dirty_count’

Maybe code needs some refactor for succesfuly back-port it to 0.6-stable..

Thank you,


> 1. object_cache_push() creates a work and queue it to
>     sys->oc_push_wqueue (oc->push_count == 1)
> 2. Before queuing next work, a worker thread of oc_push_wqueue wakes
>     up and pushes the object
> 3. After completion of pushing, the worker thread decrements
>     oc->push_count atomically. The thread calls eventfd_write() because
>     oc->push_count is 0.
> 4. The thread which calls object_cache_push() detects an invalid
>     completion after its queuing iteration.
>
> This sequence can produce an invalid early finish of flush request
> from VMs, especially when a number of cached object is large.
>
> Cc: Luca Lazzeroni <luca at gvnet.it>
> Signed-off-by: Hitoshi Mitake <mitake.hitoshi at lab.ntt.co.jp>
> ---
>   sheep/object_cache.c |    3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/sheep/object_cache.c b/sheep/object_cache.c
> index 1bd2a59..27d544d 100644
> --- a/sheep/object_cache.c
> +++ b/sheep/object_cache.c
> @@ -932,11 +932,12 @@ static int object_cache_push(struct object_cache *oc)
>   		unlock_cache(oc);
>   		return SD_RES_SUCCESS;
>   	}
> +
> +	uatomic_set(&oc->push_count, uatomic_read(&oc->dirty_count));
>   	list_for_each_entry_safe(entry, t, &oc->dirty_head, dirty_list) {
>   		struct push_work *pw;
>   
>   		get_cache_entry(entry);
> -		uatomic_inc(&oc->push_count);
>   		pw = xzalloc(sizeof(struct push_work));
>   		pw->work.fn = do_push_object;
>   		pw->work.done = push_object_done;