[sheepdog] [PATCH v3 2/2] cluster driver: handle pending block/notify event during reconnect

[Liu Yuan]

On Mon, Jul 08, 2013 at 09:07:15PM -0700, Kai Zhang wrote:
> Current implementation of reconnection doesn't handle pending block/notify
> event.
> 
> It is easy to handle notify event by sending it again.
> 
> However, it is a little bit complex for block event.
> This is because a block event need 4 steps.
> 1. in queue_cluster_request(), send block event by sys->cdrv->block(), and
>   add to pending_block_list.
> 2. in sd_block_handler(), queue the event to work queue of 'block' thread.
> 3. in cluster_op_done(), send unblock event by sys->cdrv->unblock().
> 4. in sd_notify_handler(), remove it from pending_block_list.
> 
> And step 1 and 3 contains broadcast operations.
> So we have to know which step has been done for a pending block event.
> 
> If step 1 has been done, we can re-queue it simply. (Any block event which sent
> by this node have been removed due to the leave event)
> If step 2 has been done, the event is handling by another thread. We have to mark
> it as 'drop' so that it will be dropped when cluster_op_done() is called later.
> if step 3 has been done, we should call sd_notify_handler() manually to finish
> it.
> 
> Signed-off-by: Kai Zhang <kyle at zelin.io>
> ---
>  sheep/group.c      |   76 ++++++++++++++++++++++++++++++++++++++++++++++++++--
>  sheep/sheep_priv.h |    8 ++++++
>  2 files changed, 82 insertions(+), 2 deletions(-)
> 
> diff --git a/sheep/group.c b/sheep/group.c
> index e5585b6..0d4e924 100644
> --- a/sheep/group.c
> +++ b/sheep/group.c
> @@ -234,6 +234,9 @@ static void cluster_op_done(struct work *work)
>  	size_t size;
>  	int ret;
>  
> +	if (req->status == REQUEST_DROPPED)
> +		goto drop;
> +
>  	sd_dprintf("%s (%p)", op_name(req->op), req);
>  
>  	msg = prepare_cluster_msg(req, &size);
> @@ -251,6 +254,13 @@ static void cluster_op_done(struct work *work)
>  	}
>  
>  	free(msg);
> +	req->status = REQUEST_DONE;
> +	return;
> +drop:
> +	list_del(&req->pending_list);
> +	req->rp.result = SD_RES_CLUSTER_ERROR;
> +	put_request(req);
> +	cluster_op_running = false;
>  }
>  
>  /*
> @@ -280,6 +290,7 @@ bool sd_block_handler(const struct sd_node *sender)
>  	req->work.done = cluster_op_done;
>  
>  	queue_work(sys->block_wqueue, &req->work);
> +	req->status = REQUEST_QUEUED;
>  	return true;
>  }
>  
> @@ -324,7 +335,7 @@ void queue_cluster_request(struct request *req)
>  
>  		free(msg);
>  	}
> -
> +	req->status = REQUEST_INIT;
>  	return;
>  error:
>  	req->rp.result = ret;
> @@ -922,6 +933,67 @@ static int send_join_request(struct sd_node *ent)
>  	return ret;
>  }
>  
> +static void requeue_cluster_request(void)
> +{
> +	struct request *req, *p;
> +	struct vdi_op_message *msg;
> +	size_t size;
> +
> +	list_for_each_entry_safe(req, p, main_thread_get(pending_notify_list),
> +				 pending_list) {
> +		sd_dprintf("found an pending notify request, op: %s",
> +			   op_name(req->op));
> +		/*
> +		 * notify has been sent but sd_notify_handler is never called,
> +		 * re-queue it
> +		 */

Well, I think we should handle it as blocking event

1 ->notify() was called successfully but sd_notify_handler due to timeout
   - simply call sd_notify_handler() to finish the request.
2 ->notify isn't called yet.
   - just requeue it

No?

Better rephrased as, for e.g,

---
->notify() was called and succeeded but sd_notify_handler isn't called yet from
cluster driver due to driver session timeout ... 
---


> +		list_del(&req->pending_list);
> +		queue_cluster_request(req);
> +	}
> +
> +	list_for_each_entry_safe(req, p, main_thread_get(pending_block_list),
> +				 pending_list) {
> +		switch (req->status) {
> +		case REQUEST_INIT:
> +			/* this request has never been executed, re-queue it */
> +			sd_dprintf("requeue a block request, op: %s",
> +				   op_name(req->op));
> +			list_del(&req->pending_list);
> +			queue_cluster_request(req);
> +			break;
> +		case REQUEST_QUEUED:
> +			/*
> +			 * This request is being handled by the 'block' thread
> +			 * and ->unblock() isn't called yet. We can't call
> +			 * ->unblock thereafter because other sheep has
> +			 * unblocked themselves due to cluster driver session
> +			 * timeout. Mark it as dropped to stop cluster_op_done()
> +			 * from calling ->unblock.
> +			 */
> +			sd_dprintf("drop pending block request, op: %s",
> +				   op_name(req->op));
> +			req->status = REQUEST_DROPPED;
> +			break;
> +		case REQUEST_DONE:
> +			/*
> +			 * ->unblock() was called and succeeded but after that
> +			 * this node session-timeouted and sd_notify_handler
> +			 * wasn't called from unblock event handler in cluster
> +			 * driver. We manually call sd_notify_handler to finish
> +			 * the request.'
> +			 */

Remove trailing '.

Thanks
Yuan