[Sheepdog] [PATCH v2 1/2] force some callbacks of cluster request running in worker thread

[Liu Yuan]

On 04/28/2012 02:52 PM, Li Wenpeng wrote:

> From: levin li <xingke.lwp at taobao.com>
> 
> add a flag SD_FLAG_CMD_WORKER to vdi_req, when a node receives a
> notified request with this flag, it will force process_main()
> running in the __sd_notify() which is running in worker thread,
> sometimes we need to notify a request to all the node, but we need
> it to run in worker thread, in this case it's useful.
> 
> Signed-off-by: levin li <xingke.lwp at taobao.com>
> ---
>  include/sheepdog_proto.h |    1 +
>  sheep/group.c            |   14 +++++++++++++-
>  2 files changed, 14 insertions(+), 1 deletion(-)
> 
> diff --git a/include/sheepdog_proto.h b/include/sheepdog_proto.h
> index 11c2c7c..50308ab 100644
> --- a/include/sheepdog_proto.h
> +++ b/include/sheepdog_proto.h
> @@ -34,6 +34,7 @@
>  #define SD_FLAG_CMD_WRITE    0x01
>  #define SD_FLAG_CMD_COW      0x02
>  #define SD_FLAG_CMD_CACHE    0x04
> +#define SD_FLAG_CMD_WORKER   0x08
>  



This header is for qemu and sheepdog communication. Please use 0x0800 in
include/sheep.h

>  #define SD_RES_SUCCESS       0x00 /* Success */

>  #define SD_RES_UNKNOWN       0x01 /* Unknown error */
> diff --git a/sheep/group.c b/sheep/group.c
> index ea02602..4fd30cb 100644
> --- a/sheep/group.c
> +++ b/sheep/group.c
> @@ -616,6 +616,17 @@ static void update_cluster_info(struct join_message *msg,
>  
>  static void __sd_notify(struct event_struct *cevent)
>  {
> +	struct work_notify *w = container_of(cevent, struct work_notify, cev);
> +	struct vdi_op_message *msg = w->msg;
> +	const struct sd_req *req = (const struct sd_req *)&msg->req;
> +	int ret = msg->rsp.result;
> +	struct sd_op_template *op = get_sd_op(req->opcode);
> +
> +	if (ret == SD_RES_SUCCESS && has_process_main(op) &&
> +			req->flags & SD_FLAG_CMD_WORKER) {
> +		ret = do_process_main(op, req, (struct sd_rsp *)&msg->rsp, msg->data);
> +		msg->rsp.result = ret;
> +	}
>  }
>  
>  static void __sd_notify_done(struct event_struct *cevent)
> @@ -626,7 +637,8 @@ static void __sd_notify_done(struct event_struct *cevent)
>  	int ret = msg->rsp.result;
>  	struct sd_op_template *op = get_sd_op(msg->req.opcode);
>  
> -	if (ret == SD_RES_SUCCESS && has_process_main(op))
> +	if (ret == SD_RES_SUCCESS && has_process_main(op) &&
> +			!(msg->req.flags & SD_FLAG_CMD_WORKER))
>  		ret = do_process_main(op, (const struct sd_req *)&msg->req,
>  				      (struct sd_rsp *)&msg->rsp, msg->data);
>  


I think its time to retrofit process_main(). it doesn't necessarily run
in main thread. Actually, we should try to put it in the event queue as
possible as we can, to offload the main thread.

I'd suggest a new name scheme:
process_work() -> process_top(), which runs only local node
process_main() -> process_bottom(), which runs in all nodes, either in
worker or in main thread.

Thanks,
Yuan