[sheepdog] [PATCH v2 1/2] cluster: add the distributed lock implemented by zookeeper for object-storage

[Liu Yuan]

On Fri, Nov 29, 2013 at 05:10:31PM +0800, Robin Dong wrote:
> Implement the distributed lock by zookeeper (refer: http://zookeeper.apache.org/doc/trunk/recipes.html)
> The routine is:
>         1. create a seq-ephemeral znode in lock directory (use lock-id as dir name)
>         2. get smallest file path as owner of the lock; the other thread wait on a pthread_mutex_t
>         3. if owner of the lock release it (or the owner is killed by accident), zookeeper will
>            trigger zk_watch() which will wake up all waiting threads to compete new owner of the lock
> 
> We add ->local_lock for dist_mutex to avoid many threads in one sheepdog daemon to create too many files
> in a lock directory.
> 
> Signed-off-by: Robin Dong <sanbai at taobao.com>
> ---
>  sheep/cluster.h           |   15 +++++
>  sheep/cluster/zookeeper.c |  140 ++++++++++++++++++++++++++++++++++++++++++++-
>  2 files changed, 154 insertions(+), 1 deletions(-)
> 
> diff --git a/sheep/cluster.h b/sheep/cluster.h
> index 81b5ae4..0aa058c 100644
> --- a/sheep/cluster.h
> +++ b/sheep/cluster.h
> @@ -23,6 +23,13 @@
>  #include "sheep.h"
>  #include "config.h"
>  
> +struct dist_mutex {
> +	uint32_t id;		/* id of this mutex */
> +	pthread_mutex_t wait;
> +	pthread_mutex_t local_lock;
> +	char ephemeral_path[MAX_NODE_STR_LEN];
> +};
> +
>  /*
>   * maximum payload size sent in ->notify and ->unblock, it should be large
>   * enough to support COROSYNC_MAX_NODES * struct sd_node
> @@ -109,6 +116,14 @@ struct cluster_driver {
>  	int (*unblock)(void *msg, size_t msg_len);
>  
>  	/*
> +	 * A distributed mutex lock to avoid race condition when using swift
> +	 * interface to add/delete/list object.

This is not a swift only operation. Instead it is now as a generic locking
service. 

> +	 */
> +	int (*init_mutex)(struct dist_mutex *mutex, uint32_t id);
> +	void (*lock_mutex)(struct dist_mutex *mutex);
> +	void (*unlock_mutex)(struct dist_mutex *mutex);

Please explain every function clearly as other interface does.

Any reason to call it mutex? I'd suggest shorter name as

 ->init_lock()
 ->lock()
 ->unlock()

So struct dist_mutex -> struct cluster_lock

Finally, I think you need to initialize local/cluster driver's lock interface's
too, thought it would be currently empty operations for them.

Thanks
Yuan