[sheepdog] [PATCH v2 1/2] cluster: add the distributed lock implemented by zookeeper for object-storage

[Liu Yuan]

On Fri, Nov 29, 2013 at 05:39:54PM +0800, Robin Dong wrote:
> 2013/11/29 Liu Yuan <namei.unix at gmail.com>
> 
> > On Fri, Nov 29, 2013 at 05:10:31PM +0800, Robin Dong wrote:
> > > Implement the distributed lock by zookeeper (refer:
> > http://zookeeper.apache.org/doc/trunk/recipes.html)
> > > The routine is:
> > >         1. create a seq-ephemeral znode in lock directory (use lock-id
> > as dir name)
> > >         2. get smallest file path as owner of the lock; the other thread
> > wait on a pthread_mutex_t
> > >         3. if owner of the lock release it (or the owner is killed by
> > accident), zookeeper will
> > >            trigger zk_watch() which will wake up all waiting threads to
> > compete new owner of the lock
> > >
> > > We add ->local_lock for dist_mutex to avoid many threads in one sheepdog
> > daemon to create too many files
> > > in a lock directory.
> > >
> > > Signed-off-by: Robin Dong <sanbai at taobao.com>
> > > ---
> > >  sheep/cluster.h           |   15 +++++
> > >  sheep/cluster/zookeeper.c |  140
> > ++++++++++++++++++++++++++++++++++++++++++++-
> > >  2 files changed, 154 insertions(+), 1 deletions(-)
> > >
> > > diff --git a/sheep/cluster.h b/sheep/cluster.h
> > > index 81b5ae4..0aa058c 100644
> > > --- a/sheep/cluster.h
> > > +++ b/sheep/cluster.h
> > > @@ -23,6 +23,13 @@
> > >  #include "sheep.h"
> > >  #include "config.h"
> > >
> > > +struct dist_mutex {
> > > +     uint32_t id;            /* id of this mutex */
> > > +     pthread_mutex_t wait;
> > > +     pthread_mutex_t local_lock;
> > > +     char ephemeral_path[MAX_NODE_STR_LEN];
> > > +};
> > > +
> > >  /*
> > >   * maximum payload size sent in ->notify and ->unblock, it should be
> > large
> > >   * enough to support COROSYNC_MAX_NODES * struct sd_node
> > > @@ -109,6 +116,14 @@ struct cluster_driver {
> > >       int (*unblock)(void *msg, size_t msg_len);
> > >
> > >       /*
> > > +      * A distributed mutex lock to avoid race condition when using
> > swift
> > > +      * interface to add/delete/list object.
> >
> > This is not a swift only operation. Instead it is now as a generic locking
> > service.
> >
> > > +      */
> > > +     int (*init_mutex)(struct dist_mutex *mutex, uint32_t id);
> > > +     void (*lock_mutex)(struct dist_mutex *mutex);
> > > +     void (*unlock_mutex)(struct dist_mutex *mutex);
> >
> > Please explain every function clearly as other interface does.
> >
> > Any reason to call it mutex? I'd suggest shorter name as
> >
> 
> Because it is only mutex now, I will add dist_rwlock,dist_selock or some
> other kind of locks in the future.

I think we only need one interface. Currently it is a spinlock, and later we
can rework it as rw_lock, which is an optimiaztion of locking internaling,
instead of provide unnecessary locking interfaces.

Thanks
Yuan