[sheepdog] [PATCH v5 1/6] zookeeper: fixed concurrent startup error

[Liu Yuan]

On 06/21/2013 03:00 AM, Kai Zhang wrote:
> Current implementation of zookeeper driver has a risk when multiple sheep
> start up concurrently.
> 
> Consider the following situation:
> 1. There is a 3 node cluster: sheep1, sheep2, sheep3.
> 2. Both sheep2 and sheep3 leave cluster.
> 3. Both sheep2 and sheep3 start up after previous zookeeper session timeout.
> 4. Sheep1 leaves the cluster before sheep2 and sheep3 receiving join requests
>    from zookeeper.
> 5. When sheep2 and sheep3 receive the join requests, both of them assume they
>    are master due to zk_member_empty() returns true.
> 
> The new implementation is following the standard way that zookeeper sugguests.
> At start up, each sheep creates a ephemeral sequencial node under MASTER_ZNODE.
> The sheep who has the least number will be the master.
> 
> However, due to the join logic in zookeeper, there is a risk of hang during
> join cluster.
> This is because zk_compete_master and add_join_event are two seperate operations,
> and there is no transection protection.
> For example sheep1 got a less sequence number for master competition than sheep2,
> but a larger sequence number for join request event.
> So if the current master sheep3 quit before handling sheep2's join request,
> sheep2 will hang on joining cluster.
> 
> To solve this problem, we force sheep send join request only when it can confirm
> its a sheep with a less sequence number of master competition has joined.

Kazutaka, if your patches remove master completely, this patch won't be
necessary, right? If so, I'd like to merge your patch set first.

Thanks,
Yuan