On 09/20/2011 04:30 PM, MORITA Kazutaka wrote: > Looks great, but there seems to be some other cases we need to > consider. For example: > > 1. Start Sheepdog with three daemons. > $ for i in 0 1 2; do sheep /store/$i -z $i -p 700$i; sleep 1; done > $ collie cluster format > $ collie cluster info > Cluster status: running > > Creation time Epoch Nodes > 2011-09-20 16:43:10 1 [10.68.14.1:7000, 10.68.14.1:7001, 10.68.14.1:7002] > > 2. Then, kill sheep daemons, and start again in the same order. > > $ for i in 0 1 2; do pkill -f "sheep /store/$i"; sleep 1; done > $ for i in 0 1 2; do ./sheep/sheep /store/$i -z $i -p 700$i; sleep 1; done > $ collie cluster info > Cluster status: running > > Creation time Epoch Nodes > 2011-09-20 16:43:10 2 [10.68.14.1:7000] > 2011-09-20 16:43:10 1 [10.68.14.1:7000, 10.68.14.1:7001, 10.68.14.1:7002] > > The first daemon regards the other two nodes as left nodes, and starts > working. > > 3. Start the other two nodes again. > > $ for i in 1 2; do ./sheep/sheep /store/$i -z $i -p 700$i; sleep 1; done > $ collie cluster info > Cluster status: running > > Creation time Epoch Nodes > 2011-09-20 16:43:10 4 [10.68.14.1:7000, 10.68.14.1:7001, 10.68.14.1:7002] > 2011-09-20 16:43:10 3 [10.68.14.1:7000, 10.68.14.1:7001] > 2011-09-20 16:43:10 2 [10.68.14.1:7000] > 2011-09-20 16:43:10 1 [10.68.14.1:7000, 10.68.14.1:7001, 10.68.14.1:7002] > $ collie cluster info -p 7001 > Cluster status: running > > Creation time Epoch Nodes > 2011-09-20 16:43:10 4 [10.68.14.1:7000, 10.68.14.1:7001, 10.68.14.1:7002] > 2011-09-20 16:43:10 3 [10.68.14.1:7000, 10.68.14.1:7001] > 2011-09-20 16:43:10 2 [10.68.14.1:7000, 10.68.14.1:7002] > 2011-09-20 16:43:10 1 [10.68.14.1:7000, 10.68.14.1:7001, 10.68.14.1:7002] > $ collie cluster info -p 7002 > Cluster status: running > > Creation time Epoch Nodes > 2011-09-20 16:43:10 4 [10.68.14.1:7000, 10.68.14.1:7001, 10.68.14.1:7002] > 2011-09-20 16:43:10 3 [10.68.14.1:7000, 10.68.14.1:7001] > 2011-09-20 16:43:10 2 [10.68.14.1:7001, 10.68.14.1:7002] > 2011-09-20 16:43:10 1 [10.68.14.1:7000, 10.68.14.1:7001, 10.68.14.1:7002] > > The epoch informations become inconsistent. It is because the first > node overwrote the epochs in the other nodes. Similar situations > could happen if we start from the daemon which doesn't have the latest > epoch. > > We can get away with claiming that this doesn't happen if the > administrator is careful enough. But is there any good idea to solve > this problem? > I am really puzzled by the semantics of 'collie cluster info'...from the code, it tries to get the local epoch information, however, by semantics it suggests this command should get the cluster information. every node may have its own history, and have chances to have *different* epoch history with other nodes. So, I think we should get the same epoch history on any node of cluster. Kazutaka, how do you think to get the cluster info only from single node(would be master node in my opionion)?If possible, how do we deal with the local epoch that it is not master node? if not, we would suffer epoch inconsistency as you met. we cannot get rid of this inconsistency in *every* cases. Thanks, Yuan |