At Tue, 29 Nov 2011 07:04:09 +0000 (UTC), TaoLin wrote: > > > > After this, I tried to add a third node(10.210.77.13): > > root at node2:/home/sae# sheep /home/sae/sheep > > root at node2:/home/sae# collie node list > > Idx - Host:Port Vnodes Zone > > --------------------------------------------- > > 0 - 10.210.77.11:7000 64 189649418 > > 1 - 10.210.77.13:7000 64 223203850 > > > > > At this time, the vm came to a complete halt and each node > said as below: > > 10.210.77.12 said: > [root at novanode1 /home/sae]# collie node list > > Idx - Host:Port Vnodes Zone > > --------------------------------------------- > > 0 - 10.210.77.11:7000 64 189649418 > > [root at novanode1 /home/sae]# collie cluster info > > Cluster status: IO has halted as there are too few living nodes > > > > Cluster created at Tue Nov 29 13:28:53 2011 > > > > Epoch Time Version > > 4441234-07-22 20:29:28 3 [] > > 2011-11-29 13:53:00 2 [10.210.77.11:7000, 10.210.77.12:7000] > > 2011-11-29 13:28:53 1 [10.210.77.12:7000] > > > > > 10.210.77.11 said: > root at novacontroler:/home/sae/sheep# collie node list > > Idx - Host:Port Vnodes Zone > > --------------------------------------------- > > 0 - 10.210.77.11:7000 64 189649418 > > 1 - 10.210.77.13:7000 64 223203850 > > root at novacontroler:/home/sae/sheep# collie cluster info > > Cluster status: IO has halted as there are too few living nodes > > > > Cluster created at Wed Nov 30 02:28:53 2011 > > > > Epoch Time Version > > 2011-11-29 14:51:39 4 [10.210.77.11:7000, 10.210.77.13:7000] > > 2011-11-29 14:10:31 3 [10.210.77.11:7000] > > 2011-11-29 13:52:57 2 [10.210.77.11:7000, 10.210.77.12:7000] > > 2011-11-29 13:52:57 1 [10.210.77.12:7000] > > > > 10.210.77.13 said: > root at node2:/home/sae# collie node list > > Idx - Host:Port Vnodes Zone > > --------------------------------------------- > > 0 - 10.210.77.11:7000 64 189649418 > > 1 - 10.210.77.13:7000 64 223203850 > > root at node2:/home/sae# collie cluster info > > Cluster status: IO has halted as there are too few living nodes > > > > Cluster created at Wed Nov 30 02:28:53 2011 > > > > Epoch Time Version > > 2011-11-29 14:51:39 4 [10.210.77.11:7000, 10.210.77.13:7000] > > 2011-11-29 14:51:39 3 [10.210.77.11:7000] > > 2011-11-29 13:52:57 2 [10.210.77.11:7000, 10.210.77.12:7000] > > 2011-11-29 13:52:57 1 [10.210.77.12:7000] From your logs you sent to me: sheep11.log:Nov 28 13:32:15 __recover_one(1282) failed to open /home/sae/sheep/obj/00000004/00d4a4d200000397.tmp: No space left on device Perhaps, is your disk full? I've pushed some fixes to handle I/O errors correctly. Can you try with the devel branch again? Thanks, Kazutaka |