[Sheepdog] Sheepdog doesn't work correctly.

[MORITA Kazutaka]

At Tue, 29 Nov 2011 07:04:09 +0000 (UTC),
TaoLin wrote:
> 
> 
> 
> After this, I tried to add a third node(10.210.77.13):
> 
> root at node2:/home/sae# sheep /home/sae/sheep
> 
> root at node2:/home/sae# collie node list
> 
>    Idx - Host:Port          Vnodes       Zone
> 
> ---------------------------------------------
> 
>      0 - 10.210.77.11:7000   	64  189649418
> 
>      1 - 10.210.77.13:7000   	64  223203850
> 
> 
> 
> 
>  At this time, the vm came to a complete halt and each node 
> said as below:
> 
> 10.210.77.12 said:
> [root at novanode1 /home/sae]# collie node list
> 
>    Idx - Host:Port          Vnodes       Zone
> 
> ---------------------------------------------
> 
>      0 - 10.210.77.11:7000   	64  189649418
> 
> [root at novanode1 /home/sae]# collie cluster info
> 
> Cluster status: IO has halted as there are too few living nodes
> 
> 
> 
> Cluster created at Tue Nov 29 13:28:53 2011
> 
> 
> 
> Epoch Time           Version
> 
> 4441234-07-22 20:29:28      3 []
> 
> 2011-11-29 13:53:00      2 [10.210.77.11:7000, 10.210.77.12:7000]
> 
> 2011-11-29 13:28:53      1 [10.210.77.12:7000]
> 
> 
> 
> 
> 10.210.77.11 said:
> root at novacontroler:/home/sae/sheep# collie node list
> 
>    Idx - Host:Port          Vnodes       Zone
> 
> ---------------------------------------------
> 
>      0 - 10.210.77.11:7000   	64  189649418
> 
>      1 - 10.210.77.13:7000   	64  223203850
> 
> root at novacontroler:/home/sae/sheep# collie cluster info
> 
> Cluster status: IO has halted as there are too few living nodes
> 
> 
> 
> Cluster created at Wed Nov 30 02:28:53 2011
> 
> 
> 
> Epoch Time           Version
> 
> 2011-11-29 14:51:39      4 [10.210.77.11:7000, 10.210.77.13:7000]
> 
> 2011-11-29 14:10:31      3 [10.210.77.11:7000]
> 
> 2011-11-29 13:52:57      2 [10.210.77.11:7000, 10.210.77.12:7000]
> 
> 2011-11-29 13:52:57      1 [10.210.77.12:7000]
> 
> 
> 
> 10.210.77.13 said:
> root at node2:/home/sae# collie node list
> 
>    Idx - Host:Port          Vnodes       Zone
> 
> ---------------------------------------------
> 
>      0 - 10.210.77.11:7000   	64  189649418
> 
>      1 - 10.210.77.13:7000   	64  223203850
> 
> root at node2:/home/sae# collie cluster info
> 
> Cluster status: IO has halted as there are too few living nodes
> 
> 
> 
> Cluster created at Wed Nov 30 02:28:53 2011
> 
> 
> 
> Epoch Time           Version
> 
> 2011-11-29 14:51:39      4 [10.210.77.11:7000, 10.210.77.13:7000]
> 
> 2011-11-29 14:51:39      3 [10.210.77.11:7000]
> 
> 2011-11-29 13:52:57      2 [10.210.77.11:7000, 10.210.77.12:7000]
> 
> 2011-11-29 13:52:57      1 [10.210.77.12:7000]

From your logs you sent to me:
  sheep11.log:Nov 28 13:32:15 __recover_one(1282) failed to open /home/sae/sheep/obj/00000004/00d4a4d200000397.tmp: No space left on device

Perhaps, is your disk full?

I've pushed some fixes to handle I/O errors correctly.  Can you try
with the devel branch again?

Thanks,

Kazutaka