[sheepdog-users] i/o stuck on recovery

[Liu Yuan]

On Sun, Sep 14, 2014 at 05:57:54PM +0200, Maxim Terletskiy wrote:
> Hi!
> 
> Today I've been restarted sheep process on one of my nodes with data
> in test cluster and noticed that I/O stuck on all VMs for almost 10
> mins. Initially I've tried to do "service sheepdog restart" to
> change sheep settings(disable -D flag), but process doesn't started
> because of "previous session in zookeeper found" or so. Waited for
> 30 secs, executed "service sheepdog start", sheep started, "dog node
> recovery" showed my 3 nodes recovering. I had "pv /dev/vda >
> /dev/null" in terminal on one of my VMs. Right after restarting
> sheep pv stucked. After 5 minutes when "dog node recovery" showed me
> only one node recovering pv still was stucked. And only after 3-4
> minutes later pv continued to reading from disk. Is it normal
> behavior?
> 
>  Have 3 nodes with data in the cluster, replication 2x, about 7Tb
> data assuming to "dog node info". Sheep 0.8.0, options:
> /var/lib/sheepdog,/mnt/sheep_sda4,/mnt/sheep_sdb4,/mnt/sheep_sdc4,/mnt/sheep_sdd4,/mnt/sheep_sde4,/mnt/sheep_sdf4
> -c zookeeper:1.2.3.1:2181,1.2.3.2:2181,1.2.3.3:2181,timeout=30 -i
> host=6.7.8.9,port=7001 -y 1.2.3.4
> 
> Additionally one node have -D flag in options.
> 

This version uses single thread for recovery and the node that just rejoined
woud be bottleneck of recovery and negtively slow down the *all* the guests
because of our data distribution algorithm. The reason behind is that all
the guests would have data to be recovered on the new node and wait for the
compleletion of recovery.

For now master branch use multi-threaded recocvery and would mitigate this
problem (I hope so).

Thanks
Yuan