[sheepdog-users] i/o stuck on recovery

[Maxim Terletskiy]

Hi!

Today I've been restarted sheep process on one of my nodes with data in 
test cluster and noticed that I/O stuck on all VMs for almost 10 mins. 
Initially I've tried to do "service sheepdog restart" to change sheep 
settings(disable -D flag), but process doesn't started because of 
"previous session in zookeeper found" or so. Waited for 30 secs, 
executed "service sheepdog start", sheep started, "dog node recovery" 
showed my 3 nodes recovering. I had "pv /dev/vda > /dev/null" in 
terminal on one of my VMs. Right after restarting sheep pv stucked. 
After 5 minutes when "dog node recovery" showed me only one node 
recovering pv still was stucked. And only after 3-4 minutes later pv 
continued to reading from disk. Is it normal behavior?

  Have 3 nodes with data in the cluster, replication 2x, about 7Tb data 
assuming to "dog node info". Sheep 0.8.0, options:
/var/lib/sheepdog,/mnt/sheep_sda4,/mnt/sheep_sdb4,/mnt/sheep_sdc4,/mnt/sheep_sdd4,/mnt/sheep_sde4,/mnt/sheep_sdf4 
-c zookeeper:1.2.3.1:2181,1.2.3.2:2181,1.2.3.3:2181,timeout=30 -i 
host=6.7.8.9,port=7001 -y 1.2.3.4

Additionally one node have -D flag in options.