<div dir="ltr"><br></div><div class="gmail_extra"><br><br><div class="gmail_quote">2014-06-17 9:55 GMT+02:00 Valerio Pachera <span dir="ltr"><<a href="mailto:sirio81@gmail.com" target="_blank">sirio81@gmail.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div>Hi, I try to summarize what happened on a cluster I manage.<br></div><div>It may be interesting for other administrators.<br>
</div><div>(It's a bit long, sorry).<br></div><br></div>This cluster has 4 nodes and is running Sheepdog daemon version 0.7.0_131_g88f0024.<br>
<br></div>I'm going to focus on node id0 and node id1.<br></div>These are the mount points used by sheepdog.<br><div><div><div><br>node id0<br>/dev/mapper/vg00-sheepdsk01 217G 211G 5,2G 98% /mnt/sheep/dsk01<br>
/dev/sdb1 466G 466G 76K 100% /mnt/sheep/dsk02<br>/dev/sdc1 1,9T 1,1T 755G 60% /mnt/sheep/dsk03<br><br>node id1<br>/dev/mapper/vg00-sheepdsk01 167G 103G 64G 62% /mnt/sheep/dsk01<br>/dev/sdb1 466G 466G 12K 100% /mnt/sheep/dsk02<br>
/dev/sdc1 1,9T 1,1T 791G 58% /mnt/sheep/dsk03<br><br></div><div>As you can see, two disks (500G) are full.<br></div><div>During the night, the disk sdc1 (2T) of node id0 started giving I/O errors because of hardware problems.<br>
</div><div>Sheepdog automatically unplugged this device.<br><br></div><div>One hour later, sdc1 (2T) of node id1 also had hardware problems, started giving I/O errors and it got disconnected by sheepdog.<br></div><div><br>
</div><div>In such case, loosing two big disks on two different nodes at the same time, is equivalent of loosing 2 nodes at the same time.<br></div><div>My redundancy schema is -c 2 so, in any case, it wasn't possible to keep cluster consistency.<br>
</div><div>But let's continue with the story.<br><br></div><div>As soon as the first disk has failed, sheepdog on node id0 started the recovery and filled up sdb1 (500G).<br></div><div>The same has happened one hour later to node id1 (during recovery).<br>
<br></div><div>Notice that no nodes has left the cluster!<br><br></div><div>What did I do?<br><br></div><div>I stopped the cluster.<br></div><div>'dog cluster shutdown' wasn't enough, so I had to kill -9 all sheep daemons.<br>
<br></div><div>At this point I couldn't simply change the broken disks because I didn't have enough objects to restore the vdi(s).<br></div><div>I was able to copy the content of the broken sdc1 of ndoe id0 (by xfs_copy and xfs_repair).<br>
</div><div>Probably some objects got lost there.<br><br></div><div>I couldn't anyway run sheep on the same devices because sdb were full.<br></div><div>So I <br>- copied all the objects of /mnt/sheep/dsk01 and /mnt/sheep/dsk02 in /mnt/sheep/dsk03<br>
</div><div>- moved the meta data in /var/lib/sheepdog<br></div><div>- run sheepdog with only /var/lib/sheepdog,/mnt/sheep/dsk03<br></div><div>- as soon the cluster and the recovery started, I also run<br></div><div> dog cluster reweight<br>
<br></div><div>At the end of the recovery I run 'dog vdi check' on all vdi.<br></div><div>Only 1 small vdi of 10G was missing objects.<br></div><div>The other 'vdi check' printed only 'fixed replica'.<br>
</div><div><br></div><div>Later on, I've been also able to clone the second broken disk by ddrescue and read it's content after an intense xfs_repair.<br></div><div>I found 2 more vdi's objects but there were missing others.<br>
<br></div><div>Anyway, I tried to run a guest from the restored cluster but it wasn't starting.<br></div><div>I run the gust from a live cd (systemrescue) and run a fsck.ext4 on the guest's filesystem.<br></div><div>
It fixed ... to much ...<br></div><div>After it, all the content was in lost+found.<br></div><div>I tried with a second guest.<br></div><div>I was able to mount the filesystem before the check, but after the fsck, all the content got lost.<br>
I attach a screenshot in a second mail.<br></div><div>These guests were running during the creash.<br></div><div>The filesystem of vdi that were not "running" during the crash are fine!<br></div><div><br></div>
<div>
So the strategy of restoring objects from the disconnected/broken disks worked!<br></div><div>I suspect the filesystem issue happened before, when the second (2T) disk got broken or when the two disks of 500G started giving 'disk full'.<br>
</div><div>In such scenario, the best thing probably would have been the cluster to automatically stop.<br></div><div>I see that's more difficult to detect than a node leaving the cluster.<br></div><div><br></div><div>
I guess we may have 2 problems to reason about both related to multi disk:<br><br></div><div>1) when a disk get unplugged the weight of the node doesn't change.<br></div><div>That may lead to disk full on that node.<br>
</div><div>I don't know if in later sheepdog version the dog get disconnected from the cluster in such case, or it leads to unclear state (the node still in the cluster but unable to issue write requests)<br><br></div>
<div>2) The unlucky case of having more devices to break down in the same period on different hosts.<br></div><div>with redundancy -c 2 I may loose a single host or a single disk (on a single node).<br></div><div>with -c 2:2 I may loose 2 hosts or 2 disks on 2 different hosts.<br>
</div><div>If loose 3 hosts, the cluster halt itself, waiting for the missing node to be back up.<br></div><div>If 3 disks break down in the same time period (on different hosts), the cluster should also halt it self, or do something to keep the cluster consistency (waiting for a manual operation).<br>
<br></div><div>Thank you.<span class="HOEnZb"><font color="#888888"><br><br></font></span></div><span class="HOEnZb"><font color="#888888"><div>Valerio.<br></div></font></span></div></div></div>
</blockquote></div><br></div>