<html><body><div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000"><div>Hi again,<br></div><div><br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Micha Kersloot" <micha@kovoks.nl><br><b>To: </b>"Valerio Pachera" <sirio81@gmail.com><br><b>Cc: </b>"Lista sheepdog user" <sheepdog-users@lists.wpkg.org><br><b>Sent: </b>Friday, October 3, 2014 2:37:00 PM<br><b>Subject: </b>Re: [sheepdog-users] Fwd: dog vdi check: ovject is incosistent<br><div><br></div><div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000"><div>Hi,<br></div><div><br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Micha Kersloot" <micha@kovoks.nl><br><b>To: </b>"Valerio Pachera" <sirio81@gmail.com><br><b>Cc: </b>"Lista sheepdog user" <sheepdog-users@lists.wpkg.org><br><b>Sent: </b>Friday, October 3, 2014 12:27:55 PM<br><b>Subject: </b>Re: [sheepdog-users] Fwd: dog vdi check: ovject is incosistent<br><div><br></div><div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000"><div>Hi,<br></div><div><br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Valerio Pachera" <sirio81@gmail.com><br><b>To: </b>"Lista sheepdog user" <sheepdog-users@lists.wpkg.org><br><b>Sent: </b>Friday, October 3, 2014 12:18:50 PM<br><b>Subject: </b>[sheepdog-users] Fwd: dog vdi check: ovject is incosistent<br><div><br></div><div dir="ltr"><br><div class="gmail_quote"><div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote"><span class="">2014-10-03 10:35 GMT+02:00 Micha Kersloot <span dir="ltr"><<a href="mailto:micha@kovoks.nl" target="_blank">micha@kovoks.nl</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:times new roman,new york,times,serif;font-size:12pt;color:#000000"> Recovery kicked in and the VM continued. After recovery was complete, I tried to start the sheep daemon and that failed because of some journal errors. I added the 'skip' to the journal option and recovery kicked in again. Looked all fine to me!<div><br></div></div></div></blockquote><div><br></div></span><div>What version of sheepdog are you running?<br></div><div>May you report the options you use to run sheep?</div></div></div></div></div></div></blockquote><div><p style="margin: 0px;">sheep -v<br>Sheepdog daemon version 0.8.3</p><p style="margin: 0px;"><br></p><p style="margin: 0px;">/usr/sbin/sheep -y 10.10.0.30 -c zookeeper:10.10.0.21:2181,10.10.0.22:2181,10.10.0.30:2181 -j size=512M -w size=50G --upgrade --pidfile /var/run/sheepdog.pid /var/lib/sheepdog /mnt/sheep/30</p></div><div><br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:times new roman,new york,times,serif;font-size:12pt;color:#000000"><div>But... To be sure I did a dog vdi check and that gives me:</div><div>object 2e754900000844 is inconsistent</div></div></div></blockquote><div><br></div></span><div>Wow, I've never seen that.<br></div><div>Did you shutdown the guest before running vdi check? <br></div></div></div></div></div></div></blockquote><div>First the guest was running, then I shutdown the guest and that made no difference. Here is info about the vdi itself.</div><div><br></div><div><p style="margin: 0px;"><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">dog vdi list</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;"> Name Id Size Used Shared Creation time VDI id Copies Tag</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;"> micha_test 0 10 GB 9.7 GB 0.0 MB 2014-10-02 17:36 2e7549 2:1</span></p></div></div></blockquote><div>I've done some more reading and working with the cluster and decided using the object cache is maybe not the best solution in my situation. So I've shutdown the cluster with dog cluster shutdown, replaced the "-w size=50G" with "-n" on all nodes and restarted the cluster without any errors. Now I have major filesystem errors on the kvm guest, but dog vdi check now runs without any problems. To me it looks like there where problems with the VDI which are now corrected by sheepdog, but these corrections somehow corrupted the filesystem. So two things to do for me now. 1 testing of the current setup is more stable 2. Setting up a new cluster to see if I can reproduce the problems.</div></div></blockquote><div>Looks like i'm able to reproduce the problem:</div><div><br></div><div><p style="margin: 0px;" data-mce-style="margin: 0px;">Oct 03 15:23:03 ALERT [main] get_vdi_copy_policy(117) copy policy for 2e7549 not found, set 0<br>Oct 03 15:23:03 ALERT [main] get_vdi_copy_policy(117) copy policy for 2e7549 not found, set 0<br>Oct 03 15:23:03 ALERT [main] get_vdi_copy_policy(117) copy policy for 2e7549 not found, set 0<br>Oct 03 15:23:03 INFO [main] send_join_request(787) IPv4 ip:10.10.0.22 port:7000 going to join the cluster<br>Oct 03 15:23:03 INFO [main] replay_journal_entry(159) /mnt/sheep/22/00f19e4e000000aa, size 6144, off 1781760, 0<br>Oct 03 15:23:03 ERROR [main] replay_journal_entry(166) open No such file or directory<br>Oct 03 15:23:03 EMERG [main] check_recover_journal_file(262) PANIC: recoverying from journal file (new) failed<br>Oct 03 15:23:03 EMERG [main] crash_handler(267) sheep exits unexpectedly (Aborted).<br>Oct 03 15:23:03 EMERG [main] sd_backtrace(833) : <br>Oct 03 15:23:03 EMERG [main] sd_backtrace(833) : <br>Oct 03 15:23:03 EMERG [main] sd_backtrace(833) : <br>Oct 03 15:23:03 EMERG [main] sd_backtrace(833) : <br>Oct 03 15:23:03 EMERG [main] sd_backtrace(833) : <br>Oct 03 15:23:03 EMERG [main] sd_backtrace(833) : <br>Oct 03 15:23:03 EMERG [main] sd_backtrace(833) : <br>Oct 03 15:23:04 EMERG [main] sd_backtrace(833) : <br>Oct 03 15:25:19 INFO [main] md_add_disk(338) /mnt/sheep/22, vdisk nr 467, total disk 1<br>Oct 03 15:25:19 ALERT [main] get_vdi_copy_policy(117) copy policy for 2e7549 not found, set 0<br>Oct 03 15:25:19 ALERT [main] get_vdi_copy_policy(117) copy policy for 2e7549 not found, set 0</p><p style="margin: 0px;" data-mce-style="margin: 0px;"><br></p><p style="margin: 0px;" data-mce-style="margin: 0px;">I killed the node and it joins the cluster, then find out there is something wrong with the journal and kills itself.</p><p style="margin: 0px;" data-mce-style="margin: 0px;"><br></p><p style="margin: 0px;" data-mce-style="margin: 0px;">another node gives:</p><p style="margin: 0px;" data-mce-style="margin: 0px;"><br></p><p style="margin: 0px;" data-mce-style="margin: 0px;">Oct 03 15:23:04 ERROR [block] do_read(236) failed to read from socket: -1, Connection reset by peer<br>Oct 03 15:23:04 ERROR [block] exec_req(347) failed to read a response<br>Oct 03 15:23:04 ALERT [block] do_get_vdis(499) failed to get vdi bitmap from IPv4 ip:10.10.0.22 port:7000<br>Oct 03 15:23:04 ERROR [rw] connect_to(193) failed to connect to 10.10.0.22:7000: Connection refused<br>Oct 03 15:23:04 ERROR [rw] connect_to(193) failed to connect to 10.10.0.22:7000: Connection refused<br>Oct 03 15:23:04 ALERT [rw] fetch_object_list(931) cannot get object list from 10.10.0.22:7000<br>Oct 03 15:23:04 ALERT [rw] fetch_object_list(933) some objects may be not recovered at epoch 12<br>Oct 03 15:23:04 ERROR [rw] sheep_exec_req(1131) failed No object found, remote address: 10.10.0.21:7000, op name: READ_PEER<br>Oct 03 15:23:04 ERROR [rw] sheep_exec_req(1131) failed No object found, remote address: 10.10.0.30:7000, op name: READ_PEER<br>Oct 03 15:23:04 ERROR [rw] read_erasure_object(206) can not read 2e75490000009e idx 2<br>Oct 03 15:23:04 ERROR [rw] sheep_exec_req(1131) failed No object found, remote address: 10.10.0.21:7000, op name: READ_PEER<br>Oct 03 15:23:04 ERROR [rw] sheep_exec_req(1131) failed No object found, remote address: 10.10.0.21:7000, op name: READ_PEER<br>Oct 03 15:23:04 ERROR [rw] sheep_exec_req(1131) failed No object found, remote address: 10.10.0.30:7000, op name: READ_PEER<br>Oct 03 15:23:04 ERROR [rw] read_erasure_object(206) can not read 2e754900000758 idx 2</p><p style="margin: 0px;" data-mce-style="margin: 0px;"><br></p><p style="margin: 0px;" data-mce-style="margin: 0px;">and maybe more interesting:</p><p style="margin: 0px;" data-mce-style="margin: 0px;"><br></p><p style="margin: 0px;" data-mce-style="margin: 0px;">Oct 03 15:23:07 ERROR [rw] sheep_exec_req(1131) failed No object found, remote address: 10.10.0.30:7000, op name: READ_PEER<br>Oct 03 15:23:07 ERROR [rw] read_erasure_object(206) can not read 2e7549000008df idx 2<br>Oct 03 15:23:07 ERROR [rw] sheep_exec_req(1131) failed No object found, remote address: 10.10.0.21:7000, op name: READ_PEER<br>Oct 03 15:23:07 INFO [main] recover_object_main(863) object recovery progress 1% <br>Oct 03 15:23:07 ERROR [rw] sheep_exec_req(1131) failed No object found, remote address: 10.10.0.21:7000, op name: READ_PEER<br>Oct 03 15:23:07 ERROR [rw] sheep_exec_req(1131) failed No object found, remote address: 10.10.0.30:7000, op name: READ_PEER<br>Oct 03 15:23:07 ERROR [rw] read_erasure_object(206) can not read 2e7549000006f7 idx 2</p><p style="margin: 0px;" data-mce-style="margin: 0px;"><br></p><p style="margin: 0px;" data-mce-style="margin: 0px;">It looks like the VDI gets 'recoverd' to an instable state, hence dog vdi check gives:</p><p style="margin: 0px;" data-mce-style="margin: 0px;"><br></p><p style="margin: 0px;" data-mce-style="margin: 0px;">97.0 % [============================================================================================> ] 15 GB / 15 GB object f19e4e00000476 is inconsistent<br> 97.0 % [============================================================================================> ] 15 GB / 15 GB object f19e4e00000477 is inconsistent<br> 97.1 % [============================================================================================> ] 15 GB / 15 GB object f19e4e00000478 is inconsistent<br> 97.1 % [============================================================================================> ] 15 GB / 15 GB object f19e4e00000479 is inconsistent<br> 97.1 % [============================================================================================> ] 15 GB / 15 GB object f19e4e0000047a is inconsistent</p><p style="margin: 0px;" data-mce-style="margin: 0px;"><br></p><p style="margin: 0px;" data-mce-style="margin: 0px;">I will try again with the journal skip option enabled the first time after I pooled the plug on one of the nodes.</p><p style="margin: 0px;" data-mce-style="margin: 0px;"><br></p><p style="margin: 0px;" data-mce-style="margin: 0px;">Greets,</p><p style="margin: 0px;" data-mce-style="margin: 0px;"><br></p><p style="margin: 0px;" data-mce-style="margin: 0px;">Micha</p></div></div></body></html>