<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=gb2312">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]--><o:SmartTagType
namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="City"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="place"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="chsdate"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"\@宋体";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";
color:black;}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:blue;
text-decoration:underline;}
pre
{margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:Arial;
color:navy;}
@page Section1
{size:595.3pt 841.9pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor=white lang=ZH-CN link=blue vlink=blue>
<div class=Section1>
<p class=MsoNormal><font size=1 color=navy face=Arial><span lang=EN-US
style='font-size:9.0pt;font-family:Arial;color:navy'>Please make sure the recycle
VDI is disabled for v <st1:chsdate IsROCDate="False" IsLunarDate="False"
Day="30" Month="12" Year="1899" w:st="on">0.9.1</st1:chsdate><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 color=navy face=Arial><span lang=EN-US
style='font-size:9.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=1 color=navy face=Arial><span lang=EN-US
style='font-size:9.0pt;font-family:Arial;color:navy'>Daily snapshot will remove
the old snapshot and create a new one.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 color=navy face=Arial><span lang=EN-US
style='font-size:9.0pt;font-family:Arial;color:navy'>If recycle VDI is enabled,
when the snapshot is removed, the assigned VDI will be deleted. And so the data
is lost. <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 color=navy face=Arial><span lang=EN-US
style='font-size:9.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=1 color=navy face=Arial><span lang=EN-US
style='font-size:9.0pt;font-family:Arial;color:navy'>This bug is fixed in v <st1:chsdate
IsROCDate="False" IsLunarDate="False" Day="30" Month="12" Year="1899" w:st="on">0.7.1</st1:chsdate>.
but I am not sure in v 0.9.1.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 color=navy face=Arial><span lang=EN-US
style='font-size:9.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=1 color=navy face=Arial><span lang=EN-US
style='font-size:9.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<div>
<div class=MsoNormal align=center style='text-align:center'><font size=3
color=black face="Times New Roman"><span lang=EN-US style='font-size:12.0pt;
color:windowtext'>
<hr size=2 width="100%" align=center tabindex=-1>
</span></font></div>
<p class=MsoNormal><b><font size=2 color=black face=宋体><span style='font-size:
10.0pt;font-family:宋体;color:windowtext;font-weight:bold'>发件人<span lang=EN-US>:</span></span></font></b><font
size=2 color=black face=宋体><span lang=EN-US style='font-size:10.0pt;font-family:
宋体;color:windowtext'> sheepdog-users [mailto:sheepdog-users-bounces@lists.wpkg.org]
</span></font><b><font size=2 color=black face=宋体><span style='font-size:10.0pt;
font-family:宋体;color:windowtext;font-weight:bold'>代表 </span></font></b><font
size=2 color=black face=宋体><span lang=EN-US style='font-size:10.0pt;font-family:
宋体;color:windowtext'>Thornton Prime<br>
</span></font><b><font size=2 color=black face=宋体><span style='font-size:10.0pt;
font-family:宋体;color:windowtext;font-weight:bold'>发送时间<span lang=EN-US>:</span></span></font></b><font
size=2 color=black face=宋体><span lang=EN-US style='font-size:10.0pt;font-family:
宋体;color:windowtext'> <st1:chsdate IsROCDate="False" IsLunarDate="False"
Day="27" Month="1" Year="2015" w:st="on">2015<span lang=EN-US><span lang=EN-US>年1</span></span><span
lang=EN-US><span lang=EN-US>月27</span></span><span lang=EN-US><span
lang=EN-US>日</span></span></st1:chsdate> 23:50<br>
</span></font><b><font size=2 color=black face=宋体><span style='font-size:10.0pt;
font-family:宋体;color:windowtext;font-weight:bold'>收件人<span lang=EN-US>:</span></span></font></b><font
size=2 color=black face=宋体><span lang=EN-US style='font-size:10.0pt;font-family:
宋体;color:windowtext'> Hitoshi Mitake<br>
</span></font><b><font size=2 color=black face=宋体><span style='font-size:10.0pt;
font-family:宋体;color:windowtext;font-weight:bold'>抄送<span lang=EN-US>:</span></span></font></b><font
size=2 color=black face=宋体><span lang=EN-US style='font-size:10.0pt;font-family:
宋体;color:windowtext'> Lista sheepdog user<br>
</span></font><b><font size=2 color=black face=宋体><span style='font-size:10.0pt;
font-family:宋体;color:windowtext;font-weight:bold'>主题<span lang=EN-US>:</span></span></font></b><font
size=2 color=black face=宋体><span lang=EN-US style='font-size:10.0pt;font-family:
宋体;color:windowtext'> Re: [sheepdog-users] Help? Creeping Errors "no inode
has ..." with <st1:chsdate IsROCDate="False" IsLunarDate="False" Day="30"
Month="12" Year="1899" w:st="on">0.9.1</st1:chsdate></span></font><font
color=black><span lang=EN-US style='color:windowtext'><o:p></o:p></span></font></p>
</div>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
lang=EN-US style='font-size:12.0pt'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
lang=EN-US style='font-size:12.0pt'>Thanks. I have been using cache -- so if
that is unstable that would explain a lot. I'm disabling cache to see how much
that helps.<br>
<br>
Attached is a dog cluster info. I have a few MB of logs ... I'll see where I
can post them to get the<br>
<br>
I am seeing a strong correlation between snapshots and the corrupted VDIs. All
the VDIs that have missing inodes are part of a daily snapshot schedule. All
the VDIs that are not part of the snapshot schedule are fine. All the nodes
have object cache enabled.<br>
<br>
Thanks ... I'll see if I can collect more data and reproduce the problem more
consistently.<br>
<br>
~ <st1:City w:st="on"><st1:place w:st="on">thornton</st1:place></st1:City>
prime<br>
<br>
<br>
<o:p></o:p></span></font></p>
<div style='margin-left:18.75pt;margin-top:22.5pt;margin-right:18.75pt;
margin-bottom:7.5pt'>
<div style='border:none;border-top:solid #EDEEF0 1.0pt;padding:4.0pt 0cm 0cm 0cm;
display:table'>
<div style='display:table-cell'>
<p class=MsoNormal style='vertical-align:middle'><font size=3 color=black
face="Times New Roman"><span lang=EN-US style='font-size:12.0pt'><img width=25
height=25 id="_x0000_i1025" src="cid:image001.jpg@01D03AD2.C89F7160"
photoaddress="mitake.hitoshi@lab.ntt.co.jp" photoname="Hitoshi Mitake"
name=compose-unknown-contact.jpg><o:p></o:p></span></font></p>
</div>
<div style='display:table-cell'>
<p class=MsoNormal style='vertical-align:middle'><font size=3 color=black
face="Times New Roman"><span lang=EN-US style='font-size:12.0pt'><a
href="mailto:mitake.hitoshi@lab.ntt.co.jp"><b><span style='font-weight:bold'>Hitoshi
Mitake</span></b></a><o:p></o:p></span></font></p>
</div>
<div style='display:table-cell'>
<p class=MsoNormal style='vertical-align:middle'><font size=3 color="#9fa2a5"
face="Times New Roman"><span lang=EN-US style='font-size:12.0pt;color:#9FA2A5'>January
26, 2015 at 8:17 PM</span></font><span lang=EN-US><o:p></o:p></span></p>
</div>
</div>
</div>
<div style='margin-left:18.0pt;margin-right:18.0pt' __pbrmquotes=true><pre><font
size=2 color="#888888" face="Courier New"><span lang=EN-US style='font-size:
10.0pt;color:#888888'>At Mon, 26 Jan 2015 07:11:29 -0800,<o:p></o:p></span></font></pre><pre><font
size=2 color="#888888" face="Courier New"><span lang=EN-US style='font-size:
10.0pt;color:#888888'>Thornton Prime wrote:<o:p></o:p></span></font></pre>
<blockquote style='margin-top:5.0pt;margin-bottom:5.0pt' type=cite><pre wrap=""><font
size=2 color="#888888" face="Courier New"><span lang=EN-US style='font-size:
10.0pt;color:#888888'>I've been getting increasing errors in my logs that "failed No object<o:p></o:p></span></font></pre><pre><font
size=2 color="#888888" face="Courier New"><span lang=EN-US style='font-size:
10.0pt;color:#888888'>found, remote address: XXXXXXX:7000, op name: READ_PEER" and then<o:p></o:p></span></font></pre><pre><font
size=2 color="#888888" face="Courier New"><span lang=EN-US style='font-size:
10.0pt;color:#888888'>corresponding errors that "no inode has ...." when I do a cluster check.<o:p></o:p></span></font></pre></blockquote>
<pre wrap=""><font size=2 color="#888888" face="Courier New"><span lang=EN-US
style='font-size:10.0pt;color:#888888'><o:p> </o:p></span></font></pre><pre><font
size=2 color="#888888" face="Courier New"><span lang=EN-US style='font-size:
10.0pt;color:#888888'>Could you provide detailed logs and an output of "dog cluster info"?<o:p></o:p></span></font></pre><pre><font
size=2 color="#888888" face="Courier New"><span lang=EN-US style='font-size:
10.0pt;color:#888888'><o:p> </o:p></span></font></pre>
<blockquote style='margin-top:5.0pt;margin-bottom:5.0pt' type=cite><pre wrap=""><font
size=2 color="#888888" face="Courier New"><span lang=EN-US style='font-size:
10.0pt;color:#888888'>At the beginning of last week I had no errors, and over the course of a<o:p></o:p></span></font></pre><pre><font
size=2 color="#888888" face="Courier New"><span lang=EN-US style='font-size:
10.0pt;color:#888888'>week it grew to be one VDI missing some hundred inodes, and now it is<o:p></o:p></span></font></pre><pre><font
size=2 color="#888888" face="Courier New"><span lang=EN-US style='font-size:
10.0pt;color:#888888'>multiple VDIs each missing hundreds of objects.<o:p></o:p></span></font></pre><pre><font
size=2 color="#888888" face="Courier New"><span lang=EN-US style='font-size:
10.0pt;color:#888888'><o:p> </o:p></span></font></pre><pre><font size=2
color="#888888" face="Courier New"><span lang=EN-US style='font-size:10.0pt;
color:#888888'>I haven't seen any issues with the underlying hardware, disks, or<o:p></o:p></span></font></pre><pre><font
size=2 color="#888888" face="Courier New"><span lang=EN-US style='font-size:
10.0pt;color:#888888'>zookeeper on the nodes in the course of the same time.<o:p></o:p></span></font></pre><pre><font
size=2 color="#888888" face="Courier New"><span lang=EN-US style='font-size:
10.0pt;color:#888888'><o:p> </o:p></span></font></pre><pre><font size=2
color="#888888" face="Courier New"><span lang=EN-US style='font-size:10.0pt;
color:#888888'>What is causing this data loss? How can I debug it? How can I stem it?<o:p></o:p></span></font></pre><pre><font
size=2 color="#888888" face="Courier New"><span lang=EN-US style='font-size:
10.0pt;color:#888888'>Any chances I can repair the missing inodes?<o:p></o:p></span></font></pre><pre><font
size=2 color="#888888" face="Courier New"><span lang=EN-US style='font-size:
10.0pt;color:#888888'><o:p> </o:p></span></font></pre><pre><font size=2
color="#888888" face="Courier New"><span lang=EN-US style='font-size:10.0pt;
color:#888888'>I have 5 sheepdog storage nodes, also running Zookeeper. I have another<o:p></o:p></span></font></pre><pre><font
size=2 color="#888888" face="Courier New"><span lang=EN-US style='font-size:
10.0pt;color:#888888'>8 "gateway only" nodes that are part of the node pool, but only running<o:p></o:p></span></font></pre><pre><font
size=2 color="#888888" face="Courier New"><span lang=EN-US style='font-size:
10.0pt;color:#888888'>a gateway and cache.<o:p></o:p></span></font></pre></blockquote>
<pre wrap=""><font size=2 color="#888888" face="Courier New"><span lang=EN-US
style='font-size:10.0pt;color:#888888'><o:p> </o:p></span></font></pre><pre><font
size=2 color="#888888" face="Courier New"><span lang=EN-US style='font-size:
10.0pt;color:#888888'>Object cache (a functionality which can be activated with -w option of<o:p></o:p></span></font></pre><pre><font
size=2 color="#888888" face="Courier New"><span lang=EN-US style='font-size:
10.0pt;color:#888888'>sheep) is quite unstable. Please do not use it for serious purpose.<o:p></o:p></span></font></pre><pre><font
size=2 color="#888888" face="Courier New"><span lang=EN-US style='font-size:
10.0pt;color:#888888'><o:p> </o:p></span></font></pre><pre><font size=2
color="#888888" face="Courier New"><span lang=EN-US style='font-size:10.0pt;
color:#888888'>Thanks,<o:p></o:p></span></font></pre><pre><font size=2
color="#888888" face="Courier New"><span lang=EN-US style='font-size:10.0pt;
color:#888888'>Hitoshi<o:p></o:p></span></font></pre></div>
<div style='margin-left:18.75pt;margin-top:22.5pt;margin-right:18.75pt;
margin-bottom:7.5pt'>
<div style='border:none;border-top:solid #EDEEF0 1.0pt;padding:4.0pt 0cm 0cm 0cm;
display:table'>
<div style='display:table-cell'>
<p class=MsoNormal style='vertical-align:middle'><font size=3 color=black
face="Times New Roman"><span lang=EN-US style='font-size:12.0pt'><img border=0
width=25 height=25 id="_x0000_i1026" src="cid:image002.jpg@01D03AD2.C89F7160"
photoaddress="thornton.prime@gmail.com" photoname="Thornton Prime"
name=postbox-contact.jpg><o:p></o:p></span></font></p>
</div>
<div style='display:table-cell'>
<p class=MsoNormal style='vertical-align:middle'><font size=3 color=black
face="Times New Roman"><span lang=EN-US style='font-size:12.0pt'><a
href="mailto:thornton.prime@gmail.com"><b><span style='font-weight:bold'>Thornton
Prime</span></b></a><o:p></o:p></span></font></p>
</div>
<div style='display:table-cell'>
<p class=MsoNormal style='vertical-align:middle'><font size=3 color="#9fa2a5"
face="Times New Roman"><span lang=EN-US style='font-size:12.0pt;color:#9FA2A5'>January
26, 2015 at 7:11 AM</span></font><span lang=EN-US><o:p></o:p></span></p>
</div>
</div>
</div>
<div style='margin-left:18.0pt;margin-right:18.0pt' __pbrmquotes=true>
<div>
<p class=MsoNormal><font size=3 color="#888888" face="Times New Roman"><span
lang=EN-US style='font-size:12.0pt;color:#888888'>I've been getting increasing
errors in my logs that "failed No object<br>
found, remote address: XXXXXXX:7000, op name: READ_PEER" and then<br>
corresponding errors that "no inode has ...." when I do a cluster
check.<br>
<br>
At the beginning of last week I had no errors, and over the course of a<br>
week it grew to be one VDI missing some hundred inodes, and now it is<br>
multiple VDIs each missing hundreds of objects.<br>
<br>
I haven't seen any issues with the underlying hardware, disks, or<br>
zookeeper on the nodes in the course of the same time.<br>
<br>
What is causing this data loss? How can I debug it? How can I stem it?<br>
Any chances I can repair the missing inodes?<br>
<br>
I have 5 sheepdog storage nodes, also running Zookeeper. I have another<br>
8 "gateway only" nodes that are part of the node pool, but only
running<br>
a gateway and cache.<br>
<br>
I have about dozen VDI images, and they've been fairly static for the<br>
last week while I've been testing -- not a lot of write activity.<br>
<br>
~ <st1:City w:st="on"><st1:place w:st="on">thornton</st1:place></st1:City><o:p></o:p></span></font></p>
</div>
</div>
</div>
</body>
</html>