<br><br>
<div class="gmail_quote">On Mon, Mar 19, 2012 at 11:26 AM, huxinwei <span dir="ltr"><<a href="mailto:huxinwei@huawei.com">huxinwei@huawei.com</a>></span> wrote:<br>
<blockquote style="BORDER-LEFT:#ccc 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">
<div lang="ZH-CN" vlink="purple" link="blue">
<div>
<p class="MsoNormal"><span style="FONT-FAMILY:'Calibri','sans-serif';COLOR:#1f497d;FONT-SIZE:10.5pt" lang="EN-US">Hi,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="FONT-FAMILY:'Calibri','sans-serif';COLOR:#1f497d;FONT-SIZE:10.5pt" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="FONT-FAMILY:'Calibri','sans-serif';COLOR:#1f497d;FONT-SIZE:10.5pt" lang="EN-US"> Similar problems happened to me a while ago, even without cache.<u></u><u></u></span></p>
<p style="TEXT-INDENT:10.5pt" class="MsoNormal"><span style="FONT-FAMILY:'Calibri','sans-serif';COLOR:#1f497d;FONT-SIZE:10.5pt" lang="EN-US">However, I think the problem is “What’s the expected behavior of formatting a running cluster?”<u></u><u></u></span></p>
<p class="MsoNormal"><span style="FONT-FAMILY:'Calibri','sans-serif';COLOR:#1f497d;FONT-SIZE:10.5pt" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="FONT-FAMILY:'Calibri','sans-serif';COLOR:#1f497d;FONT-SIZE:10.5pt" lang="EN-US"> Has this been discussed before ? I’m wondering if you already have an answer for this.</span></p>
</div></div></blockquote>
<div> </div>
<div>Yes, the root cause is <font color="#1f497d" size="3" face="Calibri">formatting a running cluster.</font></div>
<div><font color="#1f497d" size="3" face="Calibri">From my test, the formatting lead to object and vdi bitmap clear. It seems right to me. I have not tested it carefully. There are some other errors in your test?</font></div>
<div><font color="#1f497d" size="3" face="Calibri"></font> </div>
<div><font color="#1f497d" size="3" face="Calibri">I am not sure of the behavior of this kind of formatting. I think sheepdog should support this.</font></div>
<div><font color="#1f497d" size="3" face="Calibri"></font> </div>
<div><font color="#1f497d" size="3" face="Calibri">Thanks,</font></div>
<div><font color="#1f497d" size="3" face="Calibri">Haiting</font></div>
<blockquote style="BORDER-LEFT:#ccc 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">
<div lang="ZH-CN" vlink="purple" link="blue">
<div>
<p class="MsoNormal"><span style="FONT-FAMILY:'Calibri','sans-serif';COLOR:#1f497d;FONT-SIZE:10.5pt" lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="FONT-FAMILY:'Calibri','sans-serif';COLOR:#1f497d;FONT-SIZE:10.5pt" lang="EN-US"><u></u> <u></u></span></p>
<div style="BORDER-BOTTOM:medium none;BORDER-LEFT:blue 1.5pt solid;PADDING-BOTTOM:0cm;PADDING-LEFT:4pt;PADDING-RIGHT:0cm;BORDER-TOP:medium none;BORDER-RIGHT:medium none;PADDING-TOP:0cm">
<div>
<div style="BORDER-BOTTOM:medium none;BORDER-LEFT:medium none;PADDING-BOTTOM:0cm;PADDING-LEFT:0cm;PADDING-RIGHT:0cm;BORDER-TOP:#b5c4df 1pt solid;BORDER-RIGHT:medium none;PADDING-TOP:3pt">
<p class="MsoNormal"><b><span style="FONT-FAMILY:'Tahoma','sans-serif';FONT-SIZE:10pt" lang="EN-US">From:</span></b><span style="FONT-FAMILY:'Tahoma','sans-serif';FONT-SIZE:10pt" lang="EN-US"> <a href="mailto:sheepdog-bounces@lists.wpkg.org" target="_blank">sheepdog-bounces@lists.wpkg.org</a> [mailto:<a href="mailto:sheepdog-bounces@lists.wpkg.org" target="_blank">sheepdog-bounces@lists.wpkg.org</a>] <b>On Behalf Of </b>HaiTing Yao<br>
<b>Sent:</b> Monday, March 19, 2012 10:44 AM<br><b>To:</b> Liu Yuan<br><b>Cc:</b> HaiTing Yao; <a href="mailto:sheepdog@lists.wpkg.org" target="_blank">sheepdog@lists.wpkg.org</a><br><b>Subject:</b> Re: [Sheepdog] [PATCH] sheep: modify cached_epoch<u></u><u></u></span></p>
</div></div>
<div>
<div class="h5">
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On Fri, Mar 16, 2012 at 6:35 PM, Liu Yuan <<a href="mailto:namei.unix@gmail.com" target="_blank">namei.unix@gmail.com</a>> wrote:<u></u><u></u></span></p>
<div>
<div>
<p style="MARGIN-BOTTOM:12pt" class="MsoNormal"><span lang="EN-US">On 03/16/2012 04:43 PM, <a href="mailto:yaohaiting.wujue@gmail.com" target="_blank">yaohaiting.wujue@gmail.com</a> wrote:<br><br>> From: HaiTing Yao <<a href="mailto:wujue.yht@taobao.com" target="_blank">wujue.yht@taobao.com</a>><br>
><br>> cached_epoch is a __thread variable. If it greater than 1, format the<br>> cluster again will lead to permanent I/O error.<br>><br>> Signed-off-by: HaiTing Yao <<a href="mailto:wujue.yht@taobao.com" target="_blank">wujue.yht@taobao.com</a>><br>
> ---<br>> sheep/sdnet.c | 6 +++++-<br>> 1 files changed, 5 insertions(+), 1 deletions(-)<br>><br>> diff --git a/sheep/sdnet.c b/sheep/sdnet.c<br>> index 5db9f29..d693858 100644<br>> --- a/sheep/sdnet.c<br>
> +++ b/sheep/sdnet.c<br>> @@ -832,7 +832,11 @@ int get_sheep_fd(uint8_t *addr, uint16_t port, int node_idx, uint32_t epoch)<br>> if (before(epoch, cached_epoch)) {<br>> eprintf("requested epoch is smaller than the previous one: %d < %d\n",<br>
> epoch, cached_epoch);<br>> - return -1;<br>> + /* cluster format again */<br>> + if (sys->epoch == 1)<br>> + cached_epoch = 0;<br>
> + else<br>> + return -1;<br>> }<br>> if (after(epoch, cached_epoch)) {<br>> for (i = 0; i < SD_MAX_NODES; i++) {<br><br><u></u><u></u></span></p>
</div></div>
<p class="MsoNormal"><span lang="EN-US">Any script that can reproduce this issue?<u></u><u></u></span></p>
<blockquote style="BORDER-BOTTOM:medium none;BORDER-LEFT:#cccccc 1pt solid;PADDING-BOTTOM:0cm;PADDING-LEFT:6pt;PADDING-RIGHT:0cm;MARGIN-LEFT:4.8pt;BORDER-TOP:medium none;MARGIN-RIGHT:0cm;BORDER-RIGHT:medium none;PADDING-TOP:0cm">
<p class="MsoNormal"><span lang="EN-US"><br>Thanks,<br>Yuan<u></u><u></u></span></p></blockquote>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p></div>
<div>
<p class="MsoNormal"><span lang="EN-US">Please try this script, thanks<u></u><u></u></span></p></div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p></div>
<div>
<p class="MsoNormal"><span lang="EN-US">The error log like this<u></u><u></u></span></p></div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p></div>
<div>
<p class="MsoNormal"><span lang="EN-US"> Mar 19 10:28:14 forward_write_obj_req(304) 70912800000000<br>Mar 19 10:28:14 get_sheep_fd(834) requested epoch is smaller than the previous one: 1 < 2<br>Mar 19 10:28:14 forward_write_obj_req(337) failed to connect to <a href="http://127.0.0.1:7002/" target="_blank">127.0.0.1:7002</a><br>
Mar 19 10:28:14 do_io_request(785) failed: 1, 70912800000000 , 1, 129<br>Mar 19 10:28:14 client_handler(557) closed connection 11<u></u><u></u></span></p></div>
<div>
<p class="MsoNormal"><span lang="EN-US">test-cached.sh<u></u><u></u></span></p></div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p></div>
<div>
<p class="MsoNormal"><span lang="EN-US">set -x<u></u><u></u></span></p></div>
<div>
<p class="MsoNormal"><span lang="EN-US">sudo killall sheep<br>sudo rm -rf ~/s1 ~/s2 ~/s3 ~/s4 <u></u><u></u></span></p></div>
<div>
<p class="MsoNormal"><span lang="EN-US">echo "test cached epoch" > ~/tmp-cached<br>sudo sheep -d ~/s1 -z 1 <br>sudo sheep -d ~/s2 -z 2 -p 7002 <br>sudo sheep -d ~/s3 -z 3 -p 7003 <br>sudo sheep -d ~/s4 -z 4 -p 7004 <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">sleep 60<u></u><u></u></span></p></div>
<div>
<p class="MsoNormal"><span lang="EN-US">collie cluster format<u></u><u></u></span></p></div>
<div>
<p class="MsoNormal"><span lang="EN-US">collie vdi create v1 64M<u></u><u></u></span></p></div>
<div>
<p class="MsoNormal"><span lang="EN-US">sleep 30<u></u><u></u></span></p></div>
<div>
<p class="MsoNormal"><span lang="EN-US">collie vdi write v1 0 1024 < ~/tmp-cached <u></u><u></u></span></p></div>
<div>
<p class="MsoNormal"><span lang="EN-US">ps -ef | grep "\-z 4" | awk '{print $2}' | xargs sudo kill<u></u><u></u></span></p></div>
<div>
<p class="MsoNormal"><span lang="EN-US">sleep 60<u></u><u></u></span></p></div>
<div>
<p class="MsoNormal"><span lang="EN-US">collie vdi write v1 0 1024 < ~/tmp-cached <u></u><u></u></span></p></div>
<div>
<p class="MsoNormal"><span lang="EN-US">sleep 6<u></u><u></u></span></p></div>
<div>
<p class="MsoNormal"><span lang="EN-US">collie cluster format<u></u><u></u></span></p></div>
<div>
<p class="MsoNormal"><span lang="EN-US">collie vdi create v1 64M<u></u><u></u></span></p></div>
<div>
<p class="MsoNormal"><span lang="EN-US">sleep 60<u></u><u></u></span></p></div>
<div>
<p class="MsoNormal"><span lang="EN-US">collie vdi write v1 0 1024 < ~/tmp-cached <u></u><u></u></span></p></div>
<div>
<p class="MsoNormal"><span lang="EN-US">Best Regards<u></u><u></u></span></p></div></div></div></div></div></div></div></blockquote></div><br>