<p dir="ltr">corosync can't support more than 10 nodes and many users include us find it not stable for a long time. use zookeeper for producttion</p>
<p dir="ltr">Yuan</p>
<div class="gmail_quote">2014-2-22 AM6:07于 "Aydelott, Ryan M." <<a href="mailto:ryade@mcs.anl.gov">ryade@mcs.anl.gov</a>>写道：<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
We are running a 20 node sheepdog cluster with ~50VM’s active during the test.<br>
<br>
13.10 Ubuntu<br>
2.3.3 corosync<br>
0.8.0 sheep<br>
<br>
Spawning Sheepd: sheep -n -c corosync:172.21.5.0 /meta,/var/lib/sheepdog/disc0,/var/lib/sheepdog/disc1,/var/lib/sheepdog/disc2,/var/lib/sheepdog/disc3,/var/lib/sheepdog/disc4,/var/lib/sheepdog/disc5,/var/lib/sheepdog/disc6,/var/lib/sheepdog/disc7,/var/lib/sheepdog/disc8,/var/lib/sheepdog/disc9,/var/lib/sheepdog/disc10,/var/lib/sheepdog/disc11,/var/lib/sheepdog/disc12,/var/lib/sheepdog/disc13<br>
<br>
The issue we are encountering is that when we power off a single node, a large group of sheepd’s (16 of 20 nodes) fail, causing the cluster to fail overall. The types of errors received across the cluster are:<br>
<br>
root@a1-p:/home/ryade# pdsh -w cs[141-160]-p 'grep EMERG /meta/sheep.log' | dshbak -c<br>
pdsh@a1-p: cs158-p: ssh exited with exit code 1<br>
pdsh@a1-p: cs159-p: ssh exited with exit code 1<br>
pdsh@a1-p: cs160-p: ssh exited with exit code 1<br>
pdsh@a1-p: cs157-p: ssh exited with exit code 1<br>
pdsh@a1-p: cs141-p: ssh exited with exit code 1<br>
----------------<br>
cs145-p<br>
----------------<br>
Feb 21 13:16:43 EMERG [main] cdrv_cpg_confchg(573) PANIC: Network partition is detected<br>
Feb 21 13:16:43 EMERG [main] crash_handler(267) sheep exits unexpectedly (Aborted).<br>
Feb 21 13:16:44 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:44 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:44 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:44 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:44 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:44 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:44 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:44 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:44 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:44 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:44 EMERG [main] sd_backtrace(817) :<br>
----------------<br>
cs146-p<br>
----------------<br>
Feb 21 13:17:03 EMERG [main] cdrv_cpg_confchg(573) PANIC: Network partition is detected<br>
Feb 21 13:17:03 EMERG [main] crash_handler(267) sheep exits unexpectedly (Aborted).<br>
Feb 21 13:17:03 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:03 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:03 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:03 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:03 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:03 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:03 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:03 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:03 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:03 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:03 EMERG [main] sd_backtrace(817) :<br>
----------------<br>
cs153-p<br>
----------------<br>
Feb 21 13:17:13 EMERG [oc_push 9673] do_push_object(866) PANIC: push failed but should never fail<br>
Feb 21 13:17:13 EMERG [oc_push 9673] crash_handler(267) sheep exits unexpectedly (Aborted).<br>
Feb 21 13:17:13 EMERG [oc_push 9673] sd_backtrace(817) :<br>
Feb 21 13:17:13 EMERG [oc_push 9673] sd_backtrace(817) :<br>
Feb 21 13:17:13 EMERG [oc_push 9673] sd_backtrace(817) :<br>
Feb 21 13:17:13 EMERG [oc_push 9673] sd_backtrace(817) :<br>
Feb 21 13:17:13 EMERG [oc_push 9673] sd_backtrace(817) :<br>
Feb 21 13:17:13 EMERG [oc_push 9673] sd_backtrace(817) :<br>
Feb 21 13:17:13 EMERG [oc_push 9673] sd_backtrace(817) :<br>
Feb 21 13:17:13 EMERG [oc_push 9673] sd_backtrace(817) :<br>
----------------<br>
cs156-p<br>
----------------<br>
Feb 21 13:17:57 EMERG [oc_push 8616] do_push_object(866) PANIC: push failed but should never fail<br>
Feb 21 13:17:57 EMERG [oc_push 8616] crash_handler(267) sheep exits unexpectedly (Aborted).<br>
Feb 21 13:17:57 EMERG [oc_push 8616] sd_backtrace(817) :<br>
Feb 21 13:17:57 EMERG [oc_push 8616] sd_backtrace(817) :<br>
Feb 21 13:17:57 EMERG [oc_push 8616] sd_backtrace(817) :<br>
Feb 21 13:17:57 EMERG [oc_push 8616] sd_backtrace(817) :<br>
Feb 21 13:17:57 EMERG [oc_push 8616] sd_backtrace(817) :<br>
Feb 21 13:17:57 EMERG [oc_push 8616] sd_backtrace(817) :<br>
Feb 21 13:17:57 EMERG [oc_push 8616] sd_backtrace(817) :<br>
Feb 21 13:17:57 EMERG [oc_push 8616] sd_backtrace(817) :<br>
----------------<br>
cs155-p<br>
----------------<br>
Feb 21 13:17:16 EMERG [oc_push 1433] do_push_object(866) PANIC: push failed but should never fail<br>
Feb 21 13:17:16 EMERG [oc_push 1433] crash_handler(267) sheep exits unexpectedly (Aborted).<br>
Feb 21 13:17:16 EMERG [oc_push 1433] sd_backtrace(817) :<br>
Feb 21 13:17:16 EMERG [oc_push 1433] sd_backtrace(817) :<br>
Feb 21 13:17:16 EMERG [oc_push 1433] sd_backtrace(817) :<br>
Feb 21 13:17:16 EMERG [oc_push 1433] sd_backtrace(817) :<br>
Feb 21 13:17:16 EMERG [oc_push 1433] sd_backtrace(817) :<br>
Feb 21 13:17:16 EMERG [oc_push 1433] sd_backtrace(817) :<br>
Feb 21 13:17:16 EMERG [oc_push 1433] sd_backtrace(817) :<br>
Feb 21 13:17:16 EMERG [oc_push 1433] sd_backtrace(817) :<br>
----------------<br>
cs149-p<br>
----------------<br>
Feb 21 13:17:19 EMERG [oc_push 9777] do_push_object(866) PANIC: push failed but should never fail<br>
Feb 21 13:17:19 EMERG [oc_push 9777] crash_handler(267) sheep exits unexpectedly (Aborted).<br>
Feb 21 13:17:20 EMERG [oc_push 9777] sd_backtrace(817) :<br>
Feb 21 13:17:20 EMERG [oc_push 9777] sd_backtrace(817) :<br>
Feb 21 13:17:20 EMERG [oc_push 9777] sd_backtrace(817) :<br>
Feb 21 13:17:20 EMERG [oc_push 9777] sd_backtrace(817) :<br>
Feb 21 13:17:20 EMERG [oc_push 9777] sd_backtrace(817) :<br>
Feb 21 13:17:20 EMERG [oc_push 9777] sd_backtrace(817) :<br>
Feb 21 13:17:20 EMERG [oc_push 9777] sd_backtrace(817) :<br>
Feb 21 13:17:20 EMERG [oc_push 9777] sd_backtrace(817) :<br>
----------------<br>
cs147-p<br>
----------------<br>
Feb 21 13:17:07 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:07 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:07 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:07 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:07 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:07 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:07 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:07 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:07 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:07 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:07 EMERG [main] sd_backtrace(817) :<br>
----------------<br>
cs148-p<br>
----------------<br>
Feb 21 13:17:07 EMERG [main] cdrv_cpg_confchg(573) PANIC: Network partition is detected<br>
Feb 21 13:17:07 EMERG [main] crash_handler(267) sheep exits unexpectedly (Aborted).<br>
Feb 21 13:17:07 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:07 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:07 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:07 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:07 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:07 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:07 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:07 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:07 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:07 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:17:07 EMERG [main] sd_backtrace(817) :<br>
----------------<br>
cs142-p<br>
----------------<br>
Feb 21 13:17:21 EMERG [oc_push 31470] do_push_object(866) PANIC: push failed but should never fail<br>
Feb 21 13:17:21 EMERG [oc_push 31470] crash_handler(267) sheep exits unexpectedly (Aborted).<br>
Feb 21 13:17:21 EMERG [oc_push 31730] do_push_object(866) PANIC: push failed but should never fail<br>
----------------<br>
cs150-p<br>
----------------<br>
Feb 21 13:17:22 EMERG [oc_push 6518] do_push_object(866) PANIC: push failed but should never fail<br>
Feb 21 13:17:22 EMERG [oc_push 6518] crash_handler(267) sheep exits unexpectedly (Aborted).<br>
Feb 21 13:17:22 EMERG [oc_push 6518] sd_backtrace(817) :<br>
Feb 21 13:17:22 EMERG [oc_push 6518] sd_backtrace(817) :<br>
Feb 21 13:17:22 EMERG [oc_push 6518] sd_backtrace(817) :<br>
Feb 21 13:17:22 EMERG [oc_push 6518] sd_backtrace(817) :<br>
Feb 21 13:17:22 EMERG [oc_push 6518] sd_backtrace(817) :<br>
Feb 21 13:17:22 EMERG [oc_push 6518] sd_backtrace(817) :<br>
Feb 21 13:17:22 EMERG [oc_push 6518] sd_backtrace(817) :<br>
Feb 21 13:17:22 EMERG [oc_push 6518] sd_backtrace(817) :<br>
----------------<br>
cs152-p<br>
----------------<br>
Feb 21 13:17:10 EMERG [oc_push 17997] do_push_object(866) PANIC: push failed but should never fail<br>
Feb 21 13:17:10 EMERG [oc_push 17997] crash_handler(267) sheep exits unexpectedly (Aborted).<br>
Feb 21 13:17:10 EMERG [oc_push 17997] sd_backtrace(817) :<br>
Feb 21 13:17:10 EMERG [oc_push 17997] sd_backtrace(817) :<br>
Feb 21 13:17:10 EMERG [oc_push 17997] sd_backtrace(817) :<br>
Feb 21 13:17:10 EMERG [oc_push 17997] sd_backtrace(817) :<br>
Feb 21 13:17:10 EMERG [oc_push 17997] sd_backtrace(817) :<br>
Feb 21 13:17:10 EMERG [oc_push 17997] sd_backtrace(817) :<br>
Feb 21 13:17:10 EMERG [oc_push 17997] sd_backtrace(817) :<br>
Feb 21 13:17:10 EMERG [oc_push 17997] sd_backtrace(817) :<br>
----------------<br>
cs144-p<br>
----------------<br>
Feb 21 13:16:58 EMERG [main] cdrv_cpg_confchg(573) PANIC: Network partition is detected<br>
Feb 21 13:16:58 EMERG [main] crash_handler(267) sheep exits unexpectedly (Aborted).<br>
Feb 21 13:16:59 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:59 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:59 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:59 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:59 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:59 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:59 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:59 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:59 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:59 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:59 EMERG [main] sd_backtrace(817) :<br>
----------------<br>
cs151-p<br>
----------------<br>
Feb 21 13:17:16 EMERG [oc_push 27774] do_push_object(866) PANIC: push failed but should never fail<br>
Feb 21 13:17:16 EMERG [oc_push 27774] crash_handler(267) sheep exits unexpectedly (Aborted).<br>
Feb 21 13:17:17 EMERG [oc_push 27774] sd_backtrace(817) :<br>
Feb 21 13:17:17 EMERG [oc_push 27774] sd_backtrace(817) :<br>
Feb 21 13:17:17 EMERG [oc_push 27774] sd_backtrace(817) :<br>
Feb 21 13:17:17 EMERG [oc_push 27774] sd_backtrace(817) :<br>
Feb 21 13:17:17 EMERG [oc_push 27774] sd_backtrace(817) :<br>
Feb 21 13:17:17 EMERG [oc_push 27774] sd_backtrace(817) :<br>
Feb 21 13:17:17 EMERG [oc_push 27774] sd_backtrace(817) :<br>
Feb 21 13:17:17 EMERG [oc_push 27774] sd_backtrace(817) :<br>
----------------<br>
cs154-p<br>
----------------<br>
Feb 21 13:17:18 EMERG [oc_push 22150] do_push_object(866) PANIC: push failed but should never fail<br>
Feb 21 13:17:18 EMERG [oc_push 22150] crash_handler(267) sheep exits unexpectedly (Aborted).<br>
Feb 21 13:17:18 EMERG [oc_push 22146] do_push_object(866) PANIC: push failed but should never fail<br>
----------------<br>
cs143-p<br>
----------------<br>
Feb 21 13:16:54 EMERG [main] cdrv_cpg_confchg(573) PANIC: Network partition is detected<br>
Feb 21 13:16:54 EMERG [main] crash_handler(267) sheep exits unexpectedly (Aborted).<br>
Feb 21 13:16:54 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:54 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:54 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:54 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:54 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:54 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:54 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:54 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:54 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:54 EMERG [main] sd_backtrace(817) :<br>
Feb 21 13:16:54 EMERG [main] sd_backtrace(817) :<br>
<br>
<br>
<br>--<br>
sheepdog-users mailing lists<br>
<a href="mailto:sheepdog-users@lists.wpkg.org">sheepdog-users@lists.wpkg.org</a><br>
<a href="http://lists.wpkg.org/mailman/listinfo/sheepdog-users" target="_blank">http://lists.wpkg.org/mailman/listinfo/sheepdog-users</a><br>
<br></blockquote></div>