<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div><div>Interesting note on the data management, I have not dug much into the internals with Sheepdog yet - but the only possible explanation would be if any metadata on file object positioning was being retrieved through the backend.</div><div><br></div><div>I agree with your statements as the IO on the zookeeper nodes is very small/nowhere near the data we are pushing throughout the sheepd members.</div><div><br></div><div>Sheepdog is being started as follows for each test iteration:</div><div><br></div><div>Corosync: sheep -n -c corosync:172.21.5.0 /meta,/var/lib/sheepdog/disc0,/var/lib/sheepdog/disc1,/var/lib/sheepdog/disc2,/var/lib/sheepdog/disc3,/var/lib/sheepdog/disc4,/var/lib/sheepdog/disc5,/var/lib/sheepdog/disc6,/var/lib/sheepdog/disc7,/var/lib/sheepdog/disc8,/var/lib/sheepdog/disc9,/var/lib/sheepdog/disc10,/var/lib/sheepdog/disc11,/var/lib/sheepdog/disc12,/var/lib/sheepdog/disc13</div><div><br></div><div>Zookeeper: sheep -n -c zookeeper:172.21.5.161:2181,172.21.5.162:2181,172.21.5.163:2181,172.21.5.164:2181,172.21.5.165:2181 /meta /var/lib/sheepdog/disc0,/var/lib/sheepdog/disc1,/var/lib/sheepdog/disc2,/var/lib/sheepdog/disc3,/var/lib/sheepdog/disc4,/var/lib/sheepdog/disc5,/var/lib/sheepdog/disc6,/var/lib/sheepdog/disc7,/var/lib/sheepdog/disc8,/var/lib/sheepdog/disc9,/var/lib/sheepdog/disc10,/var/lib/sheepdog/disc11,/var/lib/sheepdog/disc12,/var/lib/sheepdog/disc13</div></div><div><span style="font-family: Arial, Helvetica, FreeSans, sans-serif; font-size: 13px; line-height: 17.33333396911621px; background-color: rgb(255, 255, 255);"><br></span></div><div><font face="Arial, Helvetica, FreeSans, sans-serif" size="2"><span style="line-height: 17px;">The driver we wrote/use is: </span></font><a href="https://github.com/devoid/nova/tree/sheepdog-nova-support-havana">https://github.com/devoid/nova/tree/sheepdog-nova-support-havana</a></div><div><br></div><div>Which builds out libvirt.xml as follows: </div><div><br></div><div><div> <disk type="network" device="disk"></div><div> <driver name="qemu" cache="writethrough"/></div><div> <source protocol="sheepdog" name="//172.21.5.141:7000/instance_f9dc065b-d05d-47cb-a3e6-b02049f049df_disk"/></div><div> <target bus="virtio" dev="vda"/></div><div> </disk></div></div><div><br></div><div><br></div><div>-ryan<br><div><div><br></div><div>On Mar 14, 2014, at 12:36 AM, Liu Yuan <<a href="mailto:namei.unix@gmail.com">namei.unix@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">On Thu, Mar 13, 2014 at 11:05:03PM +0000, Aydelott, Ryan M. wrote:<br><blockquote type="cite">I’m hoping someone can shed some light on some pretty startling performance deltas with different clustering backend in Sheepdog.<span class="Apple-converted-space"> </span><br><br>We are running a 20 node, 320 spindle (280 for data storage, 40 ssd for meta/os) sheepdog cluster backed with a 40GB/s IB network.<br><br></blockquote><br>Cool, I've never seen any one trying to run sheepdog over IB.<br><br><blockquote type="cite">In a VM running under Openstack Havana using the root/ephemeral volume driver (<a href="https://github.com/devoid/nova/tree/sheepdog-nova-support-havana">https://github.com/devoid/nova/tree/sheepdog-nova-support-havana</a>) we are seeing the following performance characteristics under 0.8.0 sheepdog, test results were obtained using: iozone -i0 -i1 -t 1 -r 128#k -s 10G (-t 4 for parallel read numbers)<br><br>Running with corosync 2.3.3\RDMA replica 3, single VM: 600MB/s on write, with about 132MB/s on read with a maximum read of 520MB/s running four read threads in parallel.<span class="Apple-converted-space"> </span><br><br>Running with zookeeper 3.4.5+dfsg-1 replica 3, single VM: 40MB/s on write, with about 40MB/s on read and a maximum read of 90MB/s running four read threads in parallel.<br><br>In both cases, multiple write threads had little/no impact on overall write performance.<br><br>Note that sheepd is spawned identically in each case, except for the references to each respective cluster backend. Have others experienced similar performance numbers with zookeeper/corosync? Are there any standard zookeeper configuration changes you make out of the box? We are taking the defaults for zookeeper package installation at this time.<br><br><br></blockquote><br>Cluster driver (corosync or zookeeper) only manages membership (mainly doing<br>heartbeat and signaling nodes in/out) and all the IOs (guest to sheep and<br>inter-sheep, dog to sheep) are all p2p and not involing cluster driver at all.<br><br>I don't why you experience such a huge difference because in theory there should<br>be no difference.<br><br>What is startup command for sheep and QEMU in both cases?<br><br>Thanks<br>Yuan</div></blockquote></div><br></div></body></html>