[sheepdog-users] Corosync Vs Zookeeper Backends

[Aydelott, Ryan M.]

Interesting note on the data management, I have not dug much into the internals with Sheepdog yet - but the only possible explanation would be if any metadata on file object positioning was being retrieved through the backend.

I agree with your statements as the IO on the zookeeper nodes is very small/nowhere near the data we are pushing throughout the sheepd members.

Sheepdog is being started as follows for each test iteration:

Corosync: sheep -n -c corosync:172.21.5.0 /meta,/var/lib/sheepdog/disc0,/var/lib/sheepdog/disc1,/var/lib/sheepdog/disc2,/var/lib/sheepdog/disc3,/var/lib/sheepdog/disc4,/var/lib/sheepdog/disc5,/var/lib/sheepdog/disc6,/var/lib/sheepdog/disc7,/var/lib/sheepdog/disc8,/var/lib/sheepdog/disc9,/var/lib/sheepdog/disc10,/var/lib/sheepdog/disc11,/var/lib/sheepdog/disc12,/var/lib/sheepdog/disc13

Zookeeper: sheep -n -c zookeeper:172.21.5.161:2181,172.21.5.162:2181,172.21.5.163:2181,172.21.5.164:2181,172.21.5.165:2181 /meta /var/lib/sheepdog/disc0,/var/lib/sheepdog/disc1,/var/lib/sheepdog/disc2,/var/lib/sheepdog/disc3,/var/lib/sheepdog/disc4,/var/lib/sheepdog/disc5,/var/lib/sheepdog/disc6,/var/lib/sheepdog/disc7,/var/lib/sheepdog/disc8,/var/lib/sheepdog/disc9,/var/lib/sheepdog/disc10,/var/lib/sheepdog/disc11,/var/lib/sheepdog/disc12,/var/lib/sheepdog/disc13

The driver we wrote/use is: https://github.com/devoid/nova/tree/sheepdog-nova-support-havana

Which builds out libvirt.xml as follows: 

    <disk type="network" device="disk">
      <driver name="qemu" cache="writethrough"/>
      <source protocol="sheepdog" name="//172.21.5.141:7000/instance_f9dc065b-d05d-47cb-a3e6-b02049f049df_disk"/>
      <target bus="virtio" dev="vda"/>
    </disk>


-ryan

On Mar 14, 2014, at 12:36 AM, Liu Yuan <namei.unix at gmail.com> wrote:

> On Thu, Mar 13, 2014 at 11:05:03PM +0000, Aydelott, Ryan M. wrote:
>> I’m hoping someone can shed some light on some pretty startling performance deltas with different clustering backend in Sheepdog. 
>> 
>> We are running a 20 node, 320 spindle (280 for data storage, 40 ssd for meta/os) sheepdog cluster backed with a 40GB/s IB network.
>> 
> 
> Cool, I've never seen any one trying to run sheepdog over IB.
> 
>> In a VM running under Openstack Havana using the root/ephemeral volume driver (https://github.com/devoid/nova/tree/sheepdog-nova-support-havana) we are seeing the following performance characteristics under 0.8.0 sheepdog, test results were obtained using:  iozone -i0 -i1 -t 1 -r 128#k -s 10G (-t 4 for parallel read numbers)
>> 
>> Running with corosync 2.3.3\RDMA replica 3, single VM: 600MB/s on write, with about 132MB/s on read with a maximum read of 520MB/s running four read threads in parallel. 
>> 
>> Running  with zookeeper 3.4.5+dfsg-1 replica 3, single VM: 40MB/s on write, with about 40MB/s on read and a maximum read of 90MB/s running four read threads in parallel.
>> 
>> In both cases, multiple write threads had little/no impact on overall write performance.
>> 
>> Note that sheepd is spawned identically in each case, except for the references to each respective cluster backend. Have others experienced similar performance numbers with zookeeper/corosync? Are there any standard zookeeper configuration changes you make out of the box? We are taking the defaults for zookeeper package installation at this time.
>> 
>> 
> 
> Cluster driver (corosync or zookeeper) only manages membership (mainly doing
> heartbeat and signaling nodes in/out) and all the IOs (guest to sheep and
> inter-sheep, dog to sheep) are all p2p and not involing cluster driver at all.
> 
> I don't why you experience such a huge difference because in theory there should
> be no difference.
> 
> What is startup command for sheep and QEMU in both cases?
> 
> Thanks
> Yuan

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wpkg.org/pipermail/sheepdog-users/attachments/20140314/fdd3fb38/attachment-0005.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.wpkg.org/pipermail/sheepdog-users/attachments/20140314/fdd3fb38/attachment-0003.sig>