[sheepdog-users] Corosync Vs Zookeeper Backends

[Liu Yuan]

On Thu, Mar 13, 2014 at 11:05:03PM +0000, Aydelott, Ryan M. wrote:
> I’m hoping someone can shed some light on some pretty startling performance deltas with different clustering backend in Sheepdog. 
> 
> We are running a 20 node, 320 spindle (280 for data storage, 40 ssd for meta/os) sheepdog cluster backed with a 40GB/s IB network.
> 

Cool, I've never seen any one trying to run sheepdog over IB.

> In a VM running under Openstack Havana using the root/ephemeral volume driver (https://github.com/devoid/nova/tree/sheepdog-nova-support-havana) we are seeing the following performance characteristics under 0.8.0 sheepdog, test results were obtained using:  iozone -i0 -i1 -t 1 -r 128#k -s 10G (-t 4 for parallel read numbers)
> 
> Running with corosync 2.3.3\RDMA replica 3, single VM: 600MB/s on write, with about 132MB/s on read with a maximum read of 520MB/s running four read threads in parallel. 
> 
> Running  with zookeeper 3.4.5+dfsg-1 replica 3, single VM: 40MB/s on write, with about 40MB/s on read and a maximum read of 90MB/s running four read threads in parallel.
> 
> In both cases, multiple write threads had little/no impact on overall write performance.
> 
> Note that sheepd is spawned identically in each case, except for the references to each respective cluster backend. Have others experienced similar performance numbers with zookeeper/corosync? Are there any standard zookeeper configuration changes you make out of the box? We are taking the defaults for zookeeper package installation at this time.
> 
> 

Cluster driver (corosync or zookeeper) only manages membership (mainly doing
heartbeat and signaling nodes in/out) and all the IOs (guest to sheep and
inter-sheep, dog to sheep) are all p2p and not involing cluster driver at all.

I don't why you experience such a huge difference because in theory there should
be no difference.

What is startup command for sheep and QEMU in both cases?

Thanks
Yuan