[sheepdog-users] Stability issues under high workload

[Werner van der Walt]

Hi,

I have been testing a couple of VMs (Kubuntu 12.04.2) while throwing an
additional load against them. These are KVM VMs running under Proxmox 2.3
(3 x nodes) and connecting to a remote SAN box (Ubuntu 12.04 server).
Proxmox is just running the sheepdog daemon, there is no local sheepdog
cluster configured. The nodes are connecting to a remote sheepdog cluster
running on the SAN box. Everything is using gig ethernet and a dedicated
VLAN set up for the Proxmox > SAN traffic. I am getting average throughput
of ~65MB/s from the Proxmox nodes to the SAN.
When running dd if=/dev/zero of=dd.raw count=10000000 in the VMs running
off the SAN box it creates a 5.1GB raw file in the VM itself and pushes up
the disc I/O to the SAN. It runs fine for a while but consistently crashes
(freezes) after a while (sometimes 2-3 minutes, sometimes about 8-10
minutes). When running lower disc I/O operations in the same VMs they can
run for hours without anything happening. When the VMs freeze you have to
force stop them and then restart them and all works okay again.
It looks like with the increased network I/O when dd is running it looses
connection to the backend VM and that's what freezes it. Once you stop and
restart it re-establishes the connection and works again the way it should.

Any ideas/suggestions on where to look to resolve the issue? Network, disc
and everything else has been looked at and it more than sufficient to
handle the requirements. Is there anything on the sheepdog side that can
maybe be tuned?

Thanks.

Werner

-- 

Privacy & Confidentiality Notice: This communication and the information it 
contains is intended for the person or organisation to whom it is 
addressed. Its contents are confidential and may be protected by law. 
Unauthorised use, copying or disclosure of any of it is unlawful. If you 
are not the intended recipient or received this message in error, please 
delete it immediately or contact us.
The views and opinions expressed in this email are those of the author and 
may not reflect the views and opinions of the author's employer.

*P** **Please don't print this e-mail unless you really need to*.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wpkg.org/pipermail/sheepdog-users/attachments/20130402/92899ef2/attachment-0004.html>