[sheepdog-users] Stability issues under high workload

[Liu Yuan]

On 04/03/2013 01:54 AM, Werner van der Walt wrote:
> Hi,
> 
> I have been testing a couple of VMs (Kubuntu 12.04.2) while throwing an
> additional load against them. These are KVM VMs running under Proxmox
> 2.3 (3 x nodes) and connecting to a remote SAN box (Ubuntu 12.04
> server). Proxmox is just running the sheepdog daemon, there is no local
> sheepdog cluster configured. The nodes are connecting to a remote
> sheepdog cluster running on the SAN box. Everything is using gig
> ethernet and a dedicated VLAN set up for the Proxmox > SAN traffic. I am
> getting average throughput of ~65MB/s from the Proxmox nodes to the SAN.
> When running dd if=/dev/zero of=dd.raw count=10000000 in the VMs running
> off the SAN box it creates a 5.1GB raw file in the VM itself and pushes
> up the disc I/O to the SAN. It runs fine for a while but consistently
> crashes (freezes) after a while (sometimes 2-3 minutes, sometimes about
> 8-10 minutes). When running lower disc I/O operations in the same VMs
> they can run for hours without anything happening. When the VMs freeze
> you have to force stop them and then restart them and all works okay again.
> It looks like with the increased network I/O when dd is running it
> looses connection to the backend VM and that's what freezes it. Once you
> stop and restart it re-establishes the connection and works again the
> way it should.
> 
> Any ideas/suggestions on where to look to resolve the issue? Network,
> disc and everything else has been looked at and it more than sufficient
> to handle the requirements. Is there anything on the sheepdog side that
> can maybe be tuned?
> 

You need provide more information such as how you start sheep, and would
be nice to show some error message from sheep.log. For the VM freeze, I
guess it is QEMU's sheepdog block driver bug (Kazutaka has recently
fixed a request handling bug in QEMU, which might freeze VM). Which
version of QEMU do you run? It would be great if you can try latest QEMU
from git.qemu.org.

Thanks,
Yuan