[sheepdog-users] Stability issues under high workload
namei.unix at gmail.com
Wed Apr 3 17:18:49 CEST 2013
On 04/03/2013 01:54 AM, Werner van der Walt wrote:
> I have been testing a couple of VMs (Kubuntu 12.04.2) while throwing an
> additional load against them. These are KVM VMs running under Proxmox
> 2.3 (3 x nodes) and connecting to a remote SAN box (Ubuntu 12.04
> server). Proxmox is just running the sheepdog daemon, there is no local
> sheepdog cluster configured. The nodes are connecting to a remote
> sheepdog cluster running on the SAN box. Everything is using gig
> ethernet and a dedicated VLAN set up for the Proxmox > SAN traffic. I am
> getting average throughput of ~65MB/s from the Proxmox nodes to the SAN.
> When running dd if=/dev/zero of=dd.raw count=10000000 in the VMs running
> off the SAN box it creates a 5.1GB raw file in the VM itself and pushes
> up the disc I/O to the SAN. It runs fine for a while but consistently
> crashes (freezes) after a while (sometimes 2-3 minutes, sometimes about
> 8-10 minutes). When running lower disc I/O operations in the same VMs
> they can run for hours without anything happening. When the VMs freeze
> you have to force stop them and then restart them and all works okay again.
> It looks like with the increased network I/O when dd is running it
> looses connection to the backend VM and that's what freezes it. Once you
> stop and restart it re-establishes the connection and works again the
> way it should.
> Any ideas/suggestions on where to look to resolve the issue? Network,
> disc and everything else has been looked at and it more than sufficient
> to handle the requirements. Is there anything on the sheepdog side that
> can maybe be tuned?
You need provide more information such as how you start sheep, and would
be nice to show some error message from sheep.log. For the VM freeze, I
guess it is QEMU's sheepdog block driver bug (Kazutaka has recently
fixed a request handling bug in QEMU, which might freeze VM). Which
version of QEMU do you run? It would be great if you can try latest QEMU
More information about the sheepdog-users