[sheepdog-users] Stability issues under high workload

[Werner van der Walt]

Hi Yuan,

Sheep was started using the standard boot LSB script
(/etc/init.d/sheepdog). I'm not currently using any additional startup
parameters. There are no error entries or any additional entries in
sheep.log when this happens. The VM just freezes and you have to force stop
it. Once you restart it only shows a new entry in sheep.log again where it
does the vm disk lookup.
The qemu-server version is 2.3-18 and pve-qemu-kvm is 1.4-8 which are the
ones that come with the latest version of proxmox 2.3. As pve is using a
custom qemu compile I can't do a drop in replacement with another version
compiled from git.

Regards,
Werner



On 3 April 2013 17:18, Liu Yuan <namei.unix at gmail.com> wrote:

> On 04/03/2013 01:54 AM, Werner van der Walt wrote:
> > Hi,
> >
> > I have been testing a couple of VMs (Kubuntu 12.04.2) while throwing an
> > additional load against them. These are KVM VMs running under Proxmox
> > 2.3 (3 x nodes) and connecting to a remote SAN box (Ubuntu 12.04
> > server). Proxmox is just running the sheepdog daemon, there is no local
> > sheepdog cluster configured. The nodes are connecting to a remote
> > sheepdog cluster running on the SAN box. Everything is using gig
> > ethernet and a dedicated VLAN set up for the Proxmox > SAN traffic. I am
> > getting average throughput of ~65MB/s from the Proxmox nodes to the SAN.
> > When running dd if=/dev/zero of=dd.raw count=10000000 in the VMs running
> > off the SAN box it creates a 5.1GB raw file in the VM itself and pushes
> > up the disc I/O to the SAN. It runs fine for a while but consistently
> > crashes (freezes) after a while (sometimes 2-3 minutes, sometimes about
> > 8-10 minutes). When running lower disc I/O operations in the same VMs
> > they can run for hours without anything happening. When the VMs freeze
> > you have to force stop them and then restart them and all works okay
> again.
> > It looks like with the increased network I/O when dd is running it
> > looses connection to the backend VM and that's what freezes it. Once you
> > stop and restart it re-establishes the connection and works again the
> > way it should.
> >
> > Any ideas/suggestions on where to look to resolve the issue? Network,
> > disc and everything else has been looked at and it more than sufficient
> > to handle the requirements. Is there anything on the sheepdog side that
> > can maybe be tuned?
> >
>
> You need provide more information such as how you start sheep, and would
> be nice to show some error message from sheep.log. For the VM freeze, I
> guess it is QEMU's sheepdog block driver bug (Kazutaka has recently
> fixed a request handling bug in QEMU, which might freeze VM). Which
> version of QEMU do you run? It would be great if you can try latest QEMU
> from git.qemu.org.
>
> Thanks,
> Yuan
>
> --
>
> Privacy & Confidentiality Notice: This communication and the information it
> contains is intended for the person or organisation to whom it is
> addressed. Its contents are confidential and may be protected by law.
> Unauthorised use, copying or disclosure of any of it is unlawful. If you
> are not the intended recipient or received this message in error, please
> delete it immediately or contact us.
> The views and opinions expressed in this email are those of the author and
> may not reflect the views and opinions of the author's employer.
>
> *P** **Please don't print this e-mail unless you really need to*.
>

-- 

Privacy & Confidentiality Notice: This communication and the information it 
contains is intended for the person or organisation to whom it is 
addressed. Its contents are confidential and may be protected by law. 
Unauthorised use, copying or disclosure of any of it is unlawful. If you 
are not the intended recipient or received this message in error, please 
delete it immediately or contact us.
The views and opinions expressed in this email are those of the author and 
may not reflect the views and opinions of the author's employer.

*P** **Please don't print this e-mail unless you really need to*.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wpkg.org/pipermail/sheepdog-users/attachments/20130404/b5ad0c0e/attachment-0005.html>