[sheepdog-users] 50.000 iops per VM

[Stefan Priebe]

while doing 4k random I/O each system looks like this:

iostat -x 5 output:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
            2,34    0,00   12,04    2,79    0,00   82,83

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0,00     0,00    0,00    0,00     0,00     0,00 
0,00     0,00    0,00   0,00   0,00
sdb               0,00     0,20    0,00 4452,40     0,00    14,23 
6,55     0,22    0,05   0,04  18,18
sdc               0,00     0,40    0,00 4656,40     0,00    15,03 
6,61     0,31    0,07   0,05  25,16
sdd               0,00     0,00    0,00 4319,80     0,00    13,84 
6,56     0,27    0,06   0,05  22,78
sde               0,00     0,00    0,00 4851,40     0,00    15,61 
6,59     0,29    0,06   0,05  23,54


Am 03.07.2012 22:28, schrieb Stefan Priebe:
>
> Am 03.07.2012 21:37, schrieb Matthew Law:
>> On Tue, July 3, 2012 8:16 pm, Stefan Priebe wrote:
>>> I'm trying to archieve 50.000 iops write performance per VM (4k blocks).
>>> Are there any recommandations for this? I want to start with ONE VM and
>>> then starting to scale.
>>
>> What are you getting now and what is your hardware, network setup and
>> number of sheep, disks, etc?
>
> Oh i'm sorry.
>
> For testing purposes i had:
>
> 4x dedicated sheep nodes:
> - Intel Xeon E5 8 Core 3,6Ghz
> - 64GB Memory
> - 10GBE
> - 4x Intel 240GB SSD (write: 250MB/s and 25.000 iop/s random 4k)
>
> KVM Machine:
> - Dual Xeon E5 2,6Ghz
> - 64GB Memory
> - 10GBE
>
> On the KVM Machine i used a sheep process acting as the gateway with the
> following options: -G -i 32 -g 32 -W
>
> On each sheep node i used one sheep per disk: -g 32 -i 32 -W
>
> So i had a theoretical disk performance of: 400.000 iops and 4000MB/s
> writing seq 4MB blocks. As the replication was set to copies 2 it's
> 200.000 iops and 2000MB/s.
>
> KVM disk option: directsync and virtio-blk
>
> I get the following values now:
> 1 VM:
>    4kwrite: io=12544MB, bw=142678KB/s, iops=35669, runt= 90025msec
>    4kread : io=14519MB, bw=165186KB/s, iops=41296, runt= 90003msec
>    4mwrite: io=16520MB, bw=185842KB/s, iops=45, runt= 91026msec
>    4mread : io=102936MB, bw=1135MB/s, iops=283, runt= 90684msec
>
> 2 VMs:
>    4kwrite: io=7042MB, bw=80062KB/s, iops=20015, runt= 90062msec
>    4kread : io=8672MB, bw=98661KB/s, iops=24665, runt= 90004msec
>    4mwrite: io=14008MB, bw=157443KB/s, iops=38, runt= 91107msec
>    4mread : io=43924MB, bw=498462KB/s, iops=121, runt= 90234msec
>
>    4kwrite: io=6048MB, bw=68772KB/s, iops=17192, runt= 90055msec
>    4kread : io=9151MB, bw=104107KB/s, iops=26026, runt= 90006msec
>    4mwrite: io=12716MB, bw=142693KB/s, iops=34, runt= 91253msec
>    4mread : io=59616MB, bw=675648KB/s, iops=164, runt= 90353msec
>
> Greets,
> Stefan