[sheepdog] [Qemu-devel] [PATCH] sheepdog: fix overlapping metadata update
Vasiliy Tolstov
v.tolstov at selfip.ru
Sun Aug 2 14:07:36 CEST 2015
to compare fio results from host system with file places on the ssd
(ext4,nodiscard, trimmed before run):
randrw: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
2.0.8
Starting 10 processes
Jobs: 10 (f=10): [mmmmmmmmmm] [100.0% done] [50892K/49944K /s]
[12.8K/12.5K iops] [eta 00m:00s]
randrw: (groupid=0, jobs=10): err= 0: pid=7439
read : io=739852KB, bw=18491KB/s, iops=4622 , runt= 40012msec
slat (usec): min=2 , max=12573 , avg=308.09, stdev=955.59
clat (usec): min=137 , max=107321 , avg=12750.90, stdev=7865.72
lat (usec): min=167 , max=107364 , avg=13059.31, stdev=7932.79
clat percentiles (usec):
| 1.00th=[ 2512], 5.00th=[ 4192], 10.00th=[ 5280], 20.00th=[ 6816],
| 30.00th=[ 8096], 40.00th=[ 9408], 50.00th=[10816], 60.00th=[12480],
| 70.00th=[14528], 80.00th=[17536], 90.00th=[22656], 95.00th=[28032],
| 99.00th=[40192], 99.50th=[45824], 99.90th=[59648], 99.95th=[68096],
| 99.99th=[94720]
bw (KB/s) : min= 0, max= 6679, per=27.96%, avg=5170.37, stdev=665.70
write: io=737492KB, bw=18432KB/s, iops=4607 , runt= 40012msec
slat (usec): min=4 , max=12670 , avg=305.68, stdev=945.24
clat (usec): min=31 , max=107147 , avg=11268.07, stdev=7542.08
lat (usec): min=72 , max=107181 , avg=11574.10, stdev=7615.99
clat percentiles (usec):
| 1.00th=[ 1624], 5.00th=[ 3216], 10.00th=[ 4256], 20.00th=[ 5664],
| 30.00th=[ 6816], 40.00th=[ 8032], 50.00th=[ 9408], 60.00th=[10944],
| 70.00th=[12864], 80.00th=[15552], 90.00th=[20608], 95.00th=[26240],
| 99.00th=[38144], 99.50th=[43264], 99.90th=[55552], 99.95th=[63744],
| 99.99th=[89600]
bw (KB/s) : min= 0, max= 6767, per=27.39%, avg=5047.81, stdev=992.62
lat (usec) : 50=0.01%, 100=0.01%, 250=0.06%, 500=0.12%, 750=0.06%
lat (usec) : 1000=0.06%
lat (msec) : 2=0.74%, 4=5.47%, 10=42.71%, 20=38.26%, 50=12.26%
lat (msec) : 100=0.25%, 250=0.01%
cpu : usr=0.27%, sys=67.69%, ctx=339142, majf=0, minf=70
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=184963/w=184373/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=739852KB, aggrb=18490KB/s, minb=18490KB/s, maxb=18490KB/s,
mint=40012msec, maxt=40012msec
WRITE: io=737492KB, aggrb=18431KB/s, minb=18431KB/s, maxb=18431KB/s,
mint=40012msec, maxt=40012msec
Disk stats (read/write):
dm-5: ios=183885/183290, merge=0/0, ticks=1070388/977428,
in_queue=2053044, util=35.35%, aggrios=184963/184383, aggrmerge=0/3,
aggrticks=1072368/978452, aggrin_queue=2052424, aggrutil=35.39%
sdd: ios=184963/184383, merge=0/3, ticks=1072368/978452,
in_queue=2052424, util=35.39%
2015-08-02 14:52 GMT+03:00 Vasiliy Tolstov <v.tolstov at selfip.ru>:
> 2015-07-31 15:08 GMT+03:00 Vasiliy Tolstov <v.tolstov at selfip.ru>:
>> Please wait to performance comparison. As i see Liu's patch may be
>> more slow then Hitoshi.
>
>
> I'm switch to local cluster driver to test only local ssd and not
> network overhead.
>
> But now Liu's qemu does not able to completely install vps:
> sheep runs as:
> /usr/sbin/sheep --log level=debug format=server dir=/var/log/
> dst=default --bindaddr 0.0.0.0 --cluster local --directio --upgrade
> --myaddr 192.168.240.134 --pidfile /var/run/sheepdog.pid
> /var/lib/sheepdog/sd_meta
> /var/lib/sheepdog/vg0-sd_data,/var/lib/sheepdog/vg1-sd_data,/var/lib/sheepdog/vg2-sd_data,/var/lib/sheepdog/vg3-sd_data,/var/lib/sheepdog/vg4-sd_data,/var/lib/sheepdog/vg5-sd_data
>
> qemu runs as:
> qemu-system-x86_64 -enable-kvm -name 30681 -S -machine
> pc-i440fx-1.7,accel=kvm,usb=off -m 1024 -realtime
> mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid
> 49771611-5e45-70c9-1a78-00000043bf91 -no-user-config -nodefaults
> -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/30681.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
> -no-shutdown -boot strict=on -device
> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
> virtio-scsi-pci,id=scsi0,num_queues=1,bus=pci.0,addr=0x4 -device
> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive
> file=sheepdog:30681,if=none,id=drive-scsi0-0-0-0,format=raw,cache=none,discard=ignore,aio=native
> -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
> -drive if=none,id=drive-scsi0-0-1-0,readonly=on,format=raw -device
> scsi-cd,bus=scsi0.0,channel=0,scsi-id=1,lun=0,drive=drive-scsi0-0-1-0,id=scsi0-0-1-0
> -netdev tap,fd=37,id=hostnet0,vhost=on,vhostfd=39 -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:00:42:8d,bus=pci.0,addr=0x3,rombar=0
> -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -chardev
> socket,id=charchannel0,path=/var/lib/libvirt/qemu/30681.agent,server,nowait
> -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0
> -device usb-mouse,id=input0 -device usb-kbd,id=input1 -vnc
> [::]:0,password -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2
> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -object
> rng-random,id=objrng0,filename=/dev/random -device
> virtio-rng-pci,rng=objrng0,id=rng0,max-bytes=1024,period=2000,bus=pci.0,addr=0x7
> -msg timestamp=on
>
> sheep.log:
> https://gist.github.com/raw/31201454b0afe1b86d00
>
> dmesg:
> [Sun Aug 2 14:48:45 2015] qemu-system-x86[6894]: segfault at 401364
> ip 00007f212f060ff8 sp 00007f213efe5f40 error 4 in
> qemu-system-x86_64[7f212ecbc000+518000]
>
> fio job file:
> [randrw]
> blocksize=4k
> filename=/dev/sdb
> rw=randrw
> direct=1
> buffered=0
> ioengine=libaio
> iodepth=32
> group_reporting
> numjobs=10
> runtime=40
>
> For clean test i'm create new vdi and attach it to running vm: qemu
> cache=none, io=native, discard=ignore
> Hitoshi patch:
> randrw: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
> ...
> fio-2.1.11
> Starting 10 processes
> Jobs: 10 (f=10): [m(10)] [100.0% done] [13414KB/12291KB/0KB /s]
> [3353/3072/0 iops] [eta 00m:00s]
> randrw: (groupid=0, jobs=10): err= 0: pid=1318: Sun Aug 2 11:51:25 2015
> read : io=368004KB, bw=9191.9KB/s, iops=2297, runt= 40036msec
> slat (usec): min=1, max=408908, avg=1316.35, stdev=10057.52
> clat (usec): min=229, max=972302, avg=67203.98, stdev=65126.08
> lat (usec): min=294, max=972306, avg=68520.61, stdev=66741.02
> clat percentiles (msec):
> | 1.00th=[ 4], 5.00th=[ 16], 10.00th=[ 30], 20.00th=[ 36],
> | 30.00th=[ 40], 40.00th=[ 44], 50.00th=[ 48], 60.00th=[ 52],
> | 70.00th=[ 59], 80.00th=[ 85], 90.00th=[ 128], 95.00th=[ 186],
> | 99.00th=[ 367], 99.50th=[ 441], 99.90th=[ 570], 99.95th=[ 619],
> | 99.99th=[ 799]
> bw (KB /s): min= 6, max= 1824, per=10.19%, avg=936.59, stdev=434.25
> write: io=366964KB, bw=9165.9KB/s, iops=2291, runt= 40036msec
> slat (usec): min=1, max=338238, avg=1320.61, stdev=9586.81
> clat (usec): min=912, max=972334, avg=69411.58, stdev=64715.54
> lat (usec): min=920, max=972338, avg=70732.51, stdev=66193.96
> clat percentiles (msec):
> | 1.00th=[ 8], 5.00th=[ 30], 10.00th=[ 33], 20.00th=[ 38],
> | 30.00th=[ 42], 40.00th=[ 45], 50.00th=[ 49], 60.00th=[ 53],
> | 70.00th=[ 60], 80.00th=[ 86], 90.00th=[ 128], 95.00th=[ 184],
> | 99.00th=[ 379], 99.50th=[ 441], 99.90th=[ 570], 99.95th=[ 635],
> | 99.99th=[ 824]
> bw (KB /s): min= 7, max= 1771, per=10.21%, avg=935.65, stdev=439.55
> lat (usec) : 250=0.01%, 500=0.02%, 750=0.05%, 1000=0.04%
> lat (msec) : 2=0.20%, 4=0.57%, 10=1.53%, 20=1.78%, 50=50.50%
> lat (msec) : 100=28.04%, 250=14.56%, 500=2.47%, 750=0.23%, 1000=0.03%
> cpu : usr=0.17%, sys=0.26%, ctx=16905, majf=0, minf=78
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.8%, >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
> issued : total=r=92001/w=91741/d=0, short=r=0/w=0/d=0
> latency : target=0, window=0, percentile=100.00%, depth=32
>
> Run status group 0 (all jobs):
> READ: io=368004KB, aggrb=9191KB/s, minb=9191KB/s, maxb=9191KB/s,
> mint=40036msec, maxt=40036msec
> WRITE: io=366964KB, aggrb=9165KB/s, minb=9165KB/s, maxb=9165KB/s,
> mint=40036msec, maxt=40036msec
>
> Disk stats (read/write):
> sdb: ios=91739/91385, merge=55/0, ticks=2854936/3040936,
> in_queue=5901740, util=99.82%
>
>
> --
> Vasiliy Tolstov,
> e-mail: v.tolstov at selfip.ru
--
Vasiliy Tolstov,
e-mail: v.tolstov at selfip.ru
More information about the sheepdog
mailing list