[sheepdog] filesystem errors when using sheepdog

[Vasiliy Tolstov]

2015-07-04 0:11 GMT+03:00 Vasiliy Tolstov <v.tolstov at selfip.ru>:
> I'm using latest sheepdog fomr master branch. Qemu runs with io=native
> and cache=none
> sheepdog cluster formatted with -c 2 and store format tree.
>
> Sometimes when i'm doing write (5Gb data inside virtual machine) i
> have file system errors like in attached image.
>
> sheepdog runs in 8 nodes like:
> /usr/sbin/sheep --log level=debug format=server dst=syslog --bindaddr
> 0.0.0.0 --cluster corosync --directio --upgrade --zone 1 --myaddr
> 192.168.240.133 --pidfile /var/run/sheepdog.pid
> /var/lib/sheepdog/sd_meta
> /var/lib/sheepdog/vg0-sd_data,/var/lib/sheepdog/vg1-sd_data,/var/lib/sheepdog/vg2-sd_data,/var/lib/sheepdog/vg3-sd_data,/var/lib/sheepdog/vg4-sd_data,/var/lib/sheepdog/vg5-sd_data
>
> each sd_data - ext4 formatted disk on lv (ssd disk)
>
> syslog logs:
> Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [gway
> 36188] sockfd_cache_get_long(347) 192.168.240.132:7000, idx 0
> Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:   WARN [gway
> 36188] sheep_exec_req(1188) failed No object found, remote address:
> 192.168.240.132:7000, op name: R
> EAD_PEER
> Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [gway
> 36188] sockfd_cache_put_long(384) 192.168.240.132:7000 idx 0
> Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [gway
> 36188] do_process_work(2124) failed: 2, c3f7110000024e , 8, No object
> found
> Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [main]
> gateway_op_done(145) unhandled error No object found
> Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [main]
> client_handler(990) 4, 0
> Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [main]
> tx_main(899) 353, 127.0.0.1:41198
> Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [main]
> client_handler(990) 1, 0
> Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [main]
> rx_main(841) 353, 127.0.0.1:41198
> Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [main]
> queue_request(489) READ_OBJ, 1
> Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [gway
> 36170] do_process_work(2117) 2, c3f7110000024e, 8
> Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [gway
> 36170] sockfd_cache_get_long(347) 192.168.240.132:7000, idx 0
> Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:   WARN [gway
> 36170] sheep_exec_req(1188) failed No object found, remote address:
> 192.168.240.132:7000, op name: R
> EAD_PEER
> Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [gway
> 36170] sockfd_cache_put_long(384) 192.168.240.132:7000 idx 0
> Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [gway
> 36170] do_process_work(2124) failed: 2, c3f7110000024e , 8, No object
> found
> Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [main]
> gateway_op_done(145) unhandled error No object found


qemu 2.3.0, also i have this errors with io=threads and cache=writeback

-- 
Vasiliy Tolstov,
e-mail: v.tolstov at selfip.ru