[sheepdog] filesystem errors when using sheepdog

[Vasiliy Tolstov]

I'm using latest sheepdog fomr master branch. Qemu runs with io=native
and cache=none
sheepdog cluster formatted with -c 2 and store format tree.

Sometimes when i'm doing write (5Gb data inside virtual machine) i
have file system errors like in attached image.

sheepdog runs in 8 nodes like:
/usr/sbin/sheep --log level=debug format=server dst=syslog --bindaddr
0.0.0.0 --cluster corosync --directio --upgrade --zone 1 --myaddr
192.168.240.133 --pidfile /var/run/sheepdog.pid
/var/lib/sheepdog/sd_meta
/var/lib/sheepdog/vg0-sd_data,/var/lib/sheepdog/vg1-sd_data,/var/lib/sheepdog/vg2-sd_data,/var/lib/sheepdog/vg3-sd_data,/var/lib/sheepdog/vg4-sd_data,/var/lib/sheepdog/vg5-sd_data

each sd_data - ext4 formatted disk on lv (ssd disk)

syslog logs:
Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [gway
36188] sockfd_cache_get_long(347) 192.168.240.132:7000, idx 0
Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:   WARN [gway
36188] sheep_exec_req(1188) failed No object found, remote address:
192.168.240.132:7000, op name: R
EAD_PEER
Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [gway
36188] sockfd_cache_put_long(384) 192.168.240.132:7000 idx 0
Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [gway
36188] do_process_work(2124) failed: 2, c3f7110000024e , 8, No object
found
Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [main]
gateway_op_done(145) unhandled error No object found
Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [main]
client_handler(990) 4, 0
Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [main]
tx_main(899) 353, 127.0.0.1:41198
Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [main]
client_handler(990) 1, 0
Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [main]
rx_main(841) 353, 127.0.0.1:41198
Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [main]
queue_request(489) READ_OBJ, 1
Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [gway
36170] do_process_work(2117) 2, c3f7110000024e, 8
Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [gway
36170] sockfd_cache_get_long(347) 192.168.240.132:7000, idx 0
Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:   WARN [gway
36170] sheep_exec_req(1188) failed No object found, remote address:
192.168.240.132:7000, op name: R
EAD_PEER
Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [gway
36170] sockfd_cache_put_long(384) 192.168.240.132:7000 idx 0
Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [gway
36170] do_process_work(2124) failed: 2, c3f7110000024e , 8, No object
found
Jul  4 00:06:27 cn33.z1.mn2.simplecloud.ru sheep[20822]:  DEBUG [main]
gateway_op_done(145) unhandled error No object found


-- 
Vasiliy Tolstov,
e-mail: v.tolstov at selfip.ru
-------------- next part --------------
A non-text attachment was scrubbed...
Name: shot-2015-07-04_00-06-14.png
Type: image/png
Size: 333288 bytes
Desc: not available
URL: <http://lists.wpkg.org/pipermail/sheepdog/attachments/20150704/56ef7548/attachment-0001.png>