[sheepdog] the mounted vm of using sheepfs which open the object_cache became Read-only file system

[周鹏]

Hello,i test the sheepdog on 4 linux server.
the version of sheepdog daemon is 0.6.0_22_g8032354_dirty.
Here is the command which i use to start sheep:

#!/bin/bash
mountdir=/cloud
zookeeper="192.168.10.45:2181 192.168.10.46:2181 192.168.10.41:2181 192.168.10.47:2181"
sheep /home/gateway -l 7 -p 7000 -g 64 -z 3 -D -c zookeeper:${zookeeper}
sheep -l 7 ${mountdir}/0,${mountdir}/1,${mountdir}/2,${mountdir}/3,${mountdir}/4 -w size=100,dir=${mountdir}/cache -i host=192.168.10.46,port=7010 -p 7020 -D -c zookeeper:${zookeeper}

then i start the sheepfs to mount the vm:
#sheepfs /home/sheepfs -f

#echo centos6>/home/sheepfs/vdi/mount
#losetup -fo 525336576 /home/sheepfs/volume/centos6
#vgchange -ay /dev/vg_sheepdog
#mount /dev/vg_sheepdog/lv_root /home/centos6

After a few seconds,the sheepfs running in the foreground printed:
volume_do_sync(307): failed to flush vdi 7c7833
volume_do_sync(307): failed to flush vdi 7c7833

when i write something to the mounted vm on "/home/centos6":
[root at sheep-dell root]# echo test > /home/centos6/root/test.txt
-bash: tt: Read-only file system

so i try to find the reason, i looked up the gateway log:

Aug 26 14:36:31 [main] client_handler(812) 1, rx 0, tx 3
Aug 26 14:36:31 [main] finish_rx(616) 23, 192.168.10.46:55620
Aug 26 14:36:31 [main] queue_request(357) CREATE_AND_WRITE_OBJ, 1
Aug 26 14:36:31 [gway 2582] do_process_work(1383) 1, 7c78330000029d, 1
Aug 26 14:36:31 [gway 2582] oid_is_readonly(101) vid_is_snapshot return false.
Aug 26 14:36:31 [gway 2582] gateway_forward_request(266) 7c78330000029d
Aug 26 14:36:31 [gway 2582] sockfd_cache_get(388) 192.168.10.41:7010, idx 0
Aug 26 14:36:31 [gway 2582] sockfd_cache_get(388) 192.168.10.45:7010, idx 0
Aug 26 14:36:31 [gway 2582] sockfd_cache_get(388) 192.168.10.47:7010, idx 0
Aug 26 14:36:31 [gway 2582] gateway_forward_request(313) nr_sent 3, err 0
Aug 26 14:36:31 [gway 2582] wait_forward_request(194) 1, revents 1
Aug 26 14:36:31 [gway 2582] sockfd_cache_put(423) 192.168.10.45:7010 idx 0
Aug 26 14:36:31 [gway 2582] write_info_update(113) 3, 1
Aug 26 14:36:31 [gway 2582] wait_forward_request(194) 1, revents 1
Aug 26 14:36:31 [gway 2582] sockfd_cache_put(423) 192.168.10.47:7010 idx 0
Aug 26 14:36:31 [gway 2582] write_info_update(113) 2, 1
Aug 26 14:36:31 [gway 2582] wait_forward_request(194) 0, revents 1
Aug 26 14:36:31 [gway 2582] sockfd_cache_put(423) 192.168.10.41:7010 idx 0
Aug 26 14:36:31 [gway 2582] write_info_update(113) 1, 0
Aug 26 14:36:31 [main] client_handler(812) 4, rx 0, tx 3
Aug 26 14:36:31 [main] finish_tx(703) connection from: 23, 192.168.10.46:55620
Aug 26 14:36:31 [main] client_handler(812) 1, rx 0, tx 3
Aug 26 14:36:31 [main] finish_rx(616) 24, 192.168.10.46:55621
Aug 26 14:36:31 [main] queue_request(357) WRITE_OBJ, 1
Aug 26 14:36:31 [gway 2582] do_process_work(1383) 3, 807c783300000000, 1
Aug 26 14:36:31 [gway 2582] oid_is_readonly(95) is_data_obj return true.
Aug 26 14:36:31 [gway 2582] gateway_forward_request(266) 807c783300000000
Aug 26 14:36:31 [gway 2582] sockfd_cache_get(388) 192.168.10.41:7010, idx 0
Aug 26 14:36:31 [gway 2582] sockfd_cache_get(388) 192.168.10.45:7010, idx 0
Aug 26 14:36:31 [gway 2582] sockfd_cache_get(388) 192.168.10.47:7010, idx 0
Aug 26 14:36:31 [gway 2582] gateway_forward_request(313) nr_sent 3, err 0
Aug 26 14:36:31 [gway 2582] wait_forward_request(194) 0, revents 1
Aug 26 14:36:31 [gway 2582] sockfd_cache_put(423) 192.168.10.41:7010 idx 0
Aug 26 14:36:31 [gway 2582] write_info_update(113) 3, 0
Aug 26 14:36:31 [gway 2582] wait_forward_request(194) 1, revents 1
Aug 26 14:36:31 [gway 2582] sockfd_cache_put(423) 192.168.10.47:7010 idx 0
Aug 26 14:36:31 [gway 2582] write_info_update(113) 2, 1
Aug 26 14:36:31 [gway 2582] wait_forward_request(194) 0, revents 1
Aug 26 14:36:31 [gway 2582] sockfd_cache_put(423) 192.168.10.45:7010 idx 0
Aug 26 14:36:31 [gway 2582] write_info_update(113) 1, 0
Aug 26 14:36:31 [main] client_handler(812) 4, rx 0, tx 3
Aug 26 14:36:31 [main] finish_tx(703) connection from: 24, 192.168.10.46:55621
Aug 26 14:36:31 [main] client_handler(812) 1, rx 0, tx 3
Aug 26 14:36:31 [main] finish_rx(616) 25, 192.168.10.46:55622
Aug 26 14:36:31 [main] queue_request(357) WRITE_OBJ, 1
Aug 26 14:36:31 [gway 2582] do_process_work(1383) 3, 7c78330000029d, 1
Aug 26 14:36:31 [gway 2582] oid_is_readonly(101) vid_is_snapshot return false.
Aug 26 14:36:31 [gway 2582] gateway_forward_request(266) 7c78330000029d
Aug 26 14:36:31 [gway 2582] sockfd_cache_get(388) 192.168.10.41:7010, idx 0
Aug 26 14:36:31 [gway 2582] sockfd_cache_get(388) 192.168.10.45:7010, idx 0
Aug 26 14:36:31 [gway 2582] sockfd_cache_get(388) 192.168.10.47:7010, idx 0
Aug 26 14:36:31 [gway 2582] gateway_forward_request(313) nr_sent 3, err 0
Aug 26 14:36:31 [gway 2582] wait_forward_request(194) 0, revents 1
Aug 26 14:36:31 [gway 2582] sockfd_cache_put(423) 192.168.10.41:7010 idx 0
Aug 26 14:36:31 [gway 2582] write_info_update(113) 3, 0
Aug 26 14:36:31 [gway 2582] wait_forward_request(194) 0, revents 1
Aug 26 14:36:31 [gway 2582] sockfd_cache_put(423) 192.168.10.45:7010 idx 0
Aug 26 14:36:31 [gway 2582] write_info_update(113) 2, 0
Aug 26 14:36:31 [gway 2582] wait_forward_request(194) 0, revents 1
Aug 26 14:36:31 [gway 2582] sockfd_cache_put(423) 192.168.10.47:7010 idx 0
Aug 26 14:36:31 [gway 2582] write_info_update(113) 1, 0
Aug 26 14:36:31 [main] client_handler(812) 4, rx 0, tx 3
Aug 26 14:36:31 [main] finish_tx(703) connection from: 25, 192.168.10.46:55622

It can know from the log that "volume_do_sync" will do CREATE_AND_WRITE_OBJ and then write something to the cache.
In order to further confirmation, the source code is really to do so.
But when and how did the vm became Read-only file system, i can not understand.So please help me to analyze the cause,thanks.

BTW,when i test the sheepdog, i often meet the vm became Read-only.So another question, what can cause a read_only?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wpkg.org/pipermail/sheepdog/attachments/20130826/826b8299/attachment-0003.html>