[sheepdog] the mounted vm of using sheepfs which open the object_cache became Read-only file system

[Liu Yuan]

On Mon, Aug 26, 2013 at 07:33:10AM +0000, 周鹏 wrote:
> Hello,i test the sheepdog on 4 linux server.
> the version of sheepdog daemon is 0.6.0_22_g8032354_dirty.
> Here is the command which i use to start sheep:
> 
> #!/bin/bash
> mountdir=/cloud
> zookeeper="192.168.10.45:2181 192.168.10.46:2181 192.168.10.41:2181 192.168.10.47:2181"
> sheep /home/gateway -l 7 -p 7000 -g 64 -z 3 -D -c zookeeper:${zookeeper}
> sheep -l 7 ${mountdir}/0,${mountdir}/1,${mountdir}/2,${mountdir}/3,${mountdir}/4 -w size=100,dir=${mountdir}/cache -i host=192.168.10.46,port=7010 -p 7020 -D -c zookeeper:${zookeeper}
> 
> then i start the sheepfs to mount the vm:
> #sheepfs /home/sheepfs -f
> 
> #echo centos6>/home/sheepfs/vdi/mount
> #losetup -fo 525336576 /home/sheepfs/volume/centos6
> #vgchange -ay /dev/vg_sheepdog
> #mount /dev/vg_sheepdog/lv_root /home/centos6
> 
> After a few seconds,the sheepfs running in the foreground printed:
> volume_do_sync(307): failed to flush vdi 7c7833
> volume_do_sync(307): failed to flush vdi 7c7833
> 
> when i write something to the mounted vm on "/home/centos6":
> [root at sheep-dell root]# echo test > /home/centos6/root/test.txt
> -bash: tt: Read-only file system
> 
> so i try to find the reason, i looked up the gateway log:
> 
> Aug 26 14:36:31 [main] client_handler(812) 1, rx 0, tx 3
> Aug 26 14:36:31 [main] finish_rx(616) 23, 192.168.10.46:55620
> Aug 26 14:36:31 [main] queue_request(357) CREATE_AND_WRITE_OBJ, 1
> Aug 26 14:36:31 [gway 2582] do_process_work(1383) 1, 7c78330000029d, 1
> Aug 26 14:36:31 [gway 2582] oid_is_readonly(101) vid_is_snapshot return false.
> Aug 26 14:36:31 [gway 2582] gateway_forward_request(266) 7c78330000029d
> Aug 26 14:36:31 [gway 2582] sockfd_cache_get(388) 192.168.10.41:7010, idx 0
> Aug 26 14:36:31 [gway 2582] sockfd_cache_get(388) 192.168.10.45:7010, idx 0
> Aug 26 14:36:31 [gway 2582] sockfd_cache_get(388) 192.168.10.47:7010, idx 0
> Aug 26 14:36:31 [gway 2582] gateway_forward_request(313) nr_sent 3, err 0
> Aug 26 14:36:31 [gway 2582] wait_forward_request(194) 1, revents 1
> Aug 26 14:36:31 [gway 2582] sockfd_cache_put(423) 192.168.10.45:7010 idx 0
> Aug 26 14:36:31 [gway 2582] write_info_update(113) 3, 1
> Aug 26 14:36:31 [gway 2582] wait_forward_request(194) 1, revents 1
> Aug 26 14:36:31 [gway 2582] sockfd_cache_put(423) 192.168.10.47:7010 idx 0
> Aug 26 14:36:31 [gway 2582] write_info_update(113) 2, 1
> Aug 26 14:36:31 [gway 2582] wait_forward_request(194) 0, revents 1
> Aug 26 14:36:31 [gway 2582] sockfd_cache_put(423) 192.168.10.41:7010 idx 0
> Aug 26 14:36:31 [gway 2582] write_info_update(113) 1, 0
> Aug 26 14:36:31 [main] client_handler(812) 4, rx 0, tx 3
> Aug 26 14:36:31 [main] finish_tx(703) connection from: 23, 192.168.10.46:55620
> Aug 26 14:36:31 [main] client_handler(812) 1, rx 0, tx 3
> Aug 26 14:36:31 [main] finish_rx(616) 24, 192.168.10.46:55621
> Aug 26 14:36:31 [main] queue_request(357) WRITE_OBJ, 1
> Aug 26 14:36:31 [gway 2582] do_process_work(1383) 3, 807c783300000000, 1
> Aug 26 14:36:31 [gway 2582] oid_is_readonly(95) is_data_obj return true.
> Aug 26 14:36:31 [gway 2582] gateway_forward_request(266) 807c783300000000
> Aug 26 14:36:31 [gway 2582] sockfd_cache_get(388) 192.168.10.41:7010, idx 0
> Aug 26 14:36:31 [gway 2582] sockfd_cache_get(388) 192.168.10.45:7010, idx 0
> Aug 26 14:36:31 [gway 2582] sockfd_cache_get(388) 192.168.10.47:7010, idx 0
> Aug 26 14:36:31 [gway 2582] gateway_forward_request(313) nr_sent 3, err 0
> Aug 26 14:36:31 [gway 2582] wait_forward_request(194) 0, revents 1
> Aug 26 14:36:31 [gway 2582] sockfd_cache_put(423) 192.168.10.41:7010 idx 0
> Aug 26 14:36:31 [gway 2582] write_info_update(113) 3, 0
> Aug 26 14:36:31 [gway 2582] wait_forward_request(194) 1, revents 1
> Aug 26 14:36:31 [gway 2582] sockfd_cache_put(423) 192.168.10.47:7010 idx 0
> Aug 26 14:36:31 [gway 2582] write_info_update(113) 2, 1
> Aug 26 14:36:31 [gway 2582] wait_forward_request(194) 0, revents 1
> Aug 26 14:36:31 [gway 2582] sockfd_cache_put(423) 192.168.10.45:7010 idx 0
> Aug 26 14:36:31 [gway 2582] write_info_update(113) 1, 0
> Aug 26 14:36:31 [main] client_handler(812) 4, rx 0, tx 3
> Aug 26 14:36:31 [main] finish_tx(703) connection from: 24, 192.168.10.46:55621
> Aug 26 14:36:31 [main] client_handler(812) 1, rx 0, tx 3
> Aug 26 14:36:31 [main] finish_rx(616) 25, 192.168.10.46:55622
> Aug 26 14:36:31 [main] queue_request(357) WRITE_OBJ, 1
> Aug 26 14:36:31 [gway 2582] do_process_work(1383) 3, 7c78330000029d, 1
> Aug 26 14:36:31 [gway 2582] oid_is_readonly(101) vid_is_snapshot return false.
> Aug 26 14:36:31 [gway 2582] gateway_forward_request(266) 7c78330000029d
> Aug 26 14:36:31 [gway 2582] sockfd_cache_get(388) 192.168.10.41:7010, idx 0
> Aug 26 14:36:31 [gway 2582] sockfd_cache_get(388) 192.168.10.45:7010, idx 0
> Aug 26 14:36:31 [gway 2582] sockfd_cache_get(388) 192.168.10.47:7010, idx 0
> Aug 26 14:36:31 [gway 2582] gateway_forward_request(313) nr_sent 3, err 0
> Aug 26 14:36:31 [gway 2582] wait_forward_request(194) 0, revents 1
> Aug 26 14:36:31 [gway 2582] sockfd_cache_put(423) 192.168.10.41:7010 idx 0
> Aug 26 14:36:31 [gway 2582] write_info_update(113) 3, 0
> Aug 26 14:36:31 [gway 2582] wait_forward_request(194) 0, revents 1
> Aug 26 14:36:31 [gway 2582] sockfd_cache_put(423) 192.168.10.45:7010 idx 0
> Aug 26 14:36:31 [gway 2582] write_info_update(113) 2, 0
> Aug 26 14:36:31 [gway 2582] wait_forward_request(194) 0, revents 1
> Aug 26 14:36:31 [gway 2582] sockfd_cache_put(423) 192.168.10.47:7010 idx 0
> Aug 26 14:36:31 [gway 2582] write_info_update(113) 1, 0
> Aug 26 14:36:31 [main] client_handler(812) 4, rx 0, tx 3
> Aug 26 14:36:31 [main] finish_tx(703) connection from: 25, 192.168.10.46:55622
> 
> It can know from the log that "volume_do_sync" will do CREATE_AND_WRITE_OBJ and then write something to the cache.
> In order to further confirmation, the source code is really to do so.
> But when and how did the vm became Read-only file system, i can not understand.So please help me to analyze the cause,thanks.

If it is snapshot, it is read-only vdi.

> 
> BTW,when i test the sheepdog, i often meet the vm became Read-only.So another question, what can cause a read_only?
> 

Please use latest stable-0.7 or stable-0.6 branch and report problems if any.

Thanks
Yuan