[sheepdog-users] not all vdi are shown after cluster restart

[Liu Yuan]

On Sun, Aug 11, 2013 at 10:47:25AM +0900, MORITA Kazutaka wrote:
> At Fri, 9 Aug 2013 19:33:15 +0800,
> Liu Yuan wrote:
> > 
> > On Fri, Aug 09, 2013 at 07:10:55PM +0800, Liu Yuan wrote:
> > > On Fri, Aug 09, 2013 at 11:25:35AM +0200, Valerio Pachera wrote:
> > > > 2013/8/9 Liu Yuan <namei.unix at gmail.com>:
> > > > > Any 'failed' or error messages in sheep.log?
> > > > 
> > > > I just tried to restart the cluster a second time but it didn't help.
> > > > Here are the sheep.log messages:
> > > > 
> > > > node 0 (sheepdog001)
> > > > Aug 09 11:13:40 [main] main(797) shutdown
> > > > Aug 09 11:14:14 [main] md_add_disk(161) /mnt/sheep/dsk01/obj, nr 1
> > > > Aug 09 11:14:14 [main] md_add_disk(161) /mnt/sheep/dsk02, nr 2
> > > > Aug 09 11:14:14 [main] md_add_disk(161) /mnt/sheep/dsk03, nr 3
> > > > Aug 09 11:14:14 [main] send_join_request(1095) IPv4 ip:192.168.6.41 port:7000
> > > > Aug 09 11:14:14 [main] for_each_object_in_stale(403) /mnt/sheep/dsk01/obj/.stale
> > > > Aug 09 11:14:14 [main] for_each_object_in_stale(403) /mnt/sheep/dsk02/.stale
> > > > Aug 09 11:14:14 [main] for_each_object_in_stale(403) /mnt/sheep/dsk03/.stale
> > > > Aug 09 11:14:14 [main] init_vdi_state(195) failed to read inode header
> > > > 800e4aa600000000 0
> > > > Aug 09 11:14:14 [main] init_vdi_state(195) failed to read inode header
> > > > 80c8d12e00000000 0
> > > > Aug 09 11:14:14 [main] init_vdi_state(195) failed to read inode header
> > > > 80c8d13700000000 0
> > > > Aug 09 11:14:15 [main] init_vdi_state(195) failed to read inode header
> > > > 80f131b700000000 0
> > > > Aug 09 11:14:15 [main] init_vdi_state(195) failed to read inode header
> > > > 80c8d13e00000000 0
> > > > Aug 09 11:14:15 [main] init_vdi_state(195) failed to read inode header
> > > > 80c8d13600000000 0
> > > > Aug 09 11:14:15 [main] init_vdi_state(195) failed to read inode header
> > > > 80c8d12800000000 0
> > > > Aug 09 11:14:15 [main] init_vdi_state(195) failed to read inode header
> > > > 80c8d14400000000 0
> > > > Aug 09 11:14:15 [main] check_host_env(405) Allowed core file size 0,
> > > > suggested unlimited
> > > > Aug 09 11:14:15 [main] main(790) sheepdog daemon (version
> > > > 0.6.0_62_gdff7a77) started
> > > > Aug 09 11:14:15 [main] update_cluster_info(871) status = 4, epoch = 1,
> > > > finished: 0
> > > > Aug 09 11:15:43 [main] sd_check_join_cb(1055) 192.168.6.42:7000: ret =
> > > > 0x0, cluster_status = 0x4
> > > > Aug 09 11:15:43 [main] update_cluster_info(871) status = 4, epoch = 1,
> > > > finished: 1
> > > > Aug 09 11:15:57 [main] sd_check_join_cb(1055) 192.168.6.43:7000: ret =
> > > > 0x0, cluster_status = 0x4
> > > > Aug 09 11:15:57 [main] update_cluster_info(871) status = 4, epoch = 1,
> > > > finished: 1
> > > > Aug 09 11:16:18 [main] sd_check_join_cb(1055) 192.168.6.44:7000: ret =
> > > > 0x0, cluster_status = 0x1
> > > > Aug 09 11:16:18 [main] update_cluster_info(871) status = 1, epoch = 1,
> > > > finished: 1
> > > > Aug 09 11:16:45 [main] get_vdi_copy_number(108) No VDI copy entry for
> > > > e4aa6 found
> > > > Aug 09 11:16:45 [gway 13715] get_vdi_copy_number(108) No VDI copy
> > > > entry for e4aa6 found
> > > > Aug 09 11:16:45 [gway 13715] gateway_read_obj(60) local read
> > > > 80c8d12800000000 failed, No object found
> > > > Aug 09 11:16:45 [gway 13715] gateway_read_obj(60) local read
> > > > 80c8d12e00000000 failed, No object found
> > > > Aug 09 11:16:45 [gway 13715] gateway_read_obj(60) local read
> > > > 80c8d13600000000 failed, No object found
> > > > Aug 09 11:16:45 [gway 13715] gateway_read_obj(60) local read
> > > > 80c8d13600000000 failed, No object found
> > > > Aug 09 11:16:45 [gway 13715] gateway_read_obj(60) local read
> > > > 80c8d13700000000 failed, No object found
> > > > Aug 09 11:16:45 [gway 13715] gateway_read_obj(60) local read
> > > > 80c8d13700000000 failed, No object found
> > > > Aug 09 11:16:45 [gway 13715] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:16:45 [main] get_vdi_copy_number(108) No VDI copy entry for
> > > > c8d13e found
> > > > Aug 09 11:16:45 [gway 13715] get_vdi_copy_number(108) No VDI copy
> > > > entry for c8d13e found
> > > > Aug 09 11:16:45 [main] get_vdi_copy_number(108) No VDI copy entry for
> > > > c8d144 found
> > > > Aug 09 11:16:45 [gway 13715] get_vdi_copy_number(108) No VDI copy
> > > > entry for c8d144 found
> > > > Aug 09 11:16:45 [main] get_vdi_copy_number(108) No VDI copy entry for
> > > > f131b7 found
> > > > Aug 09 11:16:45 [gway 13715] get_vdi_copy_number(108) No VDI copy
> > > > entry for f131b7 found
> > > > Aug 09 11:17:08 [main] get_vdi_copy_number(108) No VDI copy entry for
> > > > e4aa6 found
> > > > Aug 09 11:17:08 [gway 13715] get_vdi_copy_number(108) No VDI copy
> > > > entry for e4aa6 found
> > > > Aug 09 11:17:08 [gway 13715] gateway_read_obj(60) local read
> > > > 80c8d12800000000 failed, No object found
> > > > Aug 09 11:17:08 [gway 13715] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:17:08 [gway 13715] gateway_read_obj(60) local read
> > > > 80c8d12e00000000 failed, No object found
> > > > Aug 09 11:17:08 [gway 13715] gateway_read_obj(60) local read
> > > > 80c8d13600000000 failed, No object found
> > > > Aug 09 11:17:08 [gway 13715] gateway_read_obj(60) local read
> > > > 80c8d13600000000 failed, No object found
> > > > Aug 09 11:17:08 [gway 13715] gateway_read_obj(60) local read
> > > > 80c8d13700000000 failed, No object found
> > > > Aug 09 11:17:08 [gway 13715] gateway_read_obj(60) local read
> > > > 80c8d13700000000 failed, No object found
> > > > Aug 09 11:17:08 [gway 13715] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:17:08 [main] get_vdi_copy_number(108) No VDI copy entry for
> > > > c8d13e found
> > > > Aug 09 11:17:08 [gway 13715] get_vdi_copy_number(108) No VDI copy
> > > > entry for c8d13e found
> > > > Aug 09 11:17:08 [main] get_vdi_copy_number(108) No VDI copy entry for
> > > > c8d144 found
> > > > Aug 09 11:17:08 [gway 13715] get_vdi_copy_number(108) No VDI copy
> > > > entry for c8d144 found
> > > > Aug 09 11:17:08 [main] get_vdi_copy_number(108) No VDI copy entry for
> > > > f131b7 found
> > > > Aug 09 11:17:08 [gway 13715] get_vdi_copy_number(108) No VDI copy
> > > > entry for f131b7 found
> > > > 
> > > > node 1 (sheepdog002)
> > > > Aug 09 11:13:40 [main] main(797) shutdown
> > > > Aug 09 11:15:43 [main] md_add_disk(161) /mnt/sheep/dsk01/obj, nr 1
> > > > Aug 09 11:15:43 [main] md_add_disk(161) /mnt/sheep/dsk02, nr 2
> > > > Aug 09 11:15:43 [main] md_add_disk(161) /mnt/sheep/dsk03, nr 3
> > > > Aug 09 11:15:43 [main] send_join_request(1095) IPv4 ip:192.168.6.42 port:7000
> > > > Aug 09 11:15:43 [main] for_each_object_in_stale(403) /mnt/sheep/dsk01/obj/.stale
> > > > Aug 09 11:15:43 [main] for_each_object_in_stale(403) /mnt/sheep/dsk02/.stale
> > > > Aug 09 11:15:43 [main] for_each_object_in_stale(403) /mnt/sheep/dsk03/.stale
> > > > Aug 09 11:15:45 [main] check_host_env(405) Allowed core file size 0,
> > > > suggested unlimited
> > > > Aug 09 11:15:45 [main] main(790) sheepdog daemon (version
> > > > 0.6.0_62_gdff7a77) started
> > > > Aug 09 11:15:45 [main] update_cluster_info(871) status = 4, epoch = 1,
> > > > finished: 0
> > > > Aug 09 11:15:57 [main] update_cluster_info(871) status = 4, epoch = 1,
> > > > finished: 1
> > > > Aug 09 11:16:18 [main] update_cluster_info(871) status = 1, epoch = 1,
> > > > finished: 1
> > > > Aug 09 11:16:45 [gway 3070] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:16:45 [gway 3070] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:16:45 [gway 3070] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:16:45 [gway 3070] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:16:45 [gway 3070] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:16:45 [gway 3070] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:16:45 [gway 3070] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:17:08 [gway 3070] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:17:08 [gway 3070] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:17:08 [gway 3070] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:17:08 [gway 3070] sheep_exec_req(548) failed No object found
> > > > 
> > > > node 2 (sheepdog003)
> > > > Aug 09 11:13:40 [main] main(797) shutdown
> > > > Aug 09 11:15:57 [main] md_add_disk(161) /mnt/sheep/dsk01/obj, nr 1
> > > > Aug 09 11:15:57 [main] md_add_disk(161) /mnt/sheep/dsk02, nr 2
> > > > Aug 09 11:15:57 [main] send_join_request(1095) IPv4 ip:192.168.6.43 port:7000
> > > > Aug 09 11:15:57 [main] for_each_object_in_stale(403) /mnt/sheep/dsk01/obj/.stale
> > > > Aug 09 11:15:57 [main] for_each_object_in_stale(403) /mnt/sheep/dsk02/.stale
> > > > Aug 09 11:16:00 [main] init_vdi_state(195) failed to read inode header
> > > > 80c8d12c00000000 0
> > > > Aug 09 11:16:00 [main] init_vdi_state(195) failed to read inode header
> > > > 80c8d13900000000 0
> > > > Aug 09 11:16:00 [main] check_host_env(405) Allowed core file size 0,
> > > > suggested unlimited
> > > > Aug 09 11:16:00 [main] main(790) sheepdog daemon (version
> > > > 0.6.0_62_gdff7a77) started
> > > > Aug 09 11:16:00 [main] update_cluster_info(871) status = 4, epoch = 1,
> > > > finished: 0
> > > > Aug 09 11:16:18 [main] update_cluster_info(871) status = 1, epoch = 1,
> > > > finished: 1
> > > > Aug 09 11:16:45 [gway 6116] gateway_read_obj(60) local read
> > > > 80c8d12c00000000 failed, No object found
> > > > Aug 09 11:16:45 [gway 6116] gateway_read_obj(60) local read
> > > > 80c8d13900000000 failed, No object found
> > > > Aug 09 11:16:45 [gway 6116] gateway_read_obj(60) local read
> > > > 80c8d13900000000 failed, No object found
> > > > Aug 09 11:17:08 [gway 6116] gateway_read_obj(60) local read
> > > > 80c8d12c00000000 failed, No object found
> > > > Aug 09 11:17:08 [gway 6116] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:17:08 [gway 6116] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:17:08 [gway 6116] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:17:08 [gway 6116] gateway_read_obj(60) local read
> > > > 80c8d13900000000 failed, No object found
> > > > Aug 09 11:17:08 [gway 6116] gateway_read_obj(60) local read
> > > > 80c8d13900000000 failed, No object found
> > > > 
> > > > node 3 (sheepdog004)
> > > > Aug 09 11:13:40 [main] main(797) shutdown
> > > > Aug 09 11:16:17 [main] md_add_disk(161) /mnt/sheep/dsk03, nr 1
> > > > Aug 09 11:16:17 [main] md_add_disk(161) /mnt/sheep/dsk04, nr 2
> > > > Aug 09 11:16:18 [main] send_join_request(1095) IPv4 ip:192.168.6.44 port:7000
> > > > Aug 09 11:16:18 [main] for_each_object_in_stale(403) /mnt/sheep/dsk03/.stale
> > > > Aug 09 11:16:18 [main] for_each_object_in_stale(403) /mnt/sheep/dsk04/.stale
> > > > Aug 09 11:16:20 [main] check_host_env(405) Allowed core file size 0,
> > > > suggested unlimited
> > > > Aug 09 11:16:20 [main] main(790) sheepdog daemon (version
> > > > 0.6.0_62_gdff7a77) started
> > > > Aug 09 11:16:20 [main] update_cluster_info(871) status = 1, epoch = 1,
> > > > finished: 0
> > > > Aug 09 11:16:45 [gway 25212] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:16:45 [gway 25212] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:16:45 [gway 25212] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:16:45 [gway 25212] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:16:45 [gway 25212] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:16:45 [gway 25212] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:17:08 [gway 25212] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:17:08 [gway 25212] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:17:08 [gway 25212] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:17:08 [gway 25212] sheep_exec_req(548) failed No object found
> > > > Aug 09 11:17:08 [gway 25212] sheep_exec_req(548) failed No object found
> > > 
> > > I have seen a bug in the code, both master and stable-0.6 has it. I'll post the
> > > fix soon
> > 
> > After some tests with current master, I think this problem is solved already.
> > Please update to the latest master, it is kind of stable and is the release
> > candicate for v0.7.0
> 
> Do you know which commit fixed the problem?  I think the commit should
> go into stable-0.6.

No, I didn't lookup which commit fix the problem. I just created a simple test
that master passed.

Thanks
Yuan