[sheepdog-users] while recovering, it can't boot vps

[Hitoshi Mitake]

Hi Abe-san,

On Thu, Aug 20, 2015 at 6:09 PM, 阿部　道明 <abe at atworks.co.jp> wrote:
> Hi
>
> I test the latest sheepdog v0.9.3_rc0. But I found the probrem.
>
> - environment
>  CentOS   6.5
>  sheepdog 0.9.3_rc0
>  corosync 2.3.2-1
>  5 nodes
>  3 Replication
>
> # dog node list
>   Id   Host:Port         V-Nodes       Zone
>    0   10.0.6.1:7000            91   17170442
>    1   10.0.6.2:7000            141 33947658
>    2   10.0.6.3:7000            136   50724874
>    3   10.0.6.4:7000            136   67502090
>    4   10.0.6.5:7000            137   84279306
>
> - step
> 1. stop the node2(10.0.6.3)
> 2. run the node2(10.0.6.3) and join sheepdog cluster.
> 3. while recovering,  it create the VDI(e0058b20-a4f7-43a0-89cf-aa718a73511a)
>    and boot vps with this VDI at node3(10.0.6.4).
>
> expect result
>  it boot the vps.
>
> actual result
>  it can't boot the vps with this error.
>
> qemu-kvm: -drive file=sheepdog:127.0.0.1:7000:e0058b20-a4f7-43a0-89cf-aa718a73511a,if=none,id=drive-virtio-disk0,format=raw:
> could not open disk image sheepdog:127.0.0.1:7000:e0058b20-a4f7-43a0-89cf-aa718a73511a: Operation not permitted
>
>
> After recovery complete, it can create the VDI and boot vps.
>
>
> - reproduce
> yes
>
>
> - research
> I start sheepdog with debug option.
> Because the vdi_lookup process failed, it can't boot vps.
>
>
> Aug 20 13:42:25  DEBUG [main] client_handler(974) 19, 0
> Aug 20 13:42:25  DEBUG [main] clear_client_info(915) connection seems to be dead
> Aug 20 13:42:25  DEBUG [main] clear_client_info(925) refcnt:0, fd:30, 10.0.6.1:55652
> Aug 20 13:42:25  DEBUG [main] destroy_client(906) connection from: 10.0.6.1:55652
> Aug 20 13:42:25  DEBUG [main] listen_handler(1067) accepted a new connection: 30
> Aug 20 13:42:25  DEBUG [main] listen_handler(1067) accepted a new connection: 31
> Aug 20 13:42:25  DEBUG [main] client_handler(974) 1, 0
> Aug 20 13:42:25  DEBUG [main] rx_main(835) 31, 127.0.0.1:38188
> Aug 20 13:42:25  DEBUG [main] queue_request(486) LOCK_VDI, 1
> Aug 20 13:42:25  DEBUG [main] queue_cluster_request(315) LOCK_VDI (0x7f29ac01b0e0)
> Aug 20 13:42:25  DEBUG [main] cdrv_cpg_deliver(431) 4
> Aug 20 13:42:25  DEBUG [block] do_process_work(1933) 12, 0, 89
> Aug 20 13:42:25  DEBUG [block] vdi_lookup(1249) e0058b20-a4f7-43a0-89cf-aa718a73511a left b87f20 right b87f20, 8
> Aug 20 13:42:25  DEBUG [block] do_process_work(1940) failed: 12, 0 , 89, No VDI found
> Aug 20 13:42:25  DEBUG [main] cluster_op_done(249) LOCK_VDI (0x7f29ac01b0e0)
> Aug 20 13:42:25  DEBUG [main] cdrv_cpg_deliver(431) 5
> Aug 20 13:42:25  DEBUG [main] sd_notify_handler(953) op LOCK_VDI, size: 608, from: IPv4 ip:10.0.6.4 port:7000
> Aug 20 13:42:25  DEBUG [main] client_handler(974) 4, 0
> Aug 20 13:42:25  DEBUG [main] tx_main(887) 31, 127.0.0.1:38188
> Aug 20 13:42:25  DEBUG [main] client_handler(974) 1, 0
> Aug 20 13:42:25  DEBUG [net 4601] do_read(220) connection is closed (48 bytes left)
> Aug 20 13:42:25  DEBUG [net 4601] rx_work(776) failed to read a header
> Aug 20 13:42:25  DEBUG [main] client_handler(974) 1, 0
>
>
> Do you have any idea ?

Thanks a lot for reporting and analysis. Could you provide an option
for cluster format?

In addition, didn't you find any ERROR leveled logs?

Thanks,
Hitoshi

>
> Thanks.
>
> --
> 阿部　道明 <abe at atworks.co.jp>
>
> --
> sheepdog-users mailing lists
> sheepdog-users at lists.wpkg.org
> https://lists.wpkg.org/mailman/listinfo/sheepdog-users