[sheepdog-users] Not enough active nodes for consistency-check

[Hitoshi Mitake]

At Wed, 18 Dec 2013 14:34:52 +0100,
Valerio Pachera wrote:
> 
> [1  <multipart/alternative (7bit)>]
> [1.1  <text/plain; UTF-8 (7bit)>]
> I'm doing a series of test on 2 nodes cluster.
> 
> I ended up not being able to add a 3th node and unable to run vdi check.
> 
> root at test006:~# dog node list
>   Id   Host:Port         V-Nodes       Zone
>    0   192.168.10.4:7000        67   67807424
>    1   192.168.10.6:7000        61  101361856
> 
> root at test006:~# dog vdi list
>   Name        Id    Size    Used  Shared    Creation time   VDI id  Copies
> Tag
>   wheezy_template3     0   10 GB  1.8 GB  0.0 MB 2013-12-18 12:34
> 832ac1     3
> 
> root at test006:~# dog vdi check wheezy_template3
> ABORT: Not enough active nodes for consistency-check
> 
> root at test006:~# dog cluster info
> Cluster status: running, auto-recovery enabled
> Cluster created at Wed Dec 18 09:51:34 2013
> Epoch Time           Version
> 2013-12-18 14:26:46     11 [192.168.10.4:7000, 192.168.10.6:7000]
> 2013-12-18 14:26:13     10 [192.168.10.4:7000, 192.168.10.5:7000,
> 192.168.10.6:7000]
> 2013-12-18 14:23:22      9 [192.168.10.4:7000, 192.168.10.6:7000]
> 2013-12-18 14:22:50      8 [192.168.10.4:7000, 192.168.10.5:7000,
> 192.168.10.6:7000]
> 2013-12-18 13:06:38      7 [192.168.10.4:7000, 192.168.10.6:7000]
> 2013-12-18 13:06:05      6 [192.168.10.4:7000, 192.168.10.5:7000,
> 192.168.10.6:7000]
> 2013-12-18 12:51:04      5 [192.168.10.4:7000, 192.168.10.6:7000]
> 2013-12-18 12:44:52      4 [192.168.10.6:7000]
> 2013-12-18 12:41:39      3 [192.168.10.4:7000, 192.168.10.6:7000]
> 2013-12-18 12:37:34      2 [192.168.10.6:7000]
> 2013-12-18 09:51:34      1 [192.168.10.4:7000, 192.168.10.6:7000]
> 
> 
> Sheep.log of the 3th node (test005):
> 
> Dec 18 14:26:13   INFO [main] md_add_disk(141) /mnt/sheep/dsk01, nr 1
> Dec 18 14:26:13   INFO [main] md_add_disk(141) /mnt/sheep/dsk02, nr 2
> Dec 18 14:26:14  ERROR [main] get_local_addr(595) found IPv4 address
> Dec 18 14:26:14   INFO [main] send_join_request(770) IPv4 ip:192.168.10.5
> port:7000
> Dec 18 14:26:14   INFO [main] main(880) sheepdog daemon (version 0.7.6_rc1)
> started
> Dec 18 14:26:14  ERROR [main] init_vdi_state(188) failed to read inode
> header 8083290e00000000 0
> Dec 18 14:26:14  EMERG [main] setup_backend_store(553) PANIC: failed to
> initialize store
> Dec 18 14:26:14  EMERG [main] crash_handler(257) sheep exits unexpectedly
> (Aborted).
> Dec 18 14:26:14  EMERG [main] sd_backtrace(895) sheep.c:259: crash_handler
> Dec 18 14:26:14  EMERG [main] sd_backtrace(909)
> /lib/x86_64-linux-gnu/libpthread.so.0(+0xf02f) [0x7f3a91cf002f]
> Dec 18 14:26:14  EMERG [main] sd_backtrace(909)
> /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x34) [0x7f3a910e3474]
> Dec 18 14:26:14  EMERG [main] sd_backtrace(909)
> /lib/x86_64-linux-gnu/libc.so.6(abort+0x17f) [0x7f3a910e66ef]
> Dec 18 14:26:14  EMERG [main] sd_backtrace(895) group.c:564:
> setup_backend_store
> Dec 18 14:26:14  EMERG [main] sd_backtrace(895) zookeeper.c:911:
> zk_handle_accept
> Dec 18 14:26:14  EMERG [main] sd_backtrace(895) zookeeper.c:1068:
> zk_event_handler
> Dec 18 14:26:14  EMERG [main] sd_backtrace(895) event.c:210: do_event_loop
> Dec 18 14:26:14  EMERG [main] sd_backtrace(895) sheep.c:885: main
> Dec 18 14:26:14  EMERG [main] sd_backtrace(909)
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfc) [0x7f3a910cfeac]
> Dec 18 14:26:14  EMERG [main] sd_backtrace(909) sheep() [0x405f28]
> Dec 18 14:26:14  ERROR [main] crash_handler(524) sheep pid 32050 exited
> unexpectedly.
> 
> What do you think about this?

The error message produced by "dog vdi check" means that the current
cluster doesn't have enough node for checking and repairing
objects. You are trying to check/repair VDI which is 3 replicated on
the cluster which only has 2 nodes. dog can't replicate more than 2
objects on the cluster, so dog vdi check aborted.

Thanks,
Hitoshi