[sheepdog-users] sheepdog with corosync crashed

[Vasiliy Tolstov]

I'm testing sheepdog cluster (format with -c 8:3) nodes 2 (i'm expect
to add more nodes today)
dog cluster info
Cluster status: running, auto-recovery enabled

Cluster created at Tue Jun 23 11:38:55 2015

Epoch Time           Version
2015-06-23 13:18:39      6 [192.168.240.132:7000, 192.168.240.133:7000]
2015-06-23 13:01:39      5 [192.168.240.133:7000]
2015-06-23 12:55:32      4 [192.168.240.132:7000, 192.168.240.133:7000]
2015-06-23 12:29:07      3 [192.168.240.133:7000]
2015-06-23 11:45:29      2 [192.168.240.132:7000, 192.168.240.133:7000]
2015-06-23 11:38:55      1 [192.168.240.133:7000]

23 12:55:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
queue_request(486) READ_DEL_VDIS, 1
Jun 23 12:55:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [io
18013] do_process_work(1938) c9, 0, 4
Jun 23 12:55:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
client_handler(974) 4, 0
Jun 23 12:55:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
tx_main(887) 20, 127.0.0.1:56353
Jun 23 12:55:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
client_handler(974) 19, 0
Jun 23 12:55:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
clear_client_info(915) connection seems to be dead
Jun 23 12:55:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
clear_client_info(925) refcnt:0, fd:20, 127.0.0.1:56353
Jun 23 12:55:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
destroy_client(906) connection from: 127.0.0.1:56353
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
cdrv_cpg_deliver(431) 2
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
sd_leave_handler(1160) leave IPv4 ip:192.168.240.132 port:7000
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
sd_leave_handler(1162) IPv4 ip:192.168.240.133 port:7000
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
recalculate_vnodes(127) node IPv4 ip:192.168.240.133 port:7000 has 128
vnodes, free space 2707208527872
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
update_epoch_log(26) update epoch: 5, 1
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
for_each_object_in_wd(453) Create 6 threads for all path
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [rw
18071] prepare_object_list(1055) 5
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [rw
18071] wait_get_vdis_done(802) waiting for vdi list
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [rw
18071] wait_get_vdis_done(809) vdi list ready
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
sockfd_cache_del_node(475) 192.168.240.132:7000, count 1
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [rw
18071] fetch_object_list(974) 192.168.240.133:7000
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [rw
18071] sockfd_cache_get_long(338) 192.168.240.133:7000, idx 0
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
client_handler(974) 1, 0
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
rx_main(835) 17, 192.168.240.133:46965
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
queue_request(486) GET_OBJ_LIST, 1
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [io
18070] do_process_work(1938) a1, 0, 5
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
client_handler(974) 4, 0
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
tx_main(887) 17, 192.168.240.133:46965
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [rw
18071] sockfd_cache_put_long(372) 192.168.240.133:7000 idx 0
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [rw
18071] fetch_object_list(999) 0
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [rw
18071] prepare_object_list(1084) 0
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
finish_recovery(777) recovery complete: new epoch 5
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
queue_request(486) COMPLETE_RECOVERY, 1
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
queue_cluster_request(316) COMPLETE_RECOVERY (0x3462d50)
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
cdrv_cpg_deliver(431) 3
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
sd_notify_handler(930) op COMPLETE_RECOVERY, size: 176, from: IPv4
ip:192.168.240.133 port:7000
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
cluster_recovery_completion(702) new epoch 5
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
cluster_recovery_completion(710) IPv4 ip:192.168.240.133 port:7000 is
recovered at epoch 5
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
cluster_recovery_completion(712) [0] IPv4 ip:192.168.240.133 port:7000
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]: NOTICE [main]
cluster_recovery_completion(726) all nodes are recovered, epoch 5
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
purge_work_done(397) purging work done, number of units: 0
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]: message
repeated 5 times: [  DEBUG [main] purge_work_done(397) purging work
done, number of units: 0]
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
cdrv_cpg_confchg(537) mem:1, joined:0, left:1
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  EMERG [main]
cdrv_cpg_confchg(550) PANIC: a number of leaving node (1) is larger
than majority (1), network partition
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  EMERG [main]
crash_handler(268) sheep exits unexpectedly (Aborted).
Jun 23 13:01:39 cn33.z1.mn2.simplecloud.ru sheep[18010]:  EMERG [main]
sd_backtrace(847) /usr/sbin/sheep() [0x406af7]
Jun 23 13:01:40 cn33.z1.mn2.simplecloud.ru sheep[18010]:  EMERG [main]
sd_backtrace(847) /lib/x86_64-linux-gnu/libpthread.so.0(+0xf09f)
[0x7f876635e09f]
Jun 23 13:01:40 cn33.z1.mn2.simplecloud.ru sheep[18010]:  EMERG [main]
sd_backtrace(847) /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x34)
[0x7f87654e8164]
Jun 23 13:01:40 cn33.z1.mn2.simplecloud.ru sheep[18010]:  EMERG [main]
sd_backtrace(847) /lib/x86_64-linux-gnu/libc.so.6(abort+0x17f)
[0x7f87654eb3df]
Jun 23 13:01:40 cn33.z1.mn2.simplecloud.ru sheep[18010]:  EMERG [main]
sd_backtrace(847) /usr/sbin/sheep() [0x429930]
Jun 23 13:01:40 cn33.z1.mn2.simplecloud.ru sheep[18010]:  EMERG [main]
sd_backtrace(847) /usr/sbin/sheep() [0x43e960]
Jun 23 13:01:40 cn33.z1.mn2.simplecloud.ru sheep[18010]:  EMERG [main]
sd_backtrace(847) /usr/sbin/sheep() [0x428a19]
Jun 23 13:01:40 cn33.z1.mn2.simplecloud.ru sheep[18010]:  EMERG [main]
sd_backtrace(847) /usr/sbin/sheep() [0x4303e8]
Jun 23 13:01:40 cn33.z1.mn2.simplecloud.ru sheep[18010]:  EMERG [main]
sd_backtrace(847) /usr/sbin/sheep() [0x4061b4]
Jun 23 13:01:40 cn33.z1.mn2.simplecloud.ru sheep[18010]:  EMERG [main]
sd_backtrace(847)
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfc)
[0x7f87654d4eac]
Jun 23 13:01:40 cn33.z1.mn2.simplecloud.ru sheep[18010]:  EMERG [main]
sd_backtrace(847) /usr/sbin/sheep() [0x406928]
Jun 23 13:01:40 cn33.z1.mn2.simplecloud.ru sheep[18010]:  DEBUG [main]
gdb_cmd(767) cannot find gdb

-- 
Vasiliy Tolstov,
e-mail: v.tolstov at selfip.ru