[sheepdog] [PATCH 1/2] Revert "corosync: handle network partition"
Hitoshi Mitake
mitake.hitoshi at lab.ntt.co.jp
Wed Oct 1 03:29:46 CEST 2014
At Tue, 30 Sep 2014 18:29:33 +0900,
tuji wrote:
>
> > Tsuji-san, if the change seems to be good for you, could you stamp
> > your Tested-by: tag?
> >
>
> OK. I add the tag.
>
> Reported-by: Masahiro Tsuji <tuji at atworks.co.jp>
> Cc: Masahiro Tsuji <tuji at atworks.co.jp>
> Tested-by: Masahiro Tsuji <tuji at atworks.co.jp>
> Signed-off-by: Hitoshi Mitake <mitake.hitoshi at lab.ntt.co.jp>
Thanks for your checking, applied this series.
Thanks,
Hitoshi
>
>
>
> > At Tue, 30 Sep 2014 17:01:25 +0900,
> > Hitoshi Mitake wrote:
> > >
> > > This reverts commit 15df161958a38cf3f7bc83b5bc2c8a1817b3072e.
> > >
> > > The commit introduced a better handling method of network
> > > partition. But the method seems to have a possibility of losing event
> > > from corosync. According to an experiment by Tsuji-san, dispatching
> > > of leave event can be delayed because of the poll(2) in
> > > __corosync_dispatch() with corosync < 2.x.
> > >
> > > In the next commit, simpler method for detecting network partition
> > > based on a parameter of cpg callback function will be implemented. The
> > > reverted method based on poll(2) depends on semantics of fd obtained
> > > by cpg_fd_get(3) of corosync. The poll(2) based method worked well on
> > > corosync 1.x, but it doesn't work well on 2.x. Internal big change
> > > of corosync (maybe employing libqb) seemed to change the semantics so
> > > we cannot use poll(2) for detecting queued messages.
> > >
> > > With this change, sheepdog cannot use corosync 1.x as its cluster
> > > manager any more. But the version has a critical bug in its message
> > > delivery mechanism which can produce broken epoch log. So it is
> > > reasonable to force users to stop using the version and introduce
> > > this change.
> > >
> > > Reported bug: https://bugs.launchpad.net/sheepdog-project/+bug/1368503
> > >
> > > Conflicts:
> > > sheep/cluster/corosync.c
> > > [ Conflicts were resolved by Hitoshi Mitake ]
> > >
> > > Reported-by: Masahiro Tsuji <tuji at atworks.co.jp>
> > > Cc: Masahiro Tsuji <tuji at atworks.co.jp>
> > > Signed-off-by: Hitoshi Mitake <mitake.hitoshi at lab.ntt.co.jp>
> > > ---
> > > sheep/cluster/corosync.c | 45 +++------------------------------------------
> > > 1 file changed, 3 insertions(+), 42 deletions(-)
> >
> > Tsuji-san, if the change seems to be good for you, could you stamp
> > your Tested-by: tag?
> >
> > Thanks,
> > Hitoshi
> >
> > >
> > > diff --git a/sheep/cluster/corosync.c b/sheep/cluster/corosync.c
> > > index cd00733..477e442 100644
> > > --- a/sheep/cluster/corosync.c
> > > +++ b/sheep/cluster/corosync.c
> > > @@ -10,7 +10,6 @@
> > > */
> > > #include <stdio.h>
> > > #include <unistd.h>
> > > -#include <poll.h>
> > > #include <sys/epoll.h>
> > > #include <corosync/cpg.h>
> > > #include <corosync/cfg.h>
> > > @@ -41,8 +40,6 @@ static struct cpg_node cpg_nodes[COROSYNC_MAX_NODES];
> > > static size_t nr_cpg_nodes;
> > > static bool self_elect;
> > > static bool join_finished;
> > > -static int cpg_fd;
> > > -static size_t nr_majority; /* used for network partition detection */
> > >
> > > /* event types which are dispatched in corosync_dispatch() */
> > > enum corosync_event_type {
> > > @@ -358,23 +355,6 @@ static bool update_join_status(struct corosync_event *cevent)
> > > static void __corosync_dispatch(void)
> > > {
> > > struct corosync_event *cevent;
> > > - struct pollfd pfd = {
> > > - .fd = cpg_fd,
> > > - .events = POLLIN,
> > > - };
> > > -
> > > - if (poll(&pfd, 1, 0)) {
> > > - /*
> > > - * Corosync dispatches leave events one by one even
> > > - * when network partition has occurred. To count the
> > > - * number of alive nodes correctly, we postpone
> > > - * processing events if there are incoming ones.
> > > - */
> > > - sd_debug("wait for a next dispatch event");
> > > - return;
> > > - }
> > > -
> > > - nr_majority = 0;
> > >
> > > while (!list_empty(&corosync_block_event_list) ||
> > > !list_empty(&corosync_nonblock_event_list)) {
> > > @@ -554,25 +534,6 @@ static void cdrv_cpg_confchg(cpg_handle_t handle,
> > > sd_debug("mem:%zu, joined:%zu, left:%zu", member_list_entries,
> > > joined_list_entries, left_list_entries);
> > >
> > > - /* check network partition */
> > > - if (left_list_entries) {
> > > - if (nr_majority == 0) {
> > > - size_t total = member_list_entries + left_list_entries;
> > > -
> > > - /*
> > > - * we need at least 3 nodes to handle network
> > > - * partition failure
> > > - */
> > > - if (total > 2)
> > > - nr_majority = total / 2 + 1;
> > > - }
> > > -
> > > - if (member_list_entries == 0)
> > > - panic("NIC failure?");
> > > - if (member_list_entries < nr_majority)
> > > - panic("Network partition is detected");
> > > - }
> > > -
> > > /* convert cpg_address to cpg_node */
> > > build_cpg_node_list(member_sheep, member_list, member_list_entries);
> > > build_cpg_node_list(left_sheep, left_list, left_list_entries);
> > > @@ -714,7 +675,7 @@ out:
> > >
> > > static int corosync_init(const char *option)
> > > {
> > > - int ret, retry_cnt = 0;
> > > + int ret, fd, retry_cnt = 0;
> > > uint32_t nodeid;
> > > cpg_callbacks_t cb = {
> > > .cpg_deliver_fn = cdrv_cpg_deliver,
> > > @@ -759,13 +720,13 @@ again:
> > > this_node.nodeid = nodeid;
> > > this_node.pid = getpid();
> > >
> > > - ret = cpg_fd_get(cpg_handle, &cpg_fd);
> > > + ret = cpg_fd_get(cpg_handle, &fd);
> > > if (ret != CS_OK) {
> > > sd_err("failed to get cpg file descriptor (%d)", ret);
> > > return -1;
> > > }
> > >
> > > - ret = register_event(cpg_fd, corosync_handler, NULL);
> > > + ret = register_event(fd, corosync_handler, NULL);
> > > if (ret) {
> > > sd_err("failed to register corosync event handler (%d)", ret);
> > > return -1;
> > > --
> > > 1.8.3.2
> > >
>
> **********************************************
> 株式会社エーティーワークス 辻 正博
> 〒106-6137
> 東京都港区六本木6-10-1 六本木ヒルズ森タワー37F
> TEL：03-3497-0505 FAX：03-3497-0508
> http://www.atworks.co.jp/
> E-mail: tuji at atworks.co.jp
> **********************************************
>
More information about the sheepdog
mailing list