2011/11/27 MORITA Kazutaka <morita.kazutaka at lab.ntt.co.jp>: > At Sat, 26 Nov 2011 19:06:18 +0800, > Liu Yuan wrote: >> >> On 11/26/2011 05:53 PM, Yibin Shen wrote: >> >> > oops, I found a regression with this patch >> > >> > Nov 26 11:19:13 store_queue_request(936) 3, 3, 412ca6000022d8 , 10 >> > Nov 26 11:19:13 forward_write_obj_req(368) 412ca6000022d8 >> > Nov 26 11:19:13 store_queue_request_local(843) 3, 412ca6000022d8 , 10 >> > Nov 26 11:19:43 store_queue_request(967) failed: 3, 3, 412ca6000022d8 , 10, 3 >> > Nov 26 11:19:43 io_op_done(147) leaving sheepdog cluster >> > Nov 26 11:19:43 sd_leave_handler(1291) network partition bug: this >> > sheep should have exited >> > Nov 26 11:19:43 log_sigsegv(358) logger pid 9654 exiting abnormally >> > >> > >> > e.g : if a object have 3 copies, and is hashed to (local, node A, node B) >> > then in a write operation, if node A leave cluster, IO towards node A >> > will timeout after 30sec, >> > but we use a strong consistency model, so return value of >> > store_request_queue will be set to SD_RES_EIO, >> > then io_op_done (sdnet.c) function will call leave_cluster . >> > >> > 144 } else if (is_access_local(req->entry, req->nr_vnodes, >> > 145 ((struct sd_obj_req >> > *)&req->rq)->oid, copies) && >> > 146 req->rp.result == SD_RES_EIO) { >> > 147 eprintf("leaving sheepdog cluster\n"); >> > 148 leave_cluster(); >> > >> > IMO, maybe we should: >> > 1)split store_request_queue() into multiple works. >> > 2)replace strong consistency with eventual consistency or casual consistency。 >> > >> > any comments? >> > >> > thanks >> >> >> I think it is not the time to introduce other consistency models which >> bring in much complexity. >> >> Whatever consistency model you use, you still need to handle EIO. IMO, > > It is completely wrong to set SD_RES_EIO when timeout occurs because > the error means disk I/O errors. We must set SD_RES_NETWORK_ERROR in > this case so that the request will be retried after epoch is updated. > Yes , it works. > But I guess it is better to enable TCP keepalive. If we use it, the > connection will be closed after timeout automatically, so we don't > need to change network I/O code at all. > hmm, I don't think so, to solve this problem , we must enable client side tcp keepalive, so we have to modify the network I/O code, also tcp keepalive will bring in overhead. IMO, use poll plus timeout setting is the simplest solution > Thanks, > > Kazutaka > >> you could handle EIO even with current strong model. In this case, A is >> gone, you could >> >> 1) timeout the write >> 2) wait for the cluster get recovered (get a new hash) >> 3) do the write again. >> >> The newest HEAD have already removed the lines that makes sheep >> panic-out in error case. So currently, EIO will leave the node a gateway >> for VMs. This is a acceptable compromise. >> >> Thanks, >> Yuan >> >> -- >> sheepdog mailing list >> sheepdog at lists.wpkg.org >> http://lists.wpkg.org/mailman/listinfo/sheepdog > -- > sheepdog mailing list > sheepdog at lists.wpkg.org > http://lists.wpkg.org/mailman/listinfo/sheepdog |