[Sheepdog] [RFC PATCH] sheep: add client side timeout support for socket

[Liu Yuan]

On 11/27/2011 02:10 AM, MORITA Kazutaka wrote:

> At Sat, 26 Nov 2011 19:06:18 +0800,
> Liu Yuan wrote:
>>
>> On 11/26/2011 05:53 PM, Yibin Shen wrote:
>>
>>> oops, I found a regression with this patch
>>>
>>> Nov 26 11:19:13 store_queue_request(936) 3, 3, 412ca6000022d8 , 10
>>> Nov 26 11:19:13 forward_write_obj_req(368) 412ca6000022d8
>>> Nov 26 11:19:13 store_queue_request_local(843) 3, 412ca6000022d8 , 10
>>> Nov 26 11:19:43 store_queue_request(967) failed: 3, 3, 412ca6000022d8 , 10, 3
>>> Nov 26 11:19:43 io_op_done(147) leaving sheepdog cluster
>>> Nov 26 11:19:43 sd_leave_handler(1291) network partition bug: this
>>> sheep should have exited
>>> Nov 26 11:19:43 log_sigsegv(358) logger pid 9654 exiting abnormally
>>>
>>>
>>> e.g :  if a object have 3 copies,  and is hashed to (local, node A, node B)
>>> then in a write operation, if node A leave cluster,  IO towards node A
>>> will timeout after 30sec,
>>> but we use a strong consistency model, so return value of
>>> store_request_queue will be set to SD_RES_EIO,
>>> then io_op_done (sdnet.c) function will call leave_cluster .
>>>
>>> 144        } else if (is_access_local(req->entry, req->nr_vnodes,
>>> 145                                   ((struct sd_obj_req
>>> *)&req->rq)->oid, copies) &&
>>> 146                   req->rp.result == SD_RES_EIO) {
>>> 147                eprintf("leaving sheepdog cluster\n");
>>> 148                leave_cluster();
>>>
>>> IMO, maybe we should:
>>> 1)split store_request_queue() into multiple works.
>>> 2)replace strong consistency with eventual consistency or casual consistency。
>>>
>>> any comments?
>>>
>>> thanks
>>
>>
>> I think it is not the time to introduce other consistency models which
>> bring in much complexity.
>>
>> Whatever consistency model you use, you still need to handle EIO. IMO,
> 
> It is completely wrong to set SD_RES_EIO when timeout occurs because
> the error means disk I/O errors.  We must set SD_RES_NETWORK_ERROR in
> this case so that the request will be retried after epoch is updated.
> 
> But I guess it is better to enable TCP keepalive.  If we use it, the
> connection will be closed after timeout automatically, so we don't
> need to change network I/O code at all.
> 


Yes, I should have read your email before reply :). If sheepdog or other
facility provides kind of auto retry mechanism, that is definitely desired.

Thanks,
Yuan