[sheepdog] [PATCH 1/3] sheep: queue local gateway request instead of directly call forward_*_obj_req

[levin li]

On 06/25/2012 10:14 PM, Liu Yuan wrote:
> On 06/25/2012 08:45 PM, levin li wrote:
>> On 06/25/2012 07:21 PM, Christoph Hellwig wrote:
>>> On Mon, Jun 25, 2012 at 07:17:07PM +0800, Liu Yuan wrote:
>>>> On 06/25/2012 07:10 PM, Christoph Hellwig wrote:
>>>>> How do you ensure this is always called from the main thread?  Calling
>>>>> queue_request from helper threads seems like it would cause major
>>>>> problems.
>>>>
>>>> What kind of problems?
>>>>
>>>> For now queue_request() is already called in worker threads context.
>>>> Looks to me we should add a thread safe version of queue_request() that
>>>> can be called in worker threads.
>>>
>>>  - get_vnode_info is expected to be called from the main thread,
>>>  - req_done is expected to be called freom the main thread
>>>  - and most serious one is request_in_recovery, which manipulates
>>>    sys->wait_rw_queue and sys->wait_obj_queue without taking locks.
>>>
>>
>> Ah, it's really a problem to call queue_request() from worker thread, and
>> now exec_local_req() is always called from worker thread.
>>
>> Sending a gateway request through network as my previous patch does seems
>> the simplest way to avoid this problem, but a connection to local node is
>> unnecessary.
>>
>> How about this, we put the work into a worker thread using queue_work()
>> and make queue_request() called in the worker_done() function.
>>
> 
> Maybe we can take advantage of sockfd cache, we can use lightweight unix
> domain FD for self-queuing requests. This require smallest code change
> and also doesn't compromise performance as bad as TCP sockfd.
> 
> Thanks,
> Yuan
> 

After further thinking, I think even we use a cached fd, it still not so efficient,
write/read_object() needs transfer about 4M data if we send a gateway request,
which I think is inefficient.

thanks,

levin