[sheepdog] [PATCH 1/3] sheep: queue local gateway request instead of directly call forward_*_obj_req

[Liu Yuan]

On 06/25/2012 08:45 PM, levin li wrote:
> On 06/25/2012 07:21 PM, Christoph Hellwig wrote:
>> On Mon, Jun 25, 2012 at 07:17:07PM +0800, Liu Yuan wrote:
>>> On 06/25/2012 07:10 PM, Christoph Hellwig wrote:
>>>> How do you ensure this is always called from the main thread?  Calling
>>>> queue_request from helper threads seems like it would cause major
>>>> problems.
>>>
>>> What kind of problems?
>>>
>>> For now queue_request() is already called in worker threads context.
>>> Looks to me we should add a thread safe version of queue_request() that
>>> can be called in worker threads.
>>
>>  - get_vnode_info is expected to be called from the main thread,
>>  - req_done is expected to be called freom the main thread
>>  - and most serious one is request_in_recovery, which manipulates
>>    sys->wait_rw_queue and sys->wait_obj_queue without taking locks.
>>
> 
> Ah, it's really a problem to call queue_request() from worker thread, and
> now exec_local_req() is always called from worker thread.
> 
> Sending a gateway request through network as my previous patch does seems
> the simplest way to avoid this problem, but a connection to local node is
> unnecessary.
> 
> How about this, we put the work into a worker thread using queue_work()
> and make queue_request() called in the worker_done() function.
> 

Maybe we can take advantage of sockfd cache, we can use lightweight unix
domain FD for self-queuing requests. This require smallest code change
and also doesn't compromise performance as bad as TCP sockfd.

Thanks,
Yuan