[Sheepdog] [PATCH v2 1/2] force some callbacks of cluster request running in worker thread

[Liu Yuan]

On 04/29/2012 06:45 PM, Christoph Hellwig wrote:

> On Sat, Apr 28, 2012 at 03:33:53PM +0800, Liu Yuan wrote:
>> I think its time to retrofit process_main(). it doesn't necessarily run
>> in main thread. Actually, we should try to put it in the event queue as
>> possible as we can, to offload the main thread.
>>
>> I'd suggest a new name scheme:
>> process_work() -> process_top(), which runs only local node
>> process_main() -> process_bottom(), which runs in all nodes, either in
>> worker or in main thread.
> 
> How would we decide when to run it in the main or worker thread?
> 


Unfortunately, there is no specific rule. At least for now, there is
only some general practice:
1) push jobs as much as possible to worker threads
2) don't use lock in main thread (without a good reason)

> I'm getting more an more scared about the threading model - there's a
> lot of assumption that certain pieces of code only run in the main
> thread, or in the event_queue and thus can avoid locking, but there's no
> good way to veryify that this actually is the case.
> 


Ditto

> I think before this gets even more complicated an tid based assertation
> mechanism ala:
> 
>    main_tid = gettid();
> 
> static inline bool assert_mainthread(void)
> {
> 	assert(gettid() == main_tid);
> }
> 
> plus variants for other "optimized" queues needs to be added over all
> places that have this assumption.
> 


This looks good to me, but I think it would be useful only when we have
abstract out shared code in the main and worker thread as function helpers.

But I think this is hard to achieve, so for now we simply leave this
responsibility and freedom to programmers, mostly for implementing
lcoal&cluster operations.

> As for the naming of the execution function, why not name them
> descriptively as what they do:
> 
>   process_on_each_node()
> 
> and
> 
>   process_local()
> 
> or similar.
> 

It is not that easy. In a bigger picture, we have local and cluster
requests, which actually implement process_local() and
process_on_each_node() semantics.

I deliberately to suggest use top and bottom to differentiate it request
scope. top() and bottom() thus are only internal logic structure for
handling requests.

Thanks,
Yuan