[Sheepdog] [PATCH 0/5] stop grabbing vnode_info references outside the main thread

[Liu Yuan]

On 05/07/2012 10:52 PM, Christoph Hellwig wrote:

> There's two reasons why I don't like using rcu for vnode_info, and only
> half of it has anything to do with scary.
> 
>  1) RCU does not fit the problem.  What RCU excels at is lock free
>     lookups in multi-threaded lookup data structures.  We have a simple
>     pointer derference here, and even without my patchset we can never
>     grab the first references outside the main thread; it's just
>     non-obvious without it.  In fact that is a pretty fundamental
>     design point of non-blocking event based architectures like sheep -
>     anything touching central data structures is done quickly and
>     non-blocking in the main event loop thread, blocking work is handed
>     off to workers just to be completed by beeing reinjected into the
>     main thread.  If this design is carefull followed there is almost
>     no need for global data structures that are access for threads
>     outside the main thread.  In fact the core sheepdog code (minus
>     farm, object_cache and the accord cluster driver) never uses any
>     thread synchronization outside the workqueue code, and if followed
>     carefully and asserted that's a very powerful architecture.
>     In fact the only smart lock-less algorithm that would fit very
>     well into this architecture would be a lockless producer/consumer
>     queue for the workqueues.  Given that both sides modify the queue
>     that's not something RCU would help with at all.
>


I am still not convinced to run away from RCU-based mechanism or data
structure.

I am very interested in your comment about lock-less queue for
workqueues. For now, URCU provides such abstraction already! It actually
provides both lock-free and wait-free queues. So would you please
elaborate it a bit (for e.g, what we could benefit from it)?

Sharing nothing between workers and main thread is not easy, if not
practical. And at least for now, sheep core code like in group.c and
cluster driver doesn't follow the concept well. They dispatch and handle
the event mostly in the main thread.

So my point is, storage system is inherently multi-threaded application.
We can't avoid concurrency at all. For example, as for backend store, we
explicitly exclude the concurrent access to the same object ( badly
influence the performance). Thanks to lock mechanism(maybe better switch
to lock-less algo), object cache dosen't have this constraint. So I
think in the future when we become more concerned about performance, we
will introduce lock-less data structure for IO at least.

I also think it is time for us to examine the current thread model.

1) is efficient enough for IO?
2) should we retrofit it so that a) main thread won't get blocked for
cluster driver. b) main thread just dispatch the requests to the work
queues and offload all the work to the dedicated workers.

For the new patch set, it seems that 1) we can't avoid to share vnode
info between main and threads, and 2) get a difficult-to-use API:
grab/get/put 3) need to pass vnode_info everywhere and change internal
API a lot.

I am not strongly against the patch set per se, but if we can handle it
alternatively by introducing RCU and solve current vnode_info issue with
minimal change, that should be considered and it is still good for us to
make use of its lock-less data structure later.

Thanks,
Yuan

Thanks,
Yuan

 
>  2) Richt now we don't have the thread model in sheep spelled out,
>     documented and verified using assertations.  Until that happens
>     ading additional complexities to the concurrency model like RCU
>     does not seem like a good idea, if not a little scary.