[sheepdog] [RFC PATCH] sdnet: threading tx/rx process

[Liu Yuan]

On 07/13/2012 05:10 PM, Christoph Hellwig wrote:
> On Thu, Jul 12, 2012 at 03:20:29PM +0800, Liu Yuan wrote:
>> On 07/12/2012 02:18 PM, HaiTing Yao wrote:
>>> 1,  Can not guarantee I/O sequence of VDI. Many tx/rx threads, many io
>>> threds. If two requests r/w the same sector of same VDI, data will be
>>> corrupted.
>>>
>>
>> I think we will never get two requests R/W the same sector, because VM's
>> kernel should(might) guarantee this, at least for linux.
> 
> At least Windows does not, we had a problem in qemu with this before,
> note that even in this case there is no ordering guarantee so I don't
> think it actually is a problem.
> 
>>> 2,  I did not test the performance of this patch, but I think it will
>>> not be better than before. The time for creating/destroing threads may
>>> be longer than the time for data translation.
>>
>> Yes, short threads for this tx/rx would be the performance bottleneck. I
>> benchmark the write performance with my patch, seems that no noticeable
>> boost.
> 
> I really don't like either implementation.  For one I'd really love to
> figure out the root cause, that is where we spend all that time doing
> epoll driven non-blocking I/O.  Second if we go for a more threaded
> network I/O model I'd like to design it based on possible contention
> points, and avoiding creating threads all the time, which leads to a
> per-connection thread model where listen_handler creates a RX and a TX
> thread per connection which then uses EPOLL to handle all I/O on it.
> 

The real benefit is, as I described before, that requests from different
sources (be it from peer recovery req or io req, or gateway req) can
overlap each other, I think this will improve the scalability and will
work out better performance. Well, I think I should test the patch with
several real boxes than on one box, because one disk shared by several
sheep might hide the real benefit.

I don't think per-connection thread model will work out. The big problem
is that, we will use too many threads for one sheep:

   nr_threads = nr_node * 8 + nr_vm

For e.g, for 1k node setup, we'll have 8000+ threads at most!  This will
be definitely a problem.

Thanks,
Yuan