[sheepdog] [RFC PATCH] sdnet: threading tx/rx process

[Christoph Hellwig]

On Mon, Jul 16, 2012 at 02:36:18PM +0800, Liu Yuan wrote:
> Oh, I thought you meant following:
> 
>   main thread		connection thread1 ... connection threadX
> 
>   epoll(listenfd)
>   accept
> 			epoll(iofd1)		epoll(iofdX)
> 			rx			rx
> 			read/write		read/write
> 			tx			tx
> 			epoll(iofd1)		epoll(iofdX)

While that seems nice I don't think it will give you good performance
because it will only process one outstanding I/O per connection.

This scheme could actually work very well if we use aio for disk I/O,
except that file creation and preallocation still wouldn't be
asynchronous.

> Well, so in your proposal, per connection we have 2
> threads(rx/tx),right?

In the original one, yes.  The other option is to use a fixed number
of RX/TX threads and just hash the connections onto them.  Given that
the RX/TX threads only do non-blocking I/O in theory one per CPU should
actually be enough.

> Still too many connections. Yes, we can avoid
> context switches, but I suspect that context switches doesn't count much
> because we'll do a disk IO in a perspective of one request. To
> conclusion, I think we can afford 2 more context switches. (We can
> optimize this by redesign thread implementation later)

My other problem with excessive workqueue use is that it makes debugging
code flows pretty hard.