On Sun, 28 Sep 2008 12:38:35 +0300 Alexander Nezhinsky <nezhinsky at gmail.com> wrote: > This patch introduces a few interconnected improvements, that > ultimately lead to significant reduction in interrupts rate for > iser/ib, while adding flexibility to tgtd event processing scheme. > > First, it implements a kind of "scheduler" list, to replace the > counter events. Event handlers can schedule other events that are > placed on the scheduler's loop. This has some small advantages in > itself, because the same event descriptor is used for all events > in the system, events are executed in order they'd been scheduled, > they can be removed from list, few instances of the same event > may be scheduled. > > But the most important change is the logic of the processing loop, > as a whole. It goes as follows: > > The scheduler checks events on the list, does their processing, > which can schedule more items, but does not process the new items > (in order to avoid infinite/long loops). > > If after processing all "old" events, some "new" ones were scheduled, > epoll_wait() is executed in non-blocking manner. This guarantees > that the file descriptors that became ready during the scheduler > processing are handled, but if there no ready fds, the remaining > scheduler events are processed immediately. > > But if after processing the scheduler list nothing new is added, > then epoll_wait is blocking as in the current scheme. > > This way we can be very flexible, because event handlers and deferred > work can not block one another. Potentially long handlers can be > split into shorter ones easily without blocking the entire target. > > Finally, IB completion queue is the first guy who benefits, because > we can postpone interrupt re-arming, until no completion entries > remain, and to schedule CQ "draining" after all events are serviced. > > When a completion event is handled, CQ is polled so that up to a > given number (now set to 8) of WCs are processed. > > If more completions are left on the CQ, essentially the same > handler is scheduled, to carry out the next round of polling, > but in the meanwhile, other events in the system can be serviced. > > If CQ is found empty, interrupts are re-armed and a handler is > scheduled to consume the completions which could sneak in > between the moment the CQ was seen empty and just before > the interrupts were re-armed. > > Thus in iSER IB there is a marked interrupts rate reduction. > Here are typical results: > Current target: 62-65 KIOPS, ~110,000 interrupt/sec, CPU% ~68% > Patched target: 65-70 KIOPS, ~65,000 interrupt/sec, CPU% ~65% > > Signed-off-by: Alexander Nezhinsky <nezhinsky at gmail.com> Looks great performance improvement. Applied, thanks a lot! -- To unsubscribe from this list: send the line "unsubscribe stgt" in the body of a message to majordomo at vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html |