[Stgt-devel] Tuning iSER for performance

[Pete Wyckoff]

erezz at Voltaire.COM wrote on Wed, 05 Mar 2008 16:08 +0200:
> Pete Wyckoff wrote:
> > erezz at Voltaire.COM wrote on Sun, 02 Mar 2008 15:17 +0200:
> >> 27920 tgtd: [16:40:38.907895] iscsi_scsi_cmd_tx_done(1707) more data or
> >> sense or bidir b
> >> 27921 tgtd: [16:40:38.908269] iser_rx_progress(1101) entry
> >
> > This is rather slow in here.  It does the RDMA write, gets the local
> > completion that the write went out, then sends the command response
> > packet.  Does it make sense that it takes 374 us for the adapter to
> > say that it finished the write?
> >   
> 
> Where did you get the 374 us from?

Subtracting the times on the above two lines.  Just an idea for
where you might go to look for performance.

> > Another comment:  there is 10 us between consecutive printfs in this
> > trace.  You're probably not using TSC for GTOD.  It's not a big
> > deal, but be aware that your "with debugging" runs may be
> > significantly slower than without.
> 
> That's for sure. However, I tried to run without debugging, and
> host_busy was <= 1.
> 
> >   Just watch to make sure that the
> > profiling output makes sense.  We hack the kernel in a bad way to
> > get around this on our machines.
> > http://git.osc.edu/?p=linux.git;a=commitdiff;h=e7ff54222523a2ae6816c9ff17ad5a8fa86cd5e1
> >   
> 
> What does this patch do? I'm not familiar with that.

Just ignore it.  Hardware-specific way to force the use of TSC.

> According to this patch, bs_rdwr_request will be called from the context
> of iser_rx_progress (sounds good). Why doesn't the current stgt code do
> that? Why do we need the extra context switch?

That's the only way to get multi-operation concurrency on the
target given existing linux userspace AIO infrastructure:  run
multiple IOs at the same time, in separate threads.

The patch I sent you is just so you can debug, to answer the
question:  Is the context switch the source of your performance
problem on a _single_ request.

		-- Pete