[Stgt-devel] iSER

[Pete Wyckoff]

robin.humble+stgt at anu.edu.au wrote on Mon, 03 Sep 2007 23:24 -0400:
> On Mon, Sep 03, 2007 at 01:02:23PM -0400, Pete Wyckoff wrote:
> >Open-iscsi has no way to specify the RDMA parameters,
> >IntiatorRecvDataSegmentLength or TargetRDSL, that size the control
> >transfer sizes.  You can set them on the target similarly to how you
> >set MRDSL above, but the initiator will insist on 8k TRDSL.  I'll
> >try to generate patches for that some day.  Data sizes are
> >(recently) chosen by the target unilaterally.
> 
> 8k's pretty small. to get ~90% of max bandwidth out of single IB
> messages (at least via verbs) something north of 50-100kB seems to be
> required(*).
> presumably you have multiple messages in flight though? if so then
> maybe a total RDMA buffer of order this size would be enough. I guess
> we'll find out :)

Yeah, small.  The [it]rdsl numbers affect only _control_ type
packets (as I was informed in an earlier thread here).  But these do
include the immediate data and unsolicited data packets, if you have
configured things as such.  For RDMA read/write, it is 512k now, but
see the #define in iscsi_rdma.c if you want to play with it.

I haven't done much testing of throughput vs packet size lately, and
in fact, ripping out the unsignaled completions may have slowed
things down some.  But they were a mess as far as tracking RDMA
resources went.  Would love some help with tuning if you are so
inclined.

		-- Pete