On Mon, Sep 03, 2007 at 01:02:23PM -0400, Pete Wyckoff wrote: >robin.humble+stgt at anu.edu.au wrote on Mon, 03 Sep 2007 09:23 -0400: >> tgtd: iscsi_iser_write_begin(1014) Unable to find send slot >> [iSER]:in iscsi_iser_write_begin nbytes 48. >The problem was two-fold: not provisioning enough resources based >on negotiated connection settings, and not having a flow control >mechanism to slow down the TX state machine when RDMA ran out of >resources. ok - great. I'll be happy to try it out. >Open-iscsi has no way to specify the RDMA parameters, >IntiatorRecvDataSegmentLength or TargetRDSL, that size the control >transfer sizes. You can set them on the target similarly to how you >set MRDSL above, but the initiator will insist on 8k TRDSL. I'll >try to generate patches for that some day. Data sizes are >(recently) chosen by the target unilaterally. 8k's pretty small. to get ~90% of max bandwidth out of single IB messages (at least via verbs) something north of 50-100kB seems to be required(*). presumably you have multiple messages in flight though? if so then maybe a total RDMA buffer of order this size would be enough. I guess we'll find out :) cheers, robin (*) http://www.cita.utoronto.ca/mediawiki/index.php/Image:Netpipe3.png |
