[Stgt-devel] dd fails with iSER

[Pete Wyckoff]

pw at osc.edu wrote on Thu, 09 Aug 2007 16:22 -0400:
> There is also a spot in /etc/iscsi/iscsid.conf where you can adjust
> the default value for the initiator:
> 
> 	node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144
> 
> Its default is a more reasonable 128k.

This advice is correct, but for the wrong reasons.  The situation is
more complex than I had thought.  The iSER spec doesn't provide
guidance on how to set the maximum data segment lentgh for RDMA
transfers.  It adds two new parameters:

   InitiatorRecvDataSegmentLength - size of data segment in control PDUs
   	sent to initiator
   TargetRecvDataSegmentLength - size of data segment in control PDUs
   	sent to target

and tells us to ignore MaxRecvDataSegmentLength.  But it doesn't say
how we should figure out the limit for data-type PDUs, i.e. for RDMA
transfers, or even if there should be one.  The phrase "control
PDUs" means non-RDMA transfers.

One approach would be to have the target RDMA the entire data
segment in a single operation.  This approach minimizes the
overhead, but doesn't let us pipeline and may not be possible for
large transfer sizes.  The OS won't let us pin all the memory
required for the transfer, perhaps.

Instead I've added another patch that changes the MaxRDSL in the
target to be whatever was negotiated for IRDSL.  Since I see no way
in the spec how the target could send data in a control type PDU,
IRDSL wasn't doing anything for us anyway.  And open-iscsi uses its
conn->max_recv_dlength as the starting point for IRDSL, which seems
reasonable.

Incidentally, Erez, it was you who added IRDSL support to open-iscsi
back in Oct 2005.  Apparently you agree with my interpretation?

So the final story is:

    - you can leave the previous patch "iser more rdma bufs" applied
	as it is harmless and probably a good idea anyway

    - apply the patch below for completeness

    - make sure you have the quoted line above in your iscsid.conf

You should see no change in behavior, but now the code is at least
rational, I think.  (git.osc.edu is up to date with this patch too.)

		-- Pete