[Stgt-devel] dd fails with iSER

[Dan Bar Dov]

On 8/11/07, Pete Wyckoff <pw at osc.edu> wrote:
>
> pw at osc.edu wrote on Thu, 09 Aug 2007 16:22 -0400:
> > There is also a spot in /etc/iscsi/iscsid.conf where you can adjust
> > the default value for the initiator:
> >
> >       node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144
> >
> > Its default is a more reasonable 128k.
>
> This advice is correct, but for the wrong reasons.  The situation is
> more complex than I had thought.  The iSER spec doesn't provide
> guidance on how to set the maximum data segment lentgh for RDMA
> transfers.  It adds two new parameters:
>
>    InitiatorRecvDataSegmentLength - size of data segment in control PDUs
>         sent to initiator
>    TargetRecvDataSegmentLength - size of data segment in control PDUs
>         sent to target
>
> and tells us to ignore MaxRecvDataSegmentLength.  But it doesn't say
> how we should figure out the limit for data-type PDUs, i.e. for RDMA
> transfers, or even if there should be one.  The phrase "control
> PDUs" means non-RDMA transfers.


There are no "data-type" PDUs in ISER, that's why no limit for them is
mentioned. Control type PDUs can carry unsolicited data, but that is true
only for write ops. As to RDMA ops, the initiator communicates the size of
transfer and registration key, allowing the target to do the transfer, be it
read or write, in on or several RDMA ops, as many as it likes. The initiator
controls the overall using the info in the control PDU plus the ISER header.
The separation of MRDSL to IRDSL and TRDSL is necessary since in RDMA each
side needs to prepare a post-recv buffer, and each side communicates to the
other side what its capabilities are.

Dan

One approach would be to have the target RDMA the entire data
> segment in a single operation.  This approach minimizes the
> overhead, but doesn't let us pipeline and may not be possible for
> large transfer sizes.  The OS won't let us pin all the memory
> required for the transfer, perhaps.
>
> Instead I've added another patch that changes the MaxRDSL in the
> target to be whatever was negotiated for IRDSL.  Since I see no way
> in the spec how the target could send data in a control type PDU,
> IRDSL wasn't doing anything for us anyway.  And open-iscsi uses its
> conn->max_recv_dlength as the starting point for IRDSL, which seems
> reasonable.
>
> Incidentally, Erez, it was you who added IRDSL support to open-iscsi
> back in Oct 2005.  Apparently you agree with my interpretation?
>
> So the final story is:
>
>     - you can leave the previous patch "iser more rdma bufs" applied
>         as it is harmless and probably a good idea anyway
>
>     - apply the patch below for completeness
>
>     - make sure you have the quoted line above in your iscsid.conf
>
> You should see no change in behavior, but now the code is at least
> rational, I think.  (git.osc.edu is up to date with this patch too.)
>
>                 -- Pete
>
>
> From e1881832761741fcc8b0baf8de3e02d5fc2b8739 Mon Sep 17 00:00:00 2001
> From: Pete Wyckoff <pw at osc.edu>
> Date: Sat, 11 Aug 2007 10:55:50 -0400
> Subject: [PATCH] iser fix initiator dlength
>
> Set the iscsi param max_xmit_dlength from the initiator_rdsl param in
> the case of RDMA.  Otherwise the data_rsp_build and r2t_build use the
> wrong size in RDMA mode.
>
> Signed-off-by: Pete Wyckoff <pw at osc.edu>
> ---
> usr/iscsi/iscsid.c |    8 ++++++++
> 1 files changed, 8 insertions(+), 0 deletions(-)
>
> diff --git a/usr/iscsi/iscsid.c b/usr/iscsi/iscsid.c
> index 15c3623..a3fa6af 100644
> --- a/usr/iscsi/iscsid.c
> +++ b/usr/iscsi/iscsid.c
> @@ -517,6 +517,14 @@ static void login_finish(struct iscsi_connection
> *conn)
>                                 break;
>                         }
>                 }
> +               /*
> +                * data_rsp_build and r2t_build use this parameter
> +                * to size packets going to the initiator.  Set it
> +                * here rather than "if rdma" in those functions.
> +                */
> +               if (conn->tp->rdma)
> +
> conn->session_param[ISCSI_PARAM_MAX_XMIT_DLENGTH].val =
> +
> conn->session_param[ISCSI_PARAM_INITIATOR_RDSL].val;
>                 memcpy(conn->isid, conn->session->isid,
> sizeof(conn->isid));
>                 conn->tsih = conn->session->tsih;
>
> --
> 1.5.2.4
>
> _______________________________________________
> Stgt-devel mailing list
> Stgt-devel at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/stgt-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.berlios.de/pipermail/stgt-devel/attachments/20070811/9fe43ea7/attachment.html