[Stgt-devel] dd fails with iSER

[Erez Zilber]

Pete Wyckoff wrote:

> pw at osc.edu wrote on Thu, 09 Aug 2007 16:22 -0400:
>   
>> There is also a spot in /etc/iscsi/iscsid.conf where you can adjust
>> the default value for the initiator:
>>
>> 	node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144
>>
>> Its default is a more reasonable 128k.
>>     
>
> This advice is correct, but for the wrong reasons.  The situation is
> more complex than I had thought.  The iSER spec doesn't provide
> guidance on how to set the maximum data segment lentgh for RDMA
> transfers.  It adds two new parameters:
>
>    InitiatorRecvDataSegmentLength - size of data segment in control PDUs
>    	sent to initiator
>    TargetRecvDataSegmentLength - size of data segment in control PDUs
>    	sent to target
>
> and tells us to ignore MaxRecvDataSegmentLength.  But it doesn't say
> how we should figure out the limit for data-type PDUs, i.e. for RDMA
> transfers, or even if there should be one.  The phrase "control
> PDUs" means non-RDMA transfers.
>
> One approach would be to have the target RDMA the entire data
> segment in a single operation.  This approach minimizes the
> overhead, but doesn't let us pipeline and may not be possible for
> large transfer sizes.  The OS won't let us pin all the memory
> required for the transfer, perhaps.
>
> Instead I've added another patch that changes the MaxRDSL in the
> target to be whatever was negotiated for IRDSL.

See what the spec says about this:

8.2 MaxRecvDataSegmentLength

[...]
Similarly, the target MUST consider the value of its local
MaxRecvDataSegmentLength (that it would have declared to the initiator)
as having the value of TargetRecvDataSegmentLength, and the value of the
remote MaxRecvDataSegmentLength (that would have been declared by the
initiator) as having the value of InitiatorRecvDataSegmentLength.

Therefore, setting MaxRDSL = IRDSL seems to be against the spec.
Instead, you should do: MaxRDSL = TRDSL

>   Since I see no way
> in the spec how the target could send data in a control type PDU,
> IRDSL wasn't doing anything for us anyway.
As Alex mentioned, the target does send control PDUs back to the initiator.
>   And open-iscsi uses its
> conn->max_recv_dlength as the starting point for IRDSL, which seems
> reasonable.
>   

The initiator follows the spec:

    } else if (iscsi_find_key_value("InitiatorRecvDataSegmentLength", text,
                    end, &value, &value_end)) {
        if (session->t->template->rdma) {
            conn->max_recv_dlength = MIN(conn->max_recv_dlength,
                             strtoul(value, NULL, 0));
        }
        text = value_end;
    } else if (iscsi_find_key_value("TargetRecvDataSegmentLength", text,
                    end, &value, &value_end)) {
        if (session->t->template->rdma) {
            conn->max_xmit_dlength = MIN(conn->max_xmit_dlength,
                             strtoul(value, NULL, 0));
        }
        text = value_end;

It sets InitiatorRecvDataSegmentLength to the minimum value that was
proposed by both sides. It sets conn->max_recv_dlength to the value that
was negotiated for InitiatorRecvDataSegmentLength (see 8.5 in the spec:
"It (InitiatorRecvDataSegmentLength) is used by the initiator and the
target to negotiate the maximum size of the data segment that a
target may send to the initiator in an iSCSI control-type PDU.")

The same thing happens with TargetRecvDataSegmentLength.

So, I suggest that we set MaxRDSL according to 8.2 in the iSER spec.

As for the default values that the target will propose - this is another
thing to consider, and is not covered by the spec (as Dan & Alex mentioned).

Erez