[stgt] iSER over iWARP

Fri Oct 4 01:20:32 CEST 2013

Hi Or,

I experimented around a bit and here are my observations:

1. tgtd (Version 1.0.24) which comes with CentOS 6.4 distribution
doesn't have iSER support. Configured targets as iscsi/tcp. Initiator
is able to discover the target and login successfully.
2. Compiled latest code from git tree (Version 1.0.39) in iSCSI mode
and configured targets as iscsi/tcp. Initiator is still able to
discover targets and login successfully.
3. Compiled code from git tree (Version 1.0.39) for iSER support and
configured targets as iscsi/tcp. Initiator fails to discover targets.
Same is the case when the targets are configured as iser.
4. Once iSER compiled tgtd is executed, even the default tgtd fails to
execute and the system needs to be rebooted.

I enabled debug on both tgtd and iscsid/iscsiadm and this is what I noticed.
In the successful run case, this is what I see on tgtd when discovery happens

tgtd: accept_connection(99) 7
tgtd: cmnd_exec_login(723) Login request (operational negotiation): 0
tgtd: login_start(561) exp_cmd_sn: 1,16777216
tgtd: iscsi_tcp_event_handler(161) connection closed
tgtd: iscsi_tcp_event_handler(167) connection closed 0x28095c8
tgtd: conn_close(101) connection closed, 0x28095c8 1

I don't see any connection from the initiator in the failure case.
Attached is the tgtd trace.

I also ran tshark on both the systems and attached is the output for
successful/failed discovery cases.

On the initiator side, since the connection wasn't established with
the target, this is what happens
iscsiadm: no authentication configured...
iscsiadm: sendtargets discovery to 10.196.86.101:3260 using isid 0x00023d000000
iscsiadm: resolved 10.196.86.101 to 10.196.86.101
iscsiadm: discovery timeouts: login 15, reopen_cnt 6, auth 45.
iscsiadm: connecting to 10.196.86.101:3260
iscsiadm: connected local port 49410 to 10.196.86.101:3260
iscsiadm: connected to discovery address 10.196.86.101
iscsiadm: discovery session to 10.196.86.101:3260 starting iSCSI login
iscsiadm: sending login PDU with current stage 1, next stage 3,
transit 0x80, isid 0x00023d000000 exp_statsn 0
iscsiadm: >    InitiatorName=iqn.1994-05.com.redhat:b1e11cac64bc
iscsiadm: >    InitiatorAlias=lab189
iscsiadm: >    SessionType=Discovery
iscsiadm: >    HeaderDigest=None
iscsiadm: >    DataDigest=None
iscsiadm: >    DefaultTime2Wait=2
iscsiadm: >    DefaultTime2Retain=0
iscsiadm: >    IFMarker=No
iscsiadm: >    OFMarker=No
iscsiadm: >    ErrorRecoveryLevel=0
iscsiadm: >    MaxRecvDataSegmentLength=32768
iscsiadm: wrote 48 bytes of PDU header
iscsiadm: wrote 244 bytes of PDU data
iscsiadm: iscsi_login: Poll return 1

iscsiadm: Connection to Discovery Address 10.196.86.101 failed
iscsiadm: Login I/O error, failed to receive a PDU
iscsiadm: retrying discovery login to 10.196.86.101
iscsiadm: disconnecting conn 0x14b8de0, fd 3
iscsiadm: discovery session to 10.196.86.101:3260 sleeping for 1
seconds before next login attempt

The above findings suggest that it's either an issue with the setup or
tgtd compiled with iser support; and not the iscsid initiator.

Thanks,
Sushma

On Thu, Oct 3, 2013 at 12:04 AM, Or Gerlitz <ogerlitz at mellanox.com> wrote:
> On 03/10/2013 10:02, Sushma R wrote:
>>
>> Yes, that's the output. Entire output from beginning below.
>
> is it more verbose when you discover tcp targets? if not, take a look on the
> iscsi initiator man pages / documentation
> and see how to make it happen, we have the understand what's the problem
> from the viewpoint of iscsid
-------------- next part --------------
[root at lab112 usr]# ./tgtd -f -d 1
tgtd: iser_ib_init(3386) listening for iser connections on port 3260
tgtd: work_timer_start(146) use timer_fd based scheduler
tgtd: work_timer_start(181) started, timeout: 500 msec
tgtd: bs_init_signalfd(249) could not open backing-store module directory /usr/lib/tgt/backing-store
tgtd: bs_init(368) use signalfd notification
tgtd: mtask_alloc(586) mtask:0x1fd86a0
tgtd: mtask_execute(500) 120 1 1 2 -1 0 ffffffffffffffff (null) 2846
tgtd: mtask_free(592) mtask:0x1fd86a0
tgtd: mtask_alloc(586) mtask:0x1fd85f0
tgtd: mtask_execute(500) 147 1 1 0 3 0 ffffffffffffffff targetname=iqn.lab112.tgt3 2846
tgtd: tgt_device_create(488) 3 0
tgtd: tgt_device_create(677) Add a logical unit 0 to the target 3
tgtd: tgt_target_create(2180) Succeed to create a new target 3
tgtd: mtask_free(592) mtask:0x1fd85f0
tgtd: mtask_alloc(586) mtask:0x1fd85f0
tgtd: mtask_execute(500) 135 1 2 0 3 0 1 path=/dev/ram0 2846
tgtd: device_mgmt(245) sz:15 params:path=/dev/ram0
tgtd: tgt_device_create(488) 3 1
tgtd: bs_thread_open(390) 16
tgtd: bs_thread_worker_fn(204) started this thread
tgtd: bs_thread_worker_fn(204) started this thread
tgtd: bs_thread_worker_fn(204) started this thread
tgtd: bs_thread_worker_fn(204) started this thread
tgtd: bs_thread_worker_fn(204) started this thread
tgtd: bs_thread_worker_fn(204) started this thread
tgtd: bs_thread_worker_fn(204) started this thread
tgtd: bs_thread_worker_fn(204) started this thread
tgtd: tgt_device_create(677) Add a logical unit 1 to the target 3
tgtd: bs_thread_worker_fn(204) started this thread
tgtd: mtask_free(592) mtask:0x1fd85f0
tgtd: bs_thread_worker_fn(204) started this thread
tgtd: bs_thread_worker_fn(204) started this thread
tgtd: bs_thread_worker_fn(204) started this thread
tgtd: bs_thread_worker_fn(204) started this thread
tgtd: bs_thread_worker_fn(204) started this thread
tgtd: bs_thread_worker_fn(204) started this thread
tgtd: bs_thread_worker_fn(204) started this thread
tgtd: mtask_alloc(586) mtask:0x1fd85f0
tgtd: mtask_execute(500) 142 1 1 3 3 0 ffffffffffffffff initiator-address=ALL 2846
tgtd: mtask_free(592) mtask:0x1fd85f0
tgtd: mtask_alloc(586) mtask:0x1fd85f0
tgtd: mtask_execute(500) 120 1 1 2 -1 0 ffffffffffffffff (null) 2846
tgtd: mtask_free(592) mtask:0x1fd85f0
-------------- next part --------------
100.771097 10.196.86.16 -> 10.196.86.101 TCP 53656 > iscsi-target [SYN] Seq=0 Win=14600 Len=0 MSS=1460 TSV=706698 TSER=0 WS=7
100.771120 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53656 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 TSV=1061273 TSER=706698 WS=7
100.771230 10.196.86.16 -> 10.196.86.101 TCP 53656 > iscsi-target [ACK] Seq=1 Ack=1 Win=14720 Len=0 TSV=706698 TSER=1061273
100.771297 10.196.86.16 -> 10.196.86.101 TCP [TCP segment of a reassembled PDU]
100.771314 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53656 [ACK] Seq=1 Ack=49 Win=14592 Len=0 TSV=1061274 TSER=706698
100.771322 10.196.86.16 -> 10.196.86.101 iSCSI Login Command
100.771325 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53656 [ACK] Seq=1 Ack=293 Win=15616 Len=0 TSV=1061274 TSER=706698
100.771733 10.196.86.101 -> 10.196.86.16 iSCSI Login Response (Success)
100.771851 10.196.86.16 -> 10.196.86.101 TCP 53656 > iscsi-target [ACK] Seq=293 Ack=193 Win=15744 Len=0 TSV=706699 TSER=1061274
100.771932 10.196.86.16 -> 10.196.86.101 TCP [TCP segment of a reassembled PDU]
100.771945 10.196.86.16 -> 10.196.86.101 iSCSI Text Command
100.771955 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53656 [ACK] Seq=193 Ack=357 Win=15616 Len=0 TSV=1061274 TSER=706699
100.771983 10.196.86.101 -> 10.196.86.16 iSCSI Text Response
100.772745 10.196.86.16 -> 10.196.86.101 TCP 53656 > iscsi-target [FIN, ACK] Seq=357 Ack=305 Win=15744 Len=0 TSV=706700 TSER=1061274
100.772913 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53656 [FIN, ACK] Seq=305 Ack=358 Win=15616 Len=0 TSV=1061275 TSER=706700
100.772950 10.196.86.16 -> 10.196.86.101 TCP 53656 > iscsi-target [ACK] Seq=358 Ack=306 Win=15744 Len=0 TSV=706700 TSER=1061275
-------------- next part --------------
1542.231032 10.196.86.16 -> 10.196.86.101 TCP 53665 > iscsi-target [SYN] Seq=0 Win=14600 Len=0 MSS=1460 TSV=2186863 TSER=0 WS=7
1542.231122 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53665 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=3 TSV=1890671 TSER=2186863
1542.231146 10.196.86.16 -> 10.196.86.101 TCP 53665 > iscsi-target [ACK] Seq=1 Ack=1 Win=14720 Len=0 TSV=2186863 TSER=1890671
1542.231203 10.196.86.16 -> 10.196.86.101 TCP [TCP segment of a reassembled PDU]
1542.231217 10.196.86.16 -> 10.196.86.101 iSCSI Login Command
1542.231271 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53665 [ACK] Seq=1 Ack=49 Win=262096 Len=0 TSV=1890672 TSER=2186863
1542.231287 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53665 [ACK] Seq=1 Ack=293 Win=261848 Len=0 TSV=1890672 TSER=2186863
1542.231295 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53665 [RST, ACK] Seq=1 Ack=293 Win=0 Len=0
1543.231575 10.196.86.16 -> 10.196.86.101 TCP 53666 > iscsi-target [SYN] Seq=0 Win=14600 Len=0 MSS=1460 TSV=2187863 TSER=0 WS=7
1543.231670 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53666 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=3 TSV=1891339 TSER=2187863
1543.231688 10.196.86.16 -> 10.196.86.101 TCP 53666 > iscsi-target [ACK] Seq=1 Ack=1 Win=14720 Len=0 TSV=2187864 TSER=1891339
1543.231753 10.196.86.16 -> 10.196.86.101 TCP [TCP segment of a reassembled PDU]
1543.231767 10.196.86.16 -> 10.196.86.101 iSCSI Login Command
1543.231776 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53666 [ACK] Seq=1 Ack=49 Win=262096 Len=0 TSV=1891340 TSER=2187864
1543.231785 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53666 [ACK] Seq=1 Ack=293 Win=261848 Len=0 TSV=1891340 TSER=2187864
1543.231791 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53666 [RST, ACK] Seq=1 Ack=293 Win=0 Len=0
1544.232153 10.196.86.16 -> 10.196.86.101 TCP 53667 > iscsi-target [SYN] Seq=0 Win=14600 Len=0 MSS=1460 TSV=2188864 TSER=0 WS=7
1544.232193 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53667 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=3 TSV=1892007 TSER=2188864
1544.232207 10.196.86.16 -> 10.196.86.101 TCP 53667 > iscsi-target [ACK] Seq=1 Ack=1 Win=14720 Len=0 TSV=2188864 TSER=1892007
1544.233661 10.196.86.16 -> 10.196.86.101 TCP [TCP segment of a reassembled PDU]
1544.233673 10.196.86.16 -> 10.196.86.101 iSCSI Login Command
1544.233682 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53667 [ACK] Seq=1 Ack=49 Win=262096 Len=0 TSV=1892008 TSER=2188866
1544.233690 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53667 [ACK] Seq=1 Ack=293 Win=261848 Len=0 TSV=1892008 TSER=2188866
1544.233699 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53667 [RST, ACK] Seq=1 Ack=293 Win=0 Len=0
1545.233969 10.196.86.16 -> 10.196.86.101 TCP 53668 > iscsi-target [SYN] Seq=0 Win=14600 Len=0 MSS=1460 TSV=2189866 TSER=0 WS=7
1545.234065 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53668 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=3 TSV=1892676 TSER=2189866
1545.234083 10.196.86.16 -> 10.196.86.101 TCP 53668 > iscsi-target [ACK] Seq=1 Ack=1 Win=14720 Len=0 TSV=2189866 TSER=1892676
1545.234142 10.196.86.16 -> 10.196.86.101 TCP [TCP segment of a reassembled PDU]
1545.234156 10.196.86.16 -> 10.196.86.101 iSCSI Login Command
1545.234166 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53668 [ACK] Seq=1 Ack=49 Win=262096 Len=0 TSV=1892676 TSER=2189866
1545.234175 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53668 [ACK] Seq=1 Ack=293 Win=261848 Len=0 TSV=1892676 TSER=2189866
1545.234186 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53668 [RST, ACK] Seq=1 Ack=293 Win=0 Len=0
1546.234469 10.196.86.16 -> 10.196.86.101 TCP 53669 > iscsi-target [SYN] Seq=0 Win=14600 Len=0 MSS=1460 TSV=2190866 TSER=0 WS=7
1546.234515 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53669 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=3 TSV=1893344 TSER=2190866
1546.234528 10.196.86.16 -> 10.196.86.101 TCP 53669 > iscsi-target [ACK] Seq=1 Ack=1 Win=14720 Len=0 TSV=2190866 TSER=1893344
1546.235742 10.196.86.16 -> 10.196.86.101 TCP [TCP segment of a reassembled PDU]
1546.235757 10.196.86.16 -> 10.196.86.101 iSCSI Login Command
1546.235764 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53669 [ACK] Seq=1 Ack=49 Win=262096 Len=0 TSV=1893345 TSER=2190868
1546.235777 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53669 [ACK] Seq=1 Ack=293 Win=261848 Len=0 TSV=1893345 TSER=2190868
1546.235780 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53669 [RST, ACK] Seq=1 Ack=293 Win=0 Len=0
1547.236050 10.196.86.16 -> 10.196.86.101 TCP 53670 > iscsi-target [SYN] Seq=0 Win=14600 Len=0 MSS=1460 TSV=2191868 TSER=0 WS=7
1547.236140 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53670 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=3 TSV=1894013 TSER=2191868
1547.236157 10.196.86.16 -> 10.196.86.101 TCP 53670 > iscsi-target [ACK] Seq=1 Ack=1 Win=14720 Len=0 TSV=2191868 TSER=1894013
1547.236206 10.196.86.16 -> 10.196.86.101 TCP [TCP segment of a reassembled PDU]
1547.236217 10.196.86.16 -> 10.196.86.101 iSCSI Login Command
1547.236229 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53670 [ACK] Seq=1 Ack=49 Win=262096 Len=0 TSV=1894013 TSER=2191868
1547.236236 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53670 [ACK] Seq=1 Ack=293 Win=261848 Len=0 TSV=1894013 TSER=2191868
1547.236253 10.196.86.101 -> 10.196.86.16 TCP iscsi-target > 53670 [RST, ACK] Seq=1 Ack=293 Win=0 Len=0