[stgt] tgtd CPU 100% problem

Wed Jul 12 09:17:07 CEST 2017

I make the program with
```
make ISCSI_RDMA=1 DEBUG=1
```

a good tgtd iser_conn_list should be a good loop list like this
```
(gdb) p iser_conn_list
$1 = {next = 0x26a4aa0, prev = 0x2693190}
(gdb) p iser_conn_list->next
$2 = (struct list_head *) 0x26a4aa0
(gdb) p iser_conn_list->next->next
$3 = (struct list_head *) 0x2693190
(gdb) p iser_conn_list->next->next->next
$4 = (struct list_head *) 0x6638a0
(gdb) p iser_conn_list->next->next->next->next
$5 = (struct list_head *) 0x26a4aa0
```

but the error iser_conn_list is,
```
(gdb) p &iser_conn_list
$18 = (struct list_head *) 0x6638a0
(gdb) p iser_conn_list->next
$22 = (struct list_head *) 0x1065730
(gdb) p iser_conn_list->next->next
$23 = (struct list_head *) 0x1065730


(gdb) p conn->conn_list
$24 = {next = 0x1065730, prev = 0x6638a0}
(gdb) p conn->conn_list->next
$25 = (struct list_head *) 0x1065730
(gdb) p conn->conn_list->next->next
$26 = (struct list_head *) 0x1065730
```

the conn is struct iser_conn *:
```
(gdb) p conn
$14 = (struct iser_conn *) 0x1edc130
(gdb) p *conn
$15 = {h = {state = 13, closed = 0, rx_iostate = 0, tx_iostate = 0,
refcount = 0, clist = {next = 0x0, prev = 0x0}, session = 0x0, tid =
0, session_param = {
      {state = 0, val = 8192}, {state = 0, val = 1}, {state = 0, val =
1}, {state = 0, val = 1}, {state = 0, val = 1}, {state = 0, val = 1},
{state = 0,
        val = 65536}, {state = 0, val = 262144}, {state = 0, val = 1},
{state = 0, val = 1}, {state = 0, val = 0}, {state = 0, val = 0},
{state = 0,
        val = 0}, {state = 0, val = 2}, {state = 0, val = 20}, {state
= 0, val = 2048}, {state = 0, val = 2048}, {state = 0, val = 1},
{state = 0, val = 0},
      {state = 0, val = 8192}, {state = 0, val = 8192}, {state = 0,
val = 0}, {state = 0, val = 8192}, {state = 0, val = 128}}, initiator
= 0x0,
    initiator_alias = 0x0, isid = "\000\000\000\000\000", tsih = 0,
cid = 0, session_type = 0, auth_method = 0, stat_sn = 0, exp_stat_sn =
0, cmd_sn = 0,
    exp_cmd_sn = 0, max_cmd_sn = 0, req = {bhs = {opcode = 0 '\000',
flags = 0 '\000', rsvd2 = "\000", hlength = 0 '\000', dlength =
"\000\000",
        lun = "\000\000\000\000\000\000\000", itt = 0, ttt = 0, statsn
= 0, exp_statsn = 0, max_statsn = 0, other = '\000' <repeats 11
times>}, ahs = 0x0,
      ahssize = 0, data = 0x0, datasize = 0}, req_buffer = 0x0, rsp =
{bhs = {opcode = 0 '\000', flags = 0 '\000', rsvd2 = "\000", hlength =
0 '\000',
        dlength = "\000\000", lun = "\000\000\000\000\000\000\000",
itt = 0, ttt = 0, statsn = 0, exp_statsn = 0, max_statsn = 0,
        other = '\000' <repeats 11 times>}, ahs = 0x0, ahssize = 0,
data = 0x0, datasize = 0}, rsp_buffer = 0x0, rsp_buffer_size = 0,
rx_buffer = 0x0,
    tx_buffer = 0x0, rx_size = 0, tx_size = 0, ttt = 0, text_datasize
= 0, text_rsp_buffer = 0x0, rx_task = 0x0, tx_task = 0x0, tx_clist =
{next = 0x0,
      prev = 0x0}, task_list = {next = 0x0, prev = 0x0}, rx_digest =
"\000\000\000", tx_digest = "\000\000\000", auth_state = 0, auth =
{chap = {
        digest_alg = 0, id = 0, challenge_size = 0, challenge = 0x0}},
tp = 0x0, stats = {txdata_octets = 0, rxdata_octets = 0, noptx_pdus =
0,
      scsicmd_pdus = 0, tmfcmd_pdus = 0, login_pdus = 0, text_pdus =
0, dataout_pdus = 0, logout_pdus = 0, snack_pdus = 0, noprx_pdus = 0,
scsirsp_pdus = 0,
      tmfrsp_pdus = 0, textrsp_pdus = 0, datain_pdus = 0,
logoutrsp_pdus = 0, r2t_pdus = 0, async_pdus = 0, rjt_pdus = 0,
digest_err = 0, timeout_err = 0,
      custom_length = 0, custom = 0x1edc130}}, qp_hndl = 0x0, cm_id =
0x1edbee0, dev = 0x1eb2800, sched_buf_alloc = {{
      handler = 0x4216d9 <iser_sched_buf_alloc>, sched_handler =
0x4216d9 <iser_sched_buf_alloc>}, {fd = 0, scheduled = 0}, data =
0x1edc130, e_list = {
      next = 0x1edc430, prev = 0x1edc430}}, buf_alloc_list = {next =
0x1edc440, prev = 0x1edc440}, sched_rdma_rd = {{
      handler = 0x421909 <iser_sched_rdma_rd>, sched_handler =
0x421909 <iser_sched_rdma_rd>}, {fd = 0, scheduled = 0}, data =
0x1edc130, e_list = {
      next = 0x1edc468, prev = 0x1edc468}}, rdma_rd_list = {next =
0x1edc478, prev = 0x1edc478}, sched_iosubmit = {{
      handler = 0x4217f1 <iser_sched_iosubmit>, sched_handler =
0x4217f1 <iser_sched_iosubmit>}, {fd = 0, scheduled = 0}, data =
0x1edc130, e_list = {
      next = 0x1edc4a0, prev = 0x1edc4a0}}, iosubmit_list = {next =
0x1edc4b0, prev = 0x1edc4b0}, sched_tx = {{handler = 0x421a67
<iser_sched_tx>,
      sched_handler = 0x421a67 <iser_sched_tx>}, {fd = 0, scheduled =
0}, data = 0x1edc130, e_list = {next = 0x1edc4d8, prev = 0x1edc4d8}},
resp_tx_list = {
    next = 0x1edc4e8, prev = 0x1edc4e8}, sent_list = {next =
0x1edc4f8, prev = 0x1edc4f8}, sched_post_recv = {{handler = 0x421c31
<iser_sched_post_recv>,
      sched_handler = 0x421c31 <iser_sched_post_recv>}, {fd = 0,
scheduled = 0}, data = 0x1edc130, e_list = {next = 0x1edc520, prev =
0x1edc520}},
  post_recv_list = {next = 0x1edc530, prev = 0x1edc530},
sched_conn_free = {{handler = 0x41f8c3 <iser_sched_conn_free>,
      sched_handler = 0x41f8c3 <iser_sched_conn_free>}, {fd = 0,
scheduled = 0}, data = 0x1edc130, e_list = {next = 0x1edc558, prev =
0x1edc558}},
  peer_addr = {ss_family = 2, __ss_align = 0, __ss_padding = '\000'
<repeats 111 times>}, peer_name = 0x1ee0a00 "", self_addr = {ss_family
= 2,
    __ss_align = 0, __ss_padding = '\000' <repeats 111 times>},
self_name = 0x1ee0a20 "\360\t\356\001", ssize = 0, rsize = 0,
max_outst_pdu = 0,
  ff_res_alloc = 0, task_pool = 0x0, pdu_data_pool = 0x0, pdu_data_mr
= 0x0, nop_in_task = 0x0, text_tx_task = 0x0, login_phase =
LOGIN_PHASE_START,
  login_res_alloc = 0, login_task_pool = 0x0, login_data_pool = 0x0,
login_data_mr = 0x0, login_rx_task = 0x0, login_tx_task = 0x0,
conn_list = {
    next = 0x1edc6e0, prev = 0x6638a0}}
```

the conn->conn_list can not be equal to iser_conn_list, and will not
end the loop of “list_for_each_entry(conn, &iser_conn_list,
conn_list)”

2017-07-12 14:05 GMT+08:00 Sagi Grimberg <sagi at grimberg.me>:
>
>
> On 11/07/17 16:34, 李春 wrote:
>>
>> Sorry for the wrong CC'd lists.
>> Is CC'd lists correct this email?
>>
>> When tgtd received the DEVICE_REMOVAL event, we found that the tgtd
>> cost 100% of one CPU
>>
>> When we attach to the process with gdb, we found that process run a
>> infinite loop in usr/iscsi/iser.c:iser_nop_work_handler() to check if
>> the conn state is STATE_FULL.
>>
>> it is a infinite loop code in iser_nop_work_handler()：
>>
>> ```
>> list_for_each_entry(conn, &iser_conn_list, conn_list) {
>> ```
>>
>>   gdb show that:
>> ```
>> gdb attach 31629
>> ...
>> 3511                    if (conn->h.state != STATE_FULL)
>> (gdb) l
>> 3506    {
>> 3507            struct iser_conn *conn;
>> 3508            struct iser_task *task;
>> 3509
>> 3510            list_for_each_entry(conn, &iser_conn_list, conn_list) {
>> 3511                    if (conn->h.state != STATE_FULL)
>> 3512                            continue;
>> 3513                    task = conn->nop_in_task;
>> 3514                    if (!task)
>> 3515                            continue;
>>
>> (gdb) p iser_conn_list->next
>> $1 = (struct list_head *) 0x1edc6e0
>> (gdb) p iser_conn_list->next->next
>> $2 = (struct list_head *) 0x1edc6e0
>> (gdb) p iser_conn_list->next->next->next
>> $3 = (struct list_head *) 0x1edc6e0
>>
>> ```
>>
>> the iser_conn_list is a endless loop.
>
>
> What is &iser_conn_list address?
>
> You should run with debug mode and see if the list is
> empty or not


-- 
pickup.lichun 李春