[stgt] tgtd 1.0.9 still exits with "iscsi_tx_handler error"
FUJITA Tomonori
fujita.tomonori at lab.ntt.co.jp
Thu Oct 14 11:43:41 CEST 2010
On Thu, 14 Oct 2010 17:45:04 +0900 (JST)
Hirokazu Takahashi <taka at valinux.co.jp> wrote:
> Hi,
>
> I got a core file from this crash.
> It looks like some command hash list was broken.
> I can upload it anywhere you like if you want.
Ok, TMF looks related too.
I'm at NY. When I'm back to Japan, I'll work on this.
Thanks,
>
> # gdb tgtd core..11528
> GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5_5.2)
> Copyright (C) 2009 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /home/taka/src/threads/usr/tgtd...done.
> [New Thread 11562]
> [New Thread 11561]
> [New Thread 11560]
> [New Thread 11559]
> [New Thread 11556]
> [New Thread 11555]
> [New Thread 11554]
> [New Thread 11553]
> [New Thread 11550]
> [New Thread 11549]
> [New Thread 11548]
> [New Thread 11547]
> [New Thread 11544]
> [New Thread 11543]
> [New Thread 11542]
> [New Thread 11541]
> [New Thread 11529]
> Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
> Loaded symbols for /lib64/libpthread.so.0
> Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
> Loaded symbols for /lib64/libc.so.6
> Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Core was generated by `tgtd -f'.
> Program terminated with signal 11, Segmentation fault.
> #0 0x0000000000412c37 in abort_task_set (mreq=0x1d51afd0, target=0x1a0db700,
> itn_id=675, tag=3758096422, lun=0x0, all=0) at target.c:1128
> 1128 list_for_each_entry_safe(cmd, tmp, list, c_hlist) {
> (gdb) bt
> #0 0x0000000000412c37 in abort_task_set (mreq=0x1d51afd0, target=0x1a0db700,
> itn_id=675, tag=3758096422, lun=0x0, all=0) at target.c:1128
> #1 0x0000000000412dfe in target_mgmt_request (tid=<value optimized out>,
> itn_id=675, req_id=489995968, function=<value optimized out>,
> lun_buf=<value optimized out>, tag=3758096422, host_no=0) at target.c:1175
> #2 0x0000000000404f5a in iscsi_tm_execute (task=0x3b6f352990)
> at iscsi/iscsid.c:1410
> #3 iscsi_task_execute (task=0x3b6f352990) at iscsi/iscsid.c:1455
> #4 0x0000000000405c32 in iscsi_task_queue (conn=0x1d56d708)
> at iscsi/iscsid.c:1532
> #5 iscsi_task_rx_done (conn=0x1d56d708) at iscsi/iscsid.c:1673
> #6 iscsi_rx_handler (conn=0x1d56d708) at iscsi/iscsid.c:2074
> #7 0x000000000040c268 in iscsi_tcp_event_handler (fd=<value optimized out>,
> events=1, data=0x3b6f352990) at iscsi/iscsi_tcp.c:158
> #8 0x000000000040ff1e in event_loop (argc=<value optimized out>,
> argv=<value optimized out>) at tgtd.c:351
> #9 main (argc=<value optimized out>, argv=<value optimized out>) at tgtd.c:522
> (gdb)
>
> > > > I started to run tgtd from the git repository. But it looks like
> > > > the tgtd is still losing memory.
> > >
> > > Just memory leak? It doesn't crash yet?
> >
> > Unfortunately, this time it just crashed:
> > Segmentation fault (core dumped) tgtd -f >& tgtd.log
> >
> >
> > > > > On Sat, 02 Oct 2010 07:34:55 +0900 (JST)
> > > > > Hirokazu Takahashi <taka at valinux.co.jp> wrote:
> > > > >
> > > > > > I tested tgtd 1.0.9 with heavy load. But unfortunatelly it exited
> > > > > > with a message "tgtd: iscsi_tx_handler(2210) error 0 0" as ever.
> > > > >
> > > > > Sorry, I haven't fixed anything.
> > > > >
> > > > > Can you try Chandra's patchset? It might work for you. Try the
> > > > > 'threads' branch:
> > > > >
> > > > > git://git.kernel.org/pub/scm/linux/kernel/git/tomo/tgt.git threads
> > > >
> > > > Tue Oct 12 20:16:43 JST 2010
> > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> > > > root 11528 0.5 1.3 182660 3408 pts/0 Sl 20:06 0:03 tgtd -f
> > > > Tue Oct 12 20:21:43 JST 2010
> > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> > > > root 11528 0.4 1.3 182660 3424 pts/0 Sl 20:06 0:04 tgtd -f
> > > > Tue Oct 12 20:26:44 JST 2010
> > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> > > > root 11528 0.4 1.3 182660 3424 pts/0 Sl 20:06 0:05 tgtd -f
> > > > Tue Oct 12 20:31:45 JST 2010
> > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> > > > root 11528 0.3 1.4 183164 3928 pts/0 Sl 20:06 0:05 tgtd -f
> > > > Tue Oct 12 20:36:46 JST 2010
> > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> > > > root 11528 0.3 1.5 183204 3968 pts/0 Dl 20:06 0:06 tgtd -f
> > > > Tue Oct 12 20:41:46 JST 2010
> > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> > > > root 11528 0.3 1.5 183428 4152 pts/0 Sl 20:06 0:07 tgtd -f
> > > > Tue Oct 12 20:46:46 JST 2010
> > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> > > > root 11528 0.3 1.6 183440 4204 pts/0 Sl 20:06 0:08 tgtd -f
> > > > Tue Oct 12 20:51:46 JST 2010
> > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> > > > root 11528 0.3 1.6 183440 4204 pts/0 Sl 20:06 0:09 tgtd -f
> > > > Tue Oct 12 20:56:46 JST 2010
> > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> > > > root 11528 0.3 1.6 183516 4280 pts/0 Sl 20:06 0:10 tgtd -f
> > > > Tue Oct 12 21:01:47 JST 2010
> > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> > > > root 11528 0.3 1.6 183660 4424 pts/0 Sl 20:06 0:11 tgtd -f
> > > > Tue Oct 12 21:07:17 JST 2010
> > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> > > > root 11528 0.3 1.7 183748 4512 pts/0 Sl 20:06 0:11 tgtd -f
> > > > Tue Oct 12 21:12:17 JST 2010
> > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> > > > root 11528 0.3 1.7 183884 4616 pts/0 Sl 20:06 0:13 tgtd -f
> > > > Tue Oct 12 21:17:17 JST 2010
> > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> > > > root 11528 0.3 1.8 184124 4840 pts/0 Sl 20:06 0:14 tgtd -f
> > > > Tue Oct 12 21:22:18 JST 2010
> > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> > > > root 11528 0.3 1.8 184124 4888 pts/0 Sl 20:06 0:14 tgtd -f
> > > > Tue Oct 12 21:27:21 JST 2010
> > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> > > > root 11528 0.3 1.8 184164 4928 pts/0 Sl 20:06 0:15 tgtd -f
> > > > Tue Oct 12 21:32:28 JST 2010
> > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> > > > root 11528 0.3 1.9 184388 5128 pts/0 Dl 20:06 0:16 tgtd -f
> > > > Tue Oct 12 21:37:28 JST 2010
> > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> > > > root 11528 0.3 2.0 184636 5400 pts/0 Sl 20:06 0:17 tgtd -f
> > > > Tue Oct 12 21:42:29 JST 2010
> > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> > > > root 11528 0.3 2.0 184636 5400 pts/0 Sl 20:06 0:18 tgtd -f
> > > >
> > > >
> > > > # tgtadm --lld iscsi --op show --mode=target
> > > > Target 1: target1
> > > > System information:
> > > > Driver: iscsi
> > > > State: ready
> > > > I_T nexus information:
> > > > I_T nexus: 4
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 7
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 8
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 12
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 14
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 16
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 18
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 24
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 26
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 29
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 35
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 39
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 44
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 47
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 48
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 49
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 50
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 51
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 52
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 53
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 54
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 55
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 56
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 57
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 58
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 59
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 60
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 61
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 62
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 63
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 64
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 65
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 66
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 67
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 69
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 70
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > LUN information:
> > > > LUN: 0
> > > > Type: controller
> > > > SCSI ID: IET 00010000
> > > > SCSI SN: beaf10
> > > > Size: 0 MB
> > > > Online: Yes
> > > > Removable media: No
> > > > Readonly: No
> > > > Backing store type: null
> > > > Backing store path: None
> > > > Backing store flags:
> > > > LUN: 1
> > > > Type: disk
> > > > SCSI ID: 1.1
> > > > SCSI SN: xvdc1
> > > > Size: 20012 MB
> > > > Online: Yes
> > > > Removable media: No
> > > > Readonly: No
> > > > Backing store type: rdwr
> > > > Backing store path: /dev/xvdc1
> > > > Backing store flags:
> > > > LUN: 2
> > > > Type: disk
> > > > SCSI ID: 1.2
> > > > SCSI SN: xvdd1
> > > > Size: 20012 MB
> > > > Online: Yes
> > > > Removable media: No
> > > > Readonly: No
> > > > Backing store type: rdwr
> > > > Backing store path: /dev/xvdd1
> > > > Backing store flags:
> > > > Account information:
> > > > ACL information:
> > > > ALL
> > > > Target 2: target2
> > > > System information:
> > > > Driver: iscsi
> > > > State: ready
> > > > I_T nexus information:
> > > > I_T nexus: 2
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 5
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 6
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 9
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 10
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 11
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 13
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 15
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 17
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 19
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 20
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 23
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 25
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 28
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 32
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 33
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 34
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 36
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 37
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 40
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest01
> > > > Connection: 0
> > > > IP Address: 172.17.112.101
> > > > I_T nexus: 41
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 42
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 43
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 46
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > I_T nexus: 68
> > > > Initiator: iqn.2010-09.jp.co.valinux:guest02
> > > > Connection: 0
> > > > IP Address: 172.17.112.102
> > > > LUN information:
> > > > LUN: 0
> > > > Type: controller
> > > > SCSI ID: IET 00020000
> > > > SCSI SN: beaf20
> > > > Size: 0 MB
> > > > Online: Yes
> > > > Removable media: No
> > > > Readonly: No
> > > > Backing store type: null
> > > > Backing store path: None
> > > > Backing store flags:
> > > > LUN: 1
> > > > Type: disk
> > > > SCSI ID: 2.1
> > > > SCSI SN: xvdc2
> > > > Size: 19987 MB
> > > > Online: Yes
> > > > Removable media: No
> > > > Readonly: No
> > > > Backing store type: rdwr
> > > > Backing store path: /dev/xvdc2
> > > > Backing store flags:
> > > > LUN: 2
> > > > Type: disk
> > > > SCSI ID: 2.2
> > > > SCSI SN: xvdd2
> > > > Size: 19987 MB
> > > > Online: Yes
> > > > Removable media: No
> > > > Readonly: No
> > > > Backing store type: rdwr
> > > > Backing store path: /dev/xvdd2
> > > > Backing store flags:
> > > > Account information:
> > > > ACL information:
> > > > ALL
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe stgt" in
> > > > the body of a message to majordomo at vger.kernel.org
> > > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > > >
> > > >
> > --
> > To unsubscribe from this list: send the line "unsubscribe stgt" in
> > the body of a message to majordomo at vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe stgt" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
--
To unsubscribe from this list: send the line "unsubscribe stgt" in
the body of a message to majordomo at vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
More information about the stgt
mailing list