[stgt] tgtd exits in iscsi_tx_handler during heavy I/Os

[Kiefer Chang]

Hi,

stgt is good and easy to use. We are using it in our private cloud environment.
We face a problem and hope we can get some help.

We are running heavy I/O on targets created by stgt.

tgtd (1.0.12, no RDMA) is running in a virtual machines and we use
files on Lustre file system as backing stores.
Every target is using a file on Lustre as its LUN 1.
The OS is based on CentOS 5.4, kernel version is 2.6.18-164.

There are about 15~ targets for this I/O tests.
(they are logged in by 1 physical node, and the disks are attached to
3 virtual machines).
I/O test is vdbench file system I/O (performing in the VM).

We also running tests to create/delete other targets repeatedly at the
same time.


After some period of time, the tgtd exits with the following message:
(Please see get the completed log file from
http://dl.dropbox.com/u/8354750/messages.kiefer.gz)


May 12 17:55:36 localhost tgtd: conn_close(101) connection closed, 0x105c6ca8 3
May 12 17:55:36 localhost tgtd: conn_close(107) sesson 0x10882630 1
May 12 17:55:39 localhost tgtd: conn_close(90) already closed 0x105c6ca8 2
May 12 17:56:47 localhost tgtd: abort_task_set(1149) found 40000009 0
May 12 17:56:47 localhost tgtd: abort_task_set(1149) found 0 0
May 12 17:56:47 localhost tgtd: abort_cmd(1125) found 40000045 e
May 12 17:57:08 localhost tgtd: conn_close(101) connection closed, 0x1074c4c8 3
May 12 17:57:08 localhost tgtd: conn_close(107) sesson 0x108a5d90 1
May 12 17:57:10 localhost tgtd: iscsi_tx_handler(2244) error 2 22
May 12 17:57:10 localhost tgtd: tgtd logger exits abnormally, pid:3296


The symptom is not always reproduce-able, sometimes it happens twice a
day, sometimes it runs well for server days.

iscsi_tx_handler(2244) error 2 22

is not the only exit message, we also saw:

iscsi_tx_handler(2244) error 0 0

By searching the mailing list we found two threads discussing
iscsi_tx_handler exit issue, but there are no further information in
those threads.
We suspect this is a TMF issues just as one thread says, but we still
can't assure.

Thanks for your time,
Any help will be valuable, thanks!
--
Kiefer Chang
--
To unsubscribe from this list: send the line "unsubscribe stgt" in
the body of a message to majordomo at vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html