[stgt] help tgt segfault

[FUJITA Tomonori]

On Fri, 19 Dec 2008 16:07:51 +0100
Tomasz Chmielewski <mangoo at wpkg.org> wrote:

> Tomasz Chmielewski schrieb:
> > FUJITA Tomonori schrieb:
> >> On Wed, 17 Dec 2008 18:51:41 +0100
> >> Tomasz Chmielewski <mangoo at wpkg.org> wrote:
> >>
> >>> FUJITA Tomonori schrieb:
> >>>
> >>>> Can you try one more time with this patch (including the previous
> >>>> patch so please do git-reset --hard first).
> >>> Here you are:
> >>>
> >>> Dec 17 18:46:27 megathecus tgtd: Target daemon logger with pid=21048 
> >>> started!
> >>
> >> Thanks a lot! This is very useful.
> >>
> >> Can you try this again? Even if tgtd doesn't crash, please send the
> >> log.
> > 
> > I couldn't crash it any more.
> > 
> > Here is a lengthy log for 25 minutes - with several device 
> > suspends/resumes.
> > I'll give it 2 more hours testing...
> 
> Not 2 hours, but some observations - is it possible that tgtd serves 
> wrong data to the initiator when the access to the media is slow?

tgtd doesn't return wrong (bogus) data.

The initiator has to give up I/O requests at some point if the target
doesn't send responses for long time. Then you see the I/O errors.

Or on the target side, the kernel gives tgtd I/O errors if the backing
store doesn't return responses for long time. Then tgtd sends the
errors to the initiator.


Try scsi_debug if you want to see I/O errors due to timeout:

vine:~# modprobe scsi_debug opts=4
vine:~# cd /sys/module/scsi_debug/parameters
vine:/sys/module/scsi_debug/parameters# echo 1 > every_nth

Then try to read the scsi_debug device. In my case:

vine:~# lsscsi
[0:0:0:0]    disk    IBM      1814      FAStT  2916  /dev/sda
[0:0:0:1]    disk    IBM      1814      FAStT  2916  /dev/sdb
[0:0:0:2]    disk    IBM      1814      FAStT  2916  /dev/sdc
[0:0:0:31]   disk    IBM      Universal Xport  2916  -
[1:0:0:0]    disk    Linux    scsi_debug       0004  /dev/sdd

vine:~# dd if=/dev/sdd of=/dev/null count=1

If you wait for some time, then you see I/O errors like:

dd: reading `/dev/sdd': Input/output error
0+0 records in
0+0 records out
0 bytes (0 B) copied, 90.2521 seconds, 0.0 kB/s


And you see something like this in the kernel log:

sd 1:0:0:0: Device offlined - not ready after error recovery
sd 1:0:0:0: [sdd] Unhandled error code<6>sd 1:0:0:0: [sdd] Result:
hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT,SUGGEST_OK
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
Buffer I/O error on device sdd, logical block 1
Buffer I/O error on device sdd, logical block 2
Buffer I/O error on device sdd, logical block 3
sd 1:0:0:0: rejecting I/O to offline device
Buffer I/O error on device sdd, logical block 0


You can replace dd with tgtd in the above example.


> During the suspend/resume cycle on the target and dd if=/dev/iscsi_disk 
> on the initiator, I got some errors:
> 
> # dd if=/dev/sdb of=/dev/null
> dd: reading `/dev/sdb': Input/output error
> 1506128+0 records in
> 1506128+0 records out
> 771137536 bytes (771 MB) copied, 555.773 seconds, 1.4 MB/s
> 
> # dmesg
> <lots of connection errors / host resets>
>   connection1:0: iscsi: detected conn error (1011)
> iscsi: host reset succeeded
>   connection2:0: iscsi: detected conn error (1011)
> iscsi: host reset succeeded
> sd 3:0:0:1: [sdb] Result: hostbyte=0x02 driverbyte=0x00
> end_request: I/O error, dev sdb, sector 1506384
> Buffer I/O error on device sdb, logical block 188298
> Buffer I/O error on device sdb, logical block 188299
> Buffer I/O error on device sdb, logical block 188300
> Buffer I/O error on device sdb, logical block 188301
> Buffer I/O error on device sdb, logical block 188302
> Buffer I/O error on device sdb, logical block 188303
> Buffer I/O error on device sdb, logical block 188304
> Buffer I/O error on device sdb, logical block 188305
> Buffer I/O error on device sdb, logical block 188306
> Buffer I/O error on device sdb, logical block 188307
> sd 3:0:0:1: [sdb] Result: hostbyte=0x02 driverbyte=0x00
> end_request: I/O error, dev sdb, sector 1506136
> sd 3:0:0:1: [sdb] Result: hostbyte=0x02 driverbyte=0x00
> end_request: I/O error, dev sdb, sector 1506128
>   connection2:0: iscsi: detected conn error (1011)
> iscsi: host reset succeeded
> 
> 
> 
> I had the exactly same issue (I/O errors) when tgtd:
> - is accessing a DRBD device (a device replicated over network)
> - DRBD device is being synchronized (over a slow, 2 Mbit internet link), 
> using the whole bandwidth
> - any writes (needing replication over network) done by the initiator 
> will be very slow, because of the ongoing sync and slow link

You need longer timeout on the initiator side or target side, or both.
--
To unsubscribe from this list: send the line "unsubscribe stgt" in
the body of a message to majordomo at vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html