[stgt] RHEL 5.3

Tomasz Chmielewski mangoo at wpkg.org
Wed Mar 11 14:05:42 CET 2009

FUJITA Tomonori schrieb:
> On Wed, 11 Mar 2009 09:01:50 +0100
> Tomasz Chmielewski <mangoo at wpkg.org> wrote:
>> FUJITA Tomonori schrieb:
>>> RHEL 5.3 release note says (I think that some of you guys already know
>>> this):
>>> =
>>> iSCSI target capability
>>> The iSCSI target capability, delivered as part of the Linux Target
>>> (tgt) framework, moves from Technology Preview to full support in Red
>> (...)
>>> http://www.redhat.com/docs/manuals/enterprise/
>>> We need to be more serious. :) Hopefully, this would attract more
>>> users to tgt.
>>> Feel feel to post what features you want, what areas need to be fixed,
>>> etc.
>> The only thing that bothers me is target/lun limit, caused by libc 
>> pthread limit. I thought it only affects 32 bit targets, but now I see 
>> it happens also with 64 bits.
>> I understand it's a major design problem - do you think it could be 
>> addressed? I.e., use prefork model used by Apache and just launch more 
>> tgtd processes?
> I'm not sure Apache uses such model, 'client per process with
> preforking) because it sucks (I guess it used the model):
> http://www.kegel.com/c10k.html

Apache binary can be build to run in different modes: prefork, worker etc.

> The only reason we have to use pthread is that Linux AIO is
> broken. I've been waiting for the new AIO mechanism:
> http://lwn.net/Articles/316193/
> It would improve the performance too.
> Anyway, the target/lun limit problem is bad. So here's a patch to fix
> it. I've not tested it much. So let me know how it works.

Great! I applied the patch to 0.9.4.

On a 32 bit x86 system with 256 MB RAM I created 20000 targets,
each with one lun (without connecting any initiators).

Later, I added another one target with 4200 luns.

Should be enough for a typical home server usage ;)

In all, memory usage is:

# free
             total       used       free     shared    buffers     cached
Mem:        248424     153404      95020          0        160      16944
-/+ buffers/cache:     136300     112124
Swap:      3145720     169624    2976096

There are the issues I found:

1) "tgtadm --op show --mode target", which:
- doesn't list all targets
- output for the last target is sometimes not full, i.e.:

Target 19836: iqn.2007-02.com.example:test.break19836
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: deadbeaf19836:0
            SCSI SN: beaf198360
            Size: 0 MB
            Online: Yes
            Removable media: No
            Backing store: No backing store
        LUN: 1
            Type: disk
            SCSI ID: deadbeaf19836:1
            SCSI SN: beaf198361
            Size: 1024 MB
            Online: Yes
            Removable media: No
            Backing store: /dev/sda
    Account information:

Note that the last piece of information outputted by tgtadm is:

    Account information:

And it missed some targets up to 20000.

Or, a similar output when there are lots of luns (4200):

        LUN: 3431
            Type: disk
            SCSI ID: deadbeaf20001:3431
            SCSI SN: beaf200013431
            Size: 1024 MB
            Online: Yes
            Removable media: No
            Backing store: /dev/sda
        LUN: 3432
            Type: disk
            SCSI ID: deadbeaf20001:3432
            SCSI SN: beaf200013432
            Size: 1024 MB

2) running "iscsiadm -m discovery -t sendtargets -p <TARGET_IP> while we keep
adding new targets breaks the target
(i.e. you can't add any new targets: showing target is not possible).

To reproduce:

while [ $i -ne 20000 ] ; do
    echo tgtadm --lld iscsi --op new --mode target --tid $i -T iqn.2007-02.com.example:test.break$i
    tgtadm --lld iscsi --op new --mode target --tid $i -T iqn.2007-02.com.example:test.break$i
    echo tgtadm --lld iscsi --op new --mode logicalunit --tid $i --lun 1 -b /dev/sda
    tgtadm --lld iscsi --op new --mode logicalunit --tid $i --lun 1 -b /dev/sda
    [ $? -ne 0 ] && echo Oooops && sleep 10m
    tgtadm --lld iscsi --op bind --mode target --tid $i -I ALL

On the initiator, run - you will see problems like below:

# iscsiadm -m discovery -t sendtargets -p
iscsiadm: Connection to Discovery Address closed

# iscsiadm -m discovery -t sendtargets -p
iscsiadm: buffer size 32768 too small for data length 68010

# iscsiadm -m discovery -t sendtargets -p
iscsiadm: socket 3 header read timed out
iscsiadm: Login I/O error, failed to receive a PDU
iscsiadm: retrying discovery login to
iscsiadm: socket 3 header read timed out
iscsiadm: Login I/O error, failed to receive a PDU
iscsiadm: retrying discovery login to

3) even when we use a much smaller number of targets (i.e. 350), running:

# iscsiadm -m discovery -t sendtargets -p

succeeds for the first time, but the second run fails (i.e., tgtd does not answer).

Maximum number of targets with one lun which worked for me was 96 (on 32 bit).
With 97, it was failing.

When stracing, I can see:

[pid  2108] semtimedop(294912, 0x1, 1, {4294901760, 3266314240}) = 0                                                                                        
[pid  2108] semop(294912, 0x77eafe46, 1) = 0                                                                                                                
[pid  2108] open("/dev/tty", O_RDWR|O_NONBLOCK|O_NOCTTY) = -1 ENXIO (No such device or address)                                                             
[pid  2108] writev(2, [{"*** glibc detected *** ", 23}, {"free(): invalid next size (norma"..., 34}, {": 0x", 4}, {"08f6a010", 8}, {" ***\n", 5}], 5) = 74  
[pid  2108] rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0                                                                                                
[pid  2108] tgkill(2108, 2108, SIGABRT) = 0                                                                                                                 
[pid  2108] --- SIGABRT (Aborted) @ 0 (0) ---                                                                                                               
Process 2108 detached                                                                                                                                       
[pid  2118] +++ killed by SIGABRT +++                                                                                                                       
Process 2118 detached                                                                                                                                       
[pid  2117] +++ killed by SIGABRT +++                                                                                                                       
Process 2117 detached                                                                                                                                       
[pid  2116] +++ killed by SIGABRT +++                                                                                                                       
Process 2116 detached                                                                                                                                       
[pid  2115] +++ killed by SIGABRT +++                                                                                                                       
Process 2115 detached                                                                                                                                       
[pid  2114] +++ killed by SIGABRT +++                                                                                                                       
Process 2114 detached                                                                                                                                       
[pid  2113] +++ killed by SIGABRT +++                                                                                                                       
Process 2113 detached                                                                                                                                       
[pid  2112] +++ killed by SIGABRT +++                                                                                                                       
Process 2112 detached                                                                                                                                       
[pid  2111] +++ killed by SIGABRT +++                                                                                                                       
Process 2111 detached                                                                                                                                       
[pid  2110] +++ killed by SIGABRT +++                               

Tomasz Chmielewski
To unsubscribe from this list: send the line "unsubscribe stgt" in
the body of a message to majordomo at vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

More information about the stgt mailing list