[stgt] RHEL 5.3
Tomasz Chmielewski
mangoo at wpkg.org
Wed Mar 11 14:05:42 CET 2009
FUJITA Tomonori schrieb:
> On Wed, 11 Mar 2009 09:01:50 +0100
> Tomasz Chmielewski <mangoo at wpkg.org> wrote:
>
>> FUJITA Tomonori schrieb:
>>> RHEL 5.3 release note says (I think that some of you guys already know
>>> this):
>>>
>>> =
>>> iSCSI target capability
>>>
>>> The iSCSI target capability, delivered as part of the Linux Target
>>> (tgt) framework, moves from Technology Preview to full support in Red
>> (...)
>>
>>> http://www.redhat.com/docs/manuals/enterprise/
>>>
>>>
>>> We need to be more serious. :) Hopefully, this would attract more
>>> users to tgt.
>>>
>>> Feel feel to post what features you want, what areas need to be fixed,
>>> etc.
>> The only thing that bothers me is target/lun limit, caused by libc
>> pthread limit. I thought it only affects 32 bit targets, but now I see
>> it happens also with 64 bits.
>>
>> I understand it's a major design problem - do you think it could be
>> addressed? I.e., use prefork model used by Apache and just launch more
>> tgtd processes?
>
> I'm not sure Apache uses such model, 'client per process with
> preforking) because it sucks (I guess it used the model):
>
> http://www.kegel.com/c10k.html
Apache binary can be build to run in different modes: prefork, worker etc.
> The only reason we have to use pthread is that Linux AIO is
> broken. I've been waiting for the new AIO mechanism:
>
> http://lwn.net/Articles/316193/
>
> It would improve the performance too.
>
>
> Anyway, the target/lun limit problem is bad. So here's a patch to fix
> it. I've not tested it much. So let me know how it works.
Great! I applied the patch to 0.9.4.
On a 32 bit x86 system with 256 MB RAM I created 20000 targets,
each with one lun (without connecting any initiators).
Later, I added another one target with 4200 luns.
Should be enough for a typical home server usage ;)
In all, memory usage is:
# free
total used free shared buffers cached
Mem: 248424 153404 95020 0 160 16944
-/+ buffers/cache: 136300 112124
Swap: 3145720 169624 2976096
There are the issues I found:
1) "tgtadm --op show --mode target", which:
- doesn't list all targets
- output for the last target is sometimes not full, i.e.:
Target 19836: iqn.2007-02.com.example:test.break19836
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: deadbeaf19836:0
SCSI SN: beaf198360
Size: 0 MB
Online: Yes
Removable media: No
Backing store: No backing store
LUN: 1
Type: disk
SCSI ID: deadbeaf19836:1
SCSI SN: beaf198361
Size: 1024 MB
Online: Yes
Removable media: No
Backing store: /dev/sda
Account information:
Asuperthecus:/mnt/1#
Note that the last piece of information outputted by tgtadm is:
Account information:
A
And it missed some targets up to 20000.
Or, a similar output when there are lots of luns (4200):
LUN: 3431
Type: disk
SCSI ID: deadbeaf20001:3431
SCSI SN: beaf200013431
Size: 1024 MB
Online: Yes
Removable media: No
Backing store: /dev/sda
LUN: 3432
Type: disk
SCSI ID: deadbeaf20001:3432
SCSI SN: beaf200013432
Size: 1024 MB
superthecus:/mnt/1#
2) running "iscsiadm -m discovery -t sendtargets -p <TARGET_IP> while we keep
adding new targets breaks the target
(i.e. you can't add any new targets: showing target is not possible).
To reproduce:
tgtd
i=1
while [ $i -ne 20000 ] ; do
echo tgtadm --lld iscsi --op new --mode target --tid $i -T iqn.2007-02.com.example:test.break$i
tgtadm --lld iscsi --op new --mode target --tid $i -T iqn.2007-02.com.example:test.break$i
echo tgtadm --lld iscsi --op new --mode logicalunit --tid $i --lun 1 -b /dev/sda
tgtadm --lld iscsi --op new --mode logicalunit --tid $i --lun 1 -b /dev/sda
[ $? -ne 0 ] && echo Oooops && sleep 10m
tgtadm --lld iscsi --op bind --mode target --tid $i -I ALL
i=$((i+1))
done
On the initiator, run - you will see problems like below:
# iscsiadm -m discovery -t sendtargets -p 192.168.111.177
iscsiadm: Connection to Discovery Address 192.168.111.177 closed
# iscsiadm -m discovery -t sendtargets -p 192.168.111.177
iscsiadm: buffer size 32768 too small for data length 68010
# iscsiadm -m discovery -t sendtargets -p 192.168.111.177
iscsiadm: socket 3 header read timed out
iscsiadm: Login I/O error, failed to receive a PDU
iscsiadm: retrying discovery login to 192.168.111.177
iscsiadm: socket 3 header read timed out
iscsiadm: Login I/O error, failed to receive a PDU
iscsiadm: retrying discovery login to 192.168.111.177
3) even when we use a much smaller number of targets (i.e. 350), running:
# iscsiadm -m discovery -t sendtargets -p 192.168.111.177
succeeds for the first time, but the second run fails (i.e., tgtd does not answer).
Maximum number of targets with one lun which worked for me was 96 (on 32 bit).
With 97, it was failing.
When stracing, I can see:
[pid 2108] semtimedop(294912, 0x1, 1, {4294901760, 3266314240}) = 0
[pid 2108] semop(294912, 0x77eafe46, 1) = 0
[pid 2108] open("/dev/tty", O_RDWR|O_NONBLOCK|O_NOCTTY) = -1 ENXIO (No such device or address)
[pid 2108] writev(2, [{"*** glibc detected *** ", 23}, {"free(): invalid next size (norma"..., 34}, {": 0x", 4}, {"08f6a010", 8}, {" ***\n", 5}], 5) = 74
[pid 2108] rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0
[pid 2108] tgkill(2108, 2108, SIGABRT) = 0
[pid 2108] --- SIGABRT (Aborted) @ 0 (0) ---
Process 2108 detached
[pid 2118] +++ killed by SIGABRT +++
Process 2118 detached
[pid 2117] +++ killed by SIGABRT +++
Process 2117 detached
[pid 2116] +++ killed by SIGABRT +++
Process 2116 detached
[pid 2115] +++ killed by SIGABRT +++
Process 2115 detached
[pid 2114] +++ killed by SIGABRT +++
Process 2114 detached
[pid 2113] +++ killed by SIGABRT +++
Process 2113 detached
[pid 2112] +++ killed by SIGABRT +++
Process 2112 detached
[pid 2111] +++ killed by SIGABRT +++
Process 2111 detached
[pid 2110] +++ killed by SIGABRT +++
--
Tomasz Chmielewski
http://wpkg.org
--
To unsubscribe from this list: send the line "unsubscribe stgt" in
the body of a message to majordomo at vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
More information about the stgt
mailing list