FUJITA Tomonori schrieb: > On Wed, 11 Mar 2009 09:01:50 +0100 > Tomasz Chmielewski <mangoo at wpkg.org> wrote: > >> FUJITA Tomonori schrieb: >>> RHEL 5.3 release note says (I think that some of you guys already know >>> this): >>> >>> = >>> iSCSI target capability >>> >>> The iSCSI target capability, delivered as part of the Linux Target >>> (tgt) framework, moves from Technology Preview to full support in Red >> (...) >> >>> http://www.redhat.com/docs/manuals/enterprise/ >>> >>> >>> We need to be more serious. :) Hopefully, this would attract more >>> users to tgt. >>> >>> Feel feel to post what features you want, what areas need to be fixed, >>> etc. >> The only thing that bothers me is target/lun limit, caused by libc >> pthread limit. I thought it only affects 32 bit targets, but now I see >> it happens also with 64 bits. >> >> I understand it's a major design problem - do you think it could be >> addressed? I.e., use prefork model used by Apache and just launch more >> tgtd processes? > > I'm not sure Apache uses such model, 'client per process with > preforking) because it sucks (I guess it used the model): > > http://www.kegel.com/c10k.html Apache binary can be build to run in different modes: prefork, worker etc. > The only reason we have to use pthread is that Linux AIO is > broken. I've been waiting for the new AIO mechanism: > > http://lwn.net/Articles/316193/ > > It would improve the performance too. > > > Anyway, the target/lun limit problem is bad. So here's a patch to fix > it. I've not tested it much. So let me know how it works. Great! I applied the patch to 0.9.4. On a 32 bit x86 system with 256 MB RAM I created 20000 targets, each with one lun (without connecting any initiators). Later, I added another one target with 4200 luns. Should be enough for a typical home server usage ;) In all, memory usage is: # free total used free shared buffers cached Mem: 248424 153404 95020 0 160 16944 -/+ buffers/cache: 136300 112124 Swap: 3145720 169624 2976096 There are the issues I found: 1) "tgtadm --op show --mode target", which: - doesn't list all targets - output for the last target is sometimes not full, i.e.: Target 19836: iqn.2007-02.com.example:test.break19836 System information: Driver: iscsi State: ready I_T nexus information: LUN information: LUN: 0 Type: controller SCSI ID: deadbeaf19836:0 SCSI SN: beaf198360 Size: 0 MB Online: Yes Removable media: No Backing store: No backing store LUN: 1 Type: disk SCSI ID: deadbeaf19836:1 SCSI SN: beaf198361 Size: 1024 MB Online: Yes Removable media: No Backing store: /dev/sda Account information: Asuperthecus:/mnt/1# Note that the last piece of information outputted by tgtadm is: Account information: A And it missed some targets up to 20000. Or, a similar output when there are lots of luns (4200): LUN: 3431 Type: disk SCSI ID: deadbeaf20001:3431 SCSI SN: beaf200013431 Size: 1024 MB Online: Yes Removable media: No Backing store: /dev/sda LUN: 3432 Type: disk SCSI ID: deadbeaf20001:3432 SCSI SN: beaf200013432 Size: 1024 MB superthecus:/mnt/1# 2) running "iscsiadm -m discovery -t sendtargets -p <TARGET_IP> while we keep adding new targets breaks the target (i.e. you can't add any new targets: showing target is not possible). To reproduce: tgtd i=1 while [ $i -ne 20000 ] ; do echo tgtadm --lld iscsi --op new --mode target --tid $i -T iqn.2007-02.com.example:test.break$i tgtadm --lld iscsi --op new --mode target --tid $i -T iqn.2007-02.com.example:test.break$i echo tgtadm --lld iscsi --op new --mode logicalunit --tid $i --lun 1 -b /dev/sda tgtadm --lld iscsi --op new --mode logicalunit --tid $i --lun 1 -b /dev/sda [ $? -ne 0 ] && echo Oooops && sleep 10m tgtadm --lld iscsi --op bind --mode target --tid $i -I ALL i=$((i+1)) done On the initiator, run - you will see problems like below: # iscsiadm -m discovery -t sendtargets -p 192.168.111.177 iscsiadm: Connection to Discovery Address 192.168.111.177 closed # iscsiadm -m discovery -t sendtargets -p 192.168.111.177 iscsiadm: buffer size 32768 too small for data length 68010 # iscsiadm -m discovery -t sendtargets -p 192.168.111.177 iscsiadm: socket 3 header read timed out iscsiadm: Login I/O error, failed to receive a PDU iscsiadm: retrying discovery login to 192.168.111.177 iscsiadm: socket 3 header read timed out iscsiadm: Login I/O error, failed to receive a PDU iscsiadm: retrying discovery login to 192.168.111.177 3) even when we use a much smaller number of targets (i.e. 350), running: # iscsiadm -m discovery -t sendtargets -p 192.168.111.177 succeeds for the first time, but the second run fails (i.e., tgtd does not answer). Maximum number of targets with one lun which worked for me was 96 (on 32 bit). With 97, it was failing. When stracing, I can see: [pid 2108] semtimedop(294912, 0x1, 1, {4294901760, 3266314240}) = 0 [pid 2108] semop(294912, 0x77eafe46, 1) = 0 [pid 2108] open("/dev/tty", O_RDWR|O_NONBLOCK|O_NOCTTY) = -1 ENXIO (No such device or address) [pid 2108] writev(2, [{"*** glibc detected *** ", 23}, {"free(): invalid next size (norma"..., 34}, {": 0x", 4}, {"08f6a010", 8}, {" ***\n", 5}], 5) = 74 [pid 2108] rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0 [pid 2108] tgkill(2108, 2108, SIGABRT) = 0 [pid 2108] --- SIGABRT (Aborted) @ 0 (0) --- Process 2108 detached [pid 2118] +++ killed by SIGABRT +++ Process 2118 detached [pid 2117] +++ killed by SIGABRT +++ Process 2117 detached [pid 2116] +++ killed by SIGABRT +++ Process 2116 detached [pid 2115] +++ killed by SIGABRT +++ Process 2115 detached [pid 2114] +++ killed by SIGABRT +++ Process 2114 detached [pid 2113] +++ killed by SIGABRT +++ Process 2113 detached [pid 2112] +++ killed by SIGABRT +++ Process 2112 detached [pid 2111] +++ killed by SIGABRT +++ Process 2111 detached [pid 2110] +++ killed by SIGABRT +++ -- Tomasz Chmielewski http://wpkg.org -- To unsubscribe from this list: send the line "unsubscribe stgt" in the body of a message to majordomo at vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html |