[Stgt-devel] sg_turs on stgt iscsi drive is very slow

Tue Dec 12 17:13:33 CET 2006

recompile linux-2.6-target git kernel with no kernel debug options.

* 4 sata disk raid0 local test.

dd bs=1M count=4096. write 21sec, read 20sec. so all around 200MB/s

* iet

ietd.conf

Target iqn.2001-04.com.example:storage.disk2.sys1.xyz
        # Users, who can access this target
        Lun 0 Path=/dev/md0,IOMode=wb
        Wthreads 1

dd bs=1M count=4096. write 37sec, read 44sec. so all around 100MB/s

use attached code to test IOPS, get ~5K IOPS with 512B.

* stgt

same as yesterday. i think that EPOLL should have problem.

dd bs=1M count=4096. write 121sec, read 96sec. so write is only 25% of
iet got and read is half iet got.

use attached code to test IOPS, get ~245 IOPS with 512B. so match the
sg_turs results.

vmstat 1 during write. system is not very busy and no iowait. and yes,
CS is not high at all.

 0  0    576  10760 1089980   6672    0    0     0 33152 4482   324  0
14 86  0
 0  0    576  11456 1086524   6672    0    0     0 45584 5432   345  0
28 72  0
 0  0    576  10372 1097788   6656    0    0     0 49728 4317   314  0
17 83  0
 0  0    576  10116 1087856   6684    0    0     0     0 4713   317  0
11 89  0
 0  0    576  10140 1101400   6632    0    0     0 95312 5760   370  1
26 73  0
 0  0    576  10264 1095556   6604    0    0     0     0 4584   289  0
12 88  0
 0  0    576  11064 1101700   6568    0    0     0 45584 4989   310  0
20 80  0
 0  0    576   9996 1092740   6568    0    0     0 49728 3378   202  0
13 87  0
 0  0    576  10784 1098372   6584    0    0     0 45584 4981   325  1
19 80  0
 0  0    576  11116 1086724   6568    0    0     0     0 3563   210  0
8 92  0
 0  0    576  10024 1097476   6584    0    0     0 33152 4902   299  0
17 83  0
 0  0    576  10932 1088204   6548    0   20     0 45604 6999   423  1
22 77  0
 0  0    576  11028 1095372   6560    0    0     0 49728 5004   280  1
21 78  0
 0  0    576  10232 1089868   6608    0    0     0 45584 4479   266  0
17 83  0
 0  0    576  10492 1100236   6560    0    0     0 49728 4266   279  0
22 78  0
 0  0    576  10576 1096652   6608    0    0     0 45584 3946   269  1
19 80  0

intuitive feeling is that system is not responsive. 

Ming

On Mon, 2006-12-11 at 21:04 -0500, Ming Zhang wrote:
> On Tue, 2006-12-12 at 10:56 +0900, FUJITA Tomonori wrote:
> > From: Ming Zhang <blackmagic02881 at gmail.com>
> > Subject: Re: [Stgt-devel] sg_turs on stgt iscsi drive is very slow
> > Date: Mon, 11 Dec 2006 18:47:25 -0500
> > 
> > > On Tue, 2006-12-12 at 08:21 +0900, FUJITA Tomonori wrote:
> > > > From: Ming Zhang <blackmagic02881 at gmail.com>
> > > > Subject: Re: [Stgt-devel] sg_turs on stgt iscsi drive is very slow
> > > > Date: Mon, 11 Dec 2006 18:18:06 -0500
> > > > 
> > > > > On Tue, 2006-12-12 at 08:02 +0900, FUJITA Tomonori wrote:
> > > > > > From: Ming Zhang <blackmagic02881 at gmail.com>
> > > > > > Subject: [Stgt-devel] sg_turs on stgt iscsi drive is very slow
> > > > > > Date: Mon, 11 Dec 2006 11:58:27 -0500
> > > > > > 
> > > > > > > Pulled out the linux-2.6-target git tree, compiled stgt and iet under
> > > > > > > same kernel. Export a 4 disk raid0 device to ini. Ini is open-iscsi
> > > > > > > r598.
> > > > > > > 
> > > > > > > Want to know how efficient stgt handle context switch. 
> > > > > > > 
> > > > > > > run  sg_turs -t -n=10000 /dev/sg0 on IET drive. results is 2521/sec.
> > > > > > 
> > > > > > With IET, several threads handle TUR. With stgt, the single thread
> > > > > > handles TUR (synchronously). stgt asynchronously handles only I/O
> > > > > > commands (READ, WRITE, etc), which are expected to be performed
> > > > > > efficiently.
> > > > > 
> > > > > if i set Wthreads to 1 and "ps axu" to double check there is only 1
> > > > > worker thread, i still get 5470 op/sec.
> > > > > 
> > > > > also when run stgt, the top show almost no cpu activity. 
> > > > 
> > > > With that configuration, IET still uses three threads (two network
> > > > threads and worker thread).
> > > 
> > > could u roughly explain a code flow of stgt for a TUR? why I see no cpu
> > > utilization during stgt run? for iet, i saw 4-5% cpu utilization so i
> > > know it is working. but for stgt i can not see any from top or vmstat.
> > 
> > One process reads an iSCSI pdu, and builds and sends response. Then
> > the process reads another iSCSI pdu...
> 
> if so, since build a TUR take no time and no disk activity, the cpu
> utilization should be pretty high i think. but why i see near zero cpu
> utilization.
> 
> if you think TUR is not a good test, will this be a good one? "test
> client repeat read 4KB data from same LBA. supposedly will read from
> cache. this is a typical case to test IOPS."
> 
> > 
> > 
> > > > > > So I don't think the results are related with context switch at all.
> > > > > 
> > > > > ok, it is not related to CS. but then why stgt is so slow?
> > > > > 
> > > > > it should be easy to reproduce this at u side. if u can not reproduce
> > > > > this slowness, i will check my environment again. i clone u git tress
> > > > > just yesterday.
> > > > 
> > > > Please do real workload tests. If stgt is slow with such workloads,
> > > > I'll dig into it. Thanks.
> > > 
> > > i tried simple dd read from iscsi drive and only give me 30MB/s. but i
> > > forgot if i disable the debug or not. i will run it again and report.
> > 
> > I used disktest months ago and got the comparable results. After that,
> > stgt switch to CMD_EPOLL_WAIT from AIO fd for event notification. That
> > might effect the performance, but probably stgt will switch to kevent
> > in the future.
> 
> i have a 4 disk raid0 which can run 100MB/s wire speed with iet. but
> this morning i ran at 30MB/s with stgt. i will redo more tests and
> report later.
> 
> 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: readfromdiskcache.c
Type: text/x-csrc
Size: 1372 bytes
Desc: not available
Url : https://lists.berlios.de/pipermail/stgt-devel/attachments/20061212/ce92df86/attachment.bin 