[Stgt-devel] sg_turs on stgt iscsi drive is very slow

[Ming Zhang]

On Tue, 2006-12-12 at 10:56 +0900, FUJITA Tomonori wrote:
> From: Ming Zhang <blackmagic02881 at gmail.com>
> Subject: Re: [Stgt-devel] sg_turs on stgt iscsi drive is very slow
> Date: Mon, 11 Dec 2006 18:47:25 -0500
> 
> > On Tue, 2006-12-12 at 08:21 +0900, FUJITA Tomonori wrote:
> > > From: Ming Zhang <blackmagic02881 at gmail.com>
> > > Subject: Re: [Stgt-devel] sg_turs on stgt iscsi drive is very slow
> > > Date: Mon, 11 Dec 2006 18:18:06 -0500
> > > 
> > > > On Tue, 2006-12-12 at 08:02 +0900, FUJITA Tomonori wrote:
> > > > > From: Ming Zhang <blackmagic02881 at gmail.com>
> > > > > Subject: [Stgt-devel] sg_turs on stgt iscsi drive is very slow
> > > > > Date: Mon, 11 Dec 2006 11:58:27 -0500
> > > > > 
> > > > > > Pulled out the linux-2.6-target git tree, compiled stgt and iet under
> > > > > > same kernel. Export a 4 disk raid0 device to ini. Ini is open-iscsi
> > > > > > r598.
> > > > > > 
> > > > > > Want to know how efficient stgt handle context switch. 
> > > > > > 
> > > > > > run  sg_turs -t -n=10000 /dev/sg0 on IET drive. results is 2521/sec.
> > > > > 
> > > > > With IET, several threads handle TUR. With stgt, the single thread
> > > > > handles TUR (synchronously). stgt asynchronously handles only I/O
> > > > > commands (READ, WRITE, etc), which are expected to be performed
> > > > > efficiently.
> > > > 
> > > > if i set Wthreads to 1 and "ps axu" to double check there is only 1
> > > > worker thread, i still get 5470 op/sec.
> > > > 
> > > > also when run stgt, the top show almost no cpu activity. 
> > > 
> > > With that configuration, IET still uses three threads (two network
> > > threads and worker thread).
> > 
> > could u roughly explain a code flow of stgt for a TUR? why I see no cpu
> > utilization during stgt run? for iet, i saw 4-5% cpu utilization so i
> > know it is working. but for stgt i can not see any from top or vmstat.
> 
> One process reads an iSCSI pdu, and builds and sends response. Then
> the process reads another iSCSI pdu...

if so, since build a TUR take no time and no disk activity, the cpu
utilization should be pretty high i think. but why i see near zero cpu
utilization.

if you think TUR is not a good test, will this be a good one? "test
client repeat read 4KB data from same LBA. supposedly will read from
cache. this is a typical case to test IOPS."

> 
> 
> > > > > So I don't think the results are related with context switch at all.
> > > > 
> > > > ok, it is not related to CS. but then why stgt is so slow?
> > > > 
> > > > it should be easy to reproduce this at u side. if u can not reproduce
> > > > this slowness, i will check my environment again. i clone u git tress
> > > > just yesterday.
> > > 
> > > Please do real workload tests. If stgt is slow with such workloads,
> > > I'll dig into it. Thanks.
> > 
> > i tried simple dd read from iscsi drive and only give me 30MB/s. but i
> > forgot if i disable the debug or not. i will run it again and report.
> 
> I used disktest months ago and got the comparable results. After that,
> stgt switch to CMD_EPOLL_WAIT from AIO fd for event notification. That
> might effect the performance, but probably stgt will switch to kevent
> in the future.

i have a 4 disk raid0 which can run 100MB/s wire speed with iet. but
this morning i ran at 30MB/s with stgt. i will redo more tests and
report later.