Pete Wyckoff wrote: > erezz at Voltaire.COM wrote on Mon, 10 Mar 2008 15:20 +0200: > >> Pete Wyckoff wrote: >> >>> Agreed, that's rather slow, 480 MB/s. Something else is going on. >>> Closest number I can lay my hands on says 350 kB was 94 us in the >>> pread, 3800 MB/s. >>> >> What's your setup? I'm using a RAM disk that I found here: >> >> http://marc.info/?l=linux-scsi&m=120331663227540&w=2 >> > > Well that would be rather unusual. > > Most of the world just does: > > mkdir /tmp/ramdisk > mount -t tmpfs none /tmp/ramdisk > dd if=/dev/zero bs=1M count=1024 of=/tmp/ramdisk/lun1 > tgtadm ... --backing-store /tmp/ramdisk/lun1 > > or similar. > You always learn something new :-) . We didn't use RAM disks until now. > >>> You should be measuring memory copy speed here. >>> >>> >> Do you mean that memory copy is 480 MB/sec? That's slow. >> >> >>> >>> >>>> Another question is - how does pread64 access the SCSI device? I >>>> understand that it reads from /dev/sdX. Does it call sd? How? Is there >>>> any memory copy involved? I'm asking that because I'm used to kernel >>>> space where we just call scsi_do_req. >>>> >>>> >>> It reads from wherever it put your device with ./tgtadm ... >>> --backing-store ... . Presumably a file on the file system, or a >>> raw block device like /dev/sdb. >>> >> Of course. The question is - what is the interface between pread and >> scsi-ml? That's what I still don't understand. >> > > You can start with the system call and follow it down: sys_pread64, > vfs_read, ..., ext3_readpage, ..., submit_bio, ... . But I'm > talking tmpfs, which is slightly different. And your experimental > scsi ram driver would export a block device but still goes > similarly through blkdev_readpage, submit_bh and on down. > > You shouldn't really have to care. This is internal plumbing that > better work. > > OK. So, eventually (with a real storage) this will go down to the sd driver and to scsi_mod. Is there any copy_from_user on the way or any data copy? >> strace looks like that: >> >> epoll_wait(3, {}, 1024, 2000) = 0 <2.000066> >> epoll_wait(3, {}, 1024, 2000) = 0 <2.000068> >> epoll_wait(3, {}, 1024, 2000) = 0 <2.000065> >> epoll_wait(3, {}, 1024, 2000) = 0 <2.000065> >> epoll_wait(3, {{EPOLLIN, {u32=5394880, u64=5394880}}}, 1024, 2000) = 1 >> <1.184798> >> read(10, "\320\235R\0\0\0\0\0", 8) = 8 <0.000009> >> pread(11, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., >> 131072, 0) = 131072 <0.000336> >> epoll_wait(3, {{EPOLLIN, {u32=5394880, u64=5394880}}}, 1024, 2000) = 1 >> <0.000088> >> read(10, "\320\235R\0\0\0\0\0", 8) = 8 <0.000024> >> epoll_wait(3, {{EPOLLIN, {u32=5394880, u64=5394880}}}, 1024, 2000) = 1 >> <0.000023> >> read(10, "\320\235R\0\0\0\0\0", 8) = 8 <0.000024> >> epoll_wait(3, {}, 1024, 2000) = 0 <1.998508> >> epoll_wait(3, {}, 1024, 2000) = 0 <1.999959> >> > > All the time is in pread. If a normal tmpfs fixes things, file a > bug report if you care about this scsi ram driver. Probably better > if you test it without iscsi and iser to see if it is just > inherently slow. > Looks better now (with the RAM disk that you use): epoll_wait(3, {{EPOLLIN, {u32=5394880, u64=5394880}}}, 1024, 2000) = 1 <0.000100> read(10, "\320\235R\0\0\0\0\0", 8) = 8 <0.000005> Now, I also get nicer numbers with sgp_dd (bs=512, bpt=1024, thr=8, time=1, count=102400000, dio=1): READ - 1380 MB/sec WRITE - 1420 MB/sec with small IOs (1k): READ - 40K WRITE - 20K I will try to put my hands on real fast storage and retest it. Erez |