[Stgt-devel] User-mode iSER

[Ming Zhang]

On Tue, 2006-08-01 at 16:38 -0500, Tom Tucker wrote:
> On Tue, 2006-08-01 at 17:24 -0400, Ming Zhang wrote:
> > On Tue, 2006-08-01 at 15:43 -0500, Tom Tucker wrote:
> > > On Tue, 2006-08-01 at 16:21 -0400, Ming Zhang wrote:
> > > > On Tue, 2006-08-01 at 14:00 -0500, Tom Tucker wrote:
> > > > > On Tue, 2006-08-01 at 14:49 -0400, Ming Zhang wrote:
> > > > > > my 2c. u figure ignores other device types like VT/VTL or bridge. your
> > > > > > target device type are only TYPE_DISK here.
> > > > > 
> > > > > Yeah, I should put in ellipses... These types are not precluded.
> > > > > 
> > > > > >  also scsi/block/file just
> > > > > > physical media used by TYPE_DISK.
> > > > > 
> > > > > Yes, but there seemed to be some interest in exporting a file as a SCSI
> > > > > Target, or even anonymously mapped memory...
> > > > > 
> > > > > What else should we be thinking about?
> > > > 
> > > > SPI is not mentioned in your pic.
> > > 
> > > Hmm. Doesn't this sit on the target device side underneath /dev/sdN?
> > 
> > sorry but i mean SPI transport, or interface with INI.
> > 
> > > 
> > > > 
> > > > and it will be better if kernel/user space line is marked. so people
> > > > know your intention and can comment in that if need. otherwise, i saw no
> > > > difference from stgt or scst.
> > > 
> > > Attaches is a new picture with kernel/user boundaries. I think the
> > > boundary is somewhat fuzzy at the top and bottom due to the fact that
> > > the API is different for each target type and for each network provider
> > > type. 
> > > 
> > > For example, the sd driver will use open /dev/sdN and submit ioctl to
> > > read/write the disk, whereas the file target will
> > > open("/home/thisandthat", ...) and submit lseek/read/write to read/write
> > > the file.
> > > 
> > > On the network side, the TCP provider would use socket, listen, accept,
> > > send, recv (at least initially), the RMDA provider would use
> > > rdma_resolve_addr, rdma_resolve_route, rdma_listen, rdma_accept,
> > > rdma_create_qp, rdma_post_send, etc....
> > > 
> > > My only point is that the kernel/user interface is not a straight line
> > > since some of these interfaces are higher level than others.
> > 
> > so you have even all transports in user space? zero copy socket is not
> > easy. 
> 
> I think this is the area where we will need to get fancy if we want
> higher performance. To avoid the copy, we would have to migrate to
> netchannels (if they every happen) or implement our own simple tear-away
> buffer scheme on top of a socket. I think this is phase-ii, however. 

ok, otherwise copy to user space and copy back to kernel for disk = low
performance. yes, direct io can be used here, but then u lose whole
cache benefits.

> 
> How about on the target device side? How expensive/inefficient is the
> ioctl interface to /dev/sd?
> 
> > and can FCP drivers, like a target mode driver for a QLA or LSI
> > HBA can be in user space as well?
> 
> No. These will be in the kernel, but I'm not considering these drivers
> as part of this subsystem. Are they part of IET?

no IET only contain iscsi tcp transport.

you need an in-kernel target mode driver for FCP HBA. and i think this
are one type of transports.


> 
> > 
> > ming
> > 
> > 
> > > 
> > > 
> > > 
> > > > 
> > > > also license. :P what license(s) will be used and allowed for each part.
> > > > 
> > > > ming
> > > > 
> > > > 
> > > > 
> > > > > 
> > > > > > 
> > > > > > Ming
> > > > > > 
> > > > > > 
> > > > > > On Tue, 2006-08-01 at 13:42 -0500, Tom Tucker wrote:
> > > > > > > What do people think about something like this...
> > > > > > > 
> > > > > > > The target architecture is implemented to the extent possible entirely
> > > > > > > in user-mode. The architecture intends to support multiple Target Device
> > > > > > > Types, SCSI Transport Types and Network Transport Types. The enclosed
> > > > > > > figure below illustrates the components of the architecture.
> > > > > > > 
> > > > > > > At the top of the figure are the different Target Device Type Drivers.
> > > > > > > These "drivers" are implemented in user-mode as libraries and plug into
> > > > > > > the Target Interface Layer. The Target Device Type drivers each support
> > > > > > > a particular class of device. For example, the SCSI Disk Driver supports
> > > > > > > SCSI disks, the Block Device driver supports generic block devices
> > > > > > > (e.g. /dev/md0, etc..) and the File Device Driver supports files as
> > > > > > > target devices. In most cases, the Target Device Type Drivers call
> > > > > > > existing system call interfaces to communicate with the actual target
> > > > > > > device, e.g. open, close, read, write, ioctl, etc... High performance
> > > > > > > implementations may use private kernel interfaces to improve
> > > > > > > performance. 
> > > > > > > 
> > > > > > > The Target Interface Layer implements a generic target device
> > > > > > > independent API called the Target Device API, and a SCSI transport
> > > > > > > independent API called the SCSI Transport API. This Target Interface
> > > > > > > Layer implements a target/SCSI transport switch that allows any Target
> > > > > > > Device Type to be associated with any SCSI Transport Type. 
> > > > > > > 
> > > > > > > The SCSI Transport Class Drivers implement support for the various SCSI
> > > > > > > transport types: SRP Transport implements the SCSI RDMA Protocol
> > > > > > > transport, FCP Transport implements the Fiber Channel transport type,
> > > > > > > and the iSCSI Transport implements the iSCSI transport type. These
> > > > > > > drivers sit between the Target Interface Layer and the Network Interface
> > > > > > > Layer. 
> > > > > > > 
> > > > > > > The Network Interface Layer implements a SCSI transport independent API
> > > > > > > called the Transport Class API and a network transport independent API
> > > > > > > called the Transport API. The Network Interface Layer allows a SCSI
> > > > > > > Transport Class driver to support multiple network transports. For
> > > > > > > example, the iSCSI Transport driver will support TCP, IB, and iWARP as
> > > > > > > network transports. The details of a particular SCSI Transport Class's
> > > > > > > device enumeration, login and management are implemented in the SCSI
> > > > > > > Transport Class driver (e.g. iSCSI Transport). The details of a
> > > > > > > particular network transport's connection management paradigm are
> > > > > > > implemented in the Transport Provider driver (e.g. RDMA driver).
> > > > > > > 
> > > > > > > The Transport Provider Drivers implement the Transport Provider API and
> > > > > > > provide core network I/O services to the Network Interface Layer. The
> > > > > > > Transport API is a transport independent interface for creating
> > > > > > > endpoints, service points, accepting incoming connection requests and
> > > > > > > performing I/O on an endpoint.
> > > > > > > 
> > > > > > > The Management Agent interfaces with the Target Interface Layer and
> > > > > > > performs management functions such as creating targets, devices, loading
> > > > > > > and storing persistent configurations and other management related
> > > > > > > functions.
> > > > > > > 
> > > > > > > The various API referred to above are basically simplified versions of
> > > > > > > the existing scsi_transport_template, scsi_host, scsi_host_template
> > > > > > > interfaces, etc... from the current kernel implementation. The
> > > > > > > interfaces between the various components, however, can be reduced to
> > > > > > > function calls since everything resides user mode. 
> > > > > > > 
> > > > > > > I think the only tough issue here is with copy avoidance for the network
> > > > > > > user/kernel interface and target device user/kernel interfaces.
> > > > > > > Initially, these could be prototyped without regard to this issue and
> > > > > > > see what kind of performance we could get. The RDMA network transports
> > > > > > > already provide copy avoidance, however, TCP/FC would require some
> > > > > > > cleverness.
> > > > > > > 
> > > > > > > Thoughts?
> > > > > > > 
> > > > > > > _______________________________________________
> > > > > > > Stgt-devel mailing list
> > > > > > > Stgt-devel at lists.berlios.de
> > > > > > > http://bat.berlios.de/mailman/listinfo/stgt-devel
> > > > > > 
> > > > > > _______________________________________________
> > > > > > Stgt-devel mailing list
> > > > > > Stgt-devel at lists.berlios.de
> > > > > > http://bat.berlios.de/mailman/listinfo/stgt-devel
> > > > > 
> > > > > _______________________________________________
> > > > > Stgt-devel mailing list
> > > > > Stgt-devel at lists.berlios.de
> > > > > http://bat.berlios.de/mailman/listinfo/stgt-devel
> > > > 
> > > _______________________________________________
> > > Stgt-devel mailing list
> > > Stgt-devel at lists.berlios.de
> > > http://bat.berlios.de/mailman/listinfo/stgt-devel
> > 
> 
> _______________________________________________
> Stgt-devel mailing list
> Stgt-devel at lists.berlios.de
> http://bat.berlios.de/mailman/listinfo/stgt-devel