[Stgt-devel] User-mode iSER

[Tom Tucker]

On Tue, 2006-08-01 at 17:24 -0400, Ming Zhang wrote:
> On Tue, 2006-08-01 at 15:43 -0500, Tom Tucker wrote:
> > On Tue, 2006-08-01 at 16:21 -0400, Ming Zhang wrote:
> > > On Tue, 2006-08-01 at 14:00 -0500, Tom Tucker wrote:
> > > > On Tue, 2006-08-01 at 14:49 -0400, Ming Zhang wrote:
> > > > > my 2c. u figure ignores other device types like VT/VTL or bridge. your
> > > > > target device type are only TYPE_DISK here.
> > > > 
> > > > Yeah, I should put in ellipses... These types are not precluded.
> > > > 
> > > > >  also scsi/block/file just
> > > > > physical media used by TYPE_DISK.
> > > > 
> > > > Yes, but there seemed to be some interest in exporting a file as a SCSI
> > > > Target, or even anonymously mapped memory...
> > > > 
> > > > What else should we be thinking about?
> > > 
> > > SPI is not mentioned in your pic.
> > 
> > Hmm. Doesn't this sit on the target device side underneath /dev/sdN?
> 
> sorry but i mean SPI transport, or interface with INI.
> 
> > 
> > > 
> > > and it will be better if kernel/user space line is marked. so people
> > > know your intention and can comment in that if need. otherwise, i saw no
> > > difference from stgt or scst.
> > 
> > Attaches is a new picture with kernel/user boundaries. I think the
> > boundary is somewhat fuzzy at the top and bottom due to the fact that
> > the API is different for each target type and for each network provider
> > type. 
> > 
> > For example, the sd driver will use open /dev/sdN and submit ioctl to
> > read/write the disk, whereas the file target will
> > open("/home/thisandthat", ...) and submit lseek/read/write to read/write
> > the file.
> > 
> > On the network side, the TCP provider would use socket, listen, accept,
> > send, recv (at least initially), the RMDA provider would use
> > rdma_resolve_addr, rdma_resolve_route, rdma_listen, rdma_accept,
> > rdma_create_qp, rdma_post_send, etc....
> > 
> > My only point is that the kernel/user interface is not a straight line
> > since some of these interfaces are higher level than others.
> 
> so you have even all transports in user space? zero copy socket is not
> easy. 

I think this is the area where we will need to get fancy if we want
higher performance. To avoid the copy, we would have to migrate to
netchannels (if they every happen) or implement our own simple tear-away
buffer scheme on top of a socket. I think this is phase-ii, however. 

How about on the target device side? How expensive/inefficient is the
ioctl interface to /dev/sd?

> and can FCP drivers, like a target mode driver for a QLA or LSI
> HBA can be in user space as well?

No. These will be in the kernel, but I'm not considering these drivers
as part of this subsystem. Are they part of IET?

> 
> ming
> 
> 
> > 
> > 
> > 
> > > 
> > > also license. :P what license(s) will be used and allowed for each part.
> > > 
> > > ming
> > > 
> > > 
> > > 
> > > > 
> > > > > 
> > > > > Ming
> > > > > 
> > > > > 
> > > > > On Tue, 2006-08-01 at 13:42 -0500, Tom Tucker wrote:
> > > > > > What do people think about something like this...
> > > > > > 
> > > > > > The target architecture is implemented to the extent possible entirely
> > > > > > in user-mode. The architecture intends to support multiple Target Device
> > > > > > Types, SCSI Transport Types and Network Transport Types. The enclosed
> > > > > > figure below illustrates the components of the architecture.
> > > > > > 
> > > > > > At the top of the figure are the different Target Device Type Drivers.
> > > > > > These "drivers" are implemented in user-mode as libraries and plug into
> > > > > > the Target Interface Layer. The Target Device Type drivers each support
> > > > > > a particular class of device. For example, the SCSI Disk Driver supports
> > > > > > SCSI disks, the Block Device driver supports generic block devices
> > > > > > (e.g. /dev/md0, etc..) and the File Device Driver supports files as
> > > > > > target devices. In most cases, the Target Device Type Drivers call
> > > > > > existing system call interfaces to communicate with the actual target
> > > > > > device, e.g. open, close, read, write, ioctl, etc... High performance
> > > > > > implementations may use private kernel interfaces to improve
> > > > > > performance. 
> > > > > > 
> > > > > > The Target Interface Layer implements a generic target device
> > > > > > independent API called the Target Device API, and a SCSI transport
> > > > > > independent API called the SCSI Transport API. This Target Interface
> > > > > > Layer implements a target/SCSI transport switch that allows any Target
> > > > > > Device Type to be associated with any SCSI Transport Type. 
> > > > > > 
> > > > > > The SCSI Transport Class Drivers implement support for the various SCSI
> > > > > > transport types: SRP Transport implements the SCSI RDMA Protocol
> > > > > > transport, FCP Transport implements the Fiber Channel transport type,
> > > > > > and the iSCSI Transport implements the iSCSI transport type. These
> > > > > > drivers sit between the Target Interface Layer and the Network Interface
> > > > > > Layer. 
> > > > > > 
> > > > > > The Network Interface Layer implements a SCSI transport independent API
> > > > > > called the Transport Class API and a network transport independent API
> > > > > > called the Transport API. The Network Interface Layer allows a SCSI
> > > > > > Transport Class driver to support multiple network transports. For
> > > > > > example, the iSCSI Transport driver will support TCP, IB, and iWARP as
> > > > > > network transports. The details of a particular SCSI Transport Class's
> > > > > > device enumeration, login and management are implemented in the SCSI
> > > > > > Transport Class driver (e.g. iSCSI Transport). The details of a
> > > > > > particular network transport's connection management paradigm are
> > > > > > implemented in the Transport Provider driver (e.g. RDMA driver).
> > > > > > 
> > > > > > The Transport Provider Drivers implement the Transport Provider API and
> > > > > > provide core network I/O services to the Network Interface Layer. The
> > > > > > Transport API is a transport independent interface for creating
> > > > > > endpoints, service points, accepting incoming connection requests and
> > > > > > performing I/O on an endpoint.
> > > > > > 
> > > > > > The Management Agent interfaces with the Target Interface Layer and
> > > > > > performs management functions such as creating targets, devices, loading
> > > > > > and storing persistent configurations and other management related
> > > > > > functions.
> > > > > > 
> > > > > > The various API referred to above are basically simplified versions of
> > > > > > the existing scsi_transport_template, scsi_host, scsi_host_template
> > > > > > interfaces, etc... from the current kernel implementation. The
> > > > > > interfaces between the various components, however, can be reduced to
> > > > > > function calls since everything resides user mode. 
> > > > > > 
> > > > > > I think the only tough issue here is with copy avoidance for the network
> > > > > > user/kernel interface and target device user/kernel interfaces.
> > > > > > Initially, these could be prototyped without regard to this issue and
> > > > > > see what kind of performance we could get. The RDMA network transports
> > > > > > already provide copy avoidance, however, TCP/FC would require some
> > > > > > cleverness.
> > > > > > 
> > > > > > Thoughts?
> > > > > > 
> > > > > > _______________________________________________
> > > > > > Stgt-devel mailing list
> > > > > > Stgt-devel at lists.berlios.de
> > > > > > http://bat.berlios.de/mailman/listinfo/stgt-devel
> > > > > 
> > > > > _______________________________________________
> > > > > Stgt-devel mailing list
> > > > > Stgt-devel at lists.berlios.de
> > > > > http://bat.berlios.de/mailman/listinfo/stgt-devel
> > > > 
> > > > _______________________________________________
> > > > Stgt-devel mailing list
> > > > Stgt-devel at lists.berlios.de
> > > > http://bat.berlios.de/mailman/listinfo/stgt-devel
> > > 
> > _______________________________________________
> > Stgt-devel mailing list
> > Stgt-devel at lists.berlios.de
> > http://bat.berlios.de/mailman/listinfo/stgt-devel
>