On Fri, Jun 25, 2010 at 9:21 AM, FUJITA Tomonori <fujita.tomonori at lab.ntt.co.jp> wrote: > > On Thu, 24 Jun 2010 21:34:52 +0300 > Alexander Nezhinsky <alexandern at Voltaire.COM> wrote: > > > The next objective was to avoid memory copies, so that i stopped using > > req/rsp buffers present within iscsi connection struct. There were some > > other changes so i introduced a separate iser task structure, and a common > > iscsi connection header (both tcp and iser extend it). > > You mean that you send page cache to initiators (like sendpage in the > case of tcp)? No, all data buffers in iser are RDMA-registered and submitted to backing-store in hope that it supports direct-I/O (or if it does not, then it is for a reason, e.g. copying to the page cache is faster than real i/o by order of magnitude) A small note about buffers: First-burst of iscsi is mapped to sent buffers, subsequent bursts are mapped to rdma transfers. Because we can't know beforehand which buffer will contain the sent data of the first burst, we can't use any specific connection-related buffer (which was acceptable with tcp, as it is stream-based, and we are controlling the request timing and extent of the buffer filling). So if we want to avoid copies, we have to associate arbitrary posted rx buffers with iser tasks. RDMA buffers are coming from a separate pool, so this brings us to a new "multi-buffer per scsi cmd" reality - which is not supported yet by tgt. Meanwhile, i stipulated InitialR2T=Yes, ImmediateData=No mode, so that i work with RDMA transfers only. Alternatively i can enable immediate data and somehow trick it into getting everything with one pdu. I wrote the code with all preparations towards multi-buffer cmd support, but obviously it is not working yet. > Your changes work nicely too for tcp? No - but if you adopt the dissociation of the buffers from the connection structure, then probably these buffers could be page-cache-derived? > With a separate iser lld, if a tcp-only initiator tries to connect to > a iser target, what happens? tcp and iser can't share iSCSI listen > port (3260), so we can't run tcp and iser targets at the same time? > we can, only you have to define 2 separate targets, one iser, another issci/tcp i mark the targets and check the type after connection establishment. > > I propose starting with the "rectification" of iscsi text functions, by > > changing them to accept plain char * buffers, making them independent from > > iscsi structures specifics. Such semantic changes do not alter the logic > > of the existing code. For this I can send some patches shortly. > > Sounds reasonable but as I wrote above, iser and tcp could have the > common buffer management. need to discuss this - as described above, this is not straightforward > Please send all the patches that you have now (no need to clean up > them). I just want to see the whole picture first. ok, i am going to send this in a couple of days > Btw, your new implementation uses the single thread model? Or as I did > for tcp, it uses pthread per target? it's single thread yet -- To unsubscribe from this list: send the line "unsubscribe stgt" in the body of a message to majordomo at vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html |