[Stgt-devel] User-mode iSER

[Dan Bar Dov]

> -----Original Message-----
> From: FUJITA Tomonori [mailto:fujita.tomonori at lab.ntt.co.jp] 
> Sent: Sunday, August 06, 2006 11:58 AM
> To: Dan Bar Dov
> Cc: fujita.tomonori at lab.ntt.co.jp; mingz at ele.uri.edu; 
> stgt-devel at lists.berlios.de; tom at opengridcomputing.com
> Subject: RE: [Stgt-devel] User-mode iSER
> 
> From: "Dan Bar Dov" <danb at voltaire.com>
> Subject: RE: [Stgt-devel] User-mode iSER
> Date: Sun, 6 Aug 2006 11:44:31 +0300
> 
> >  
> > 
> > > -----Original Message-----
> > > From: stgt-devel-bounces at lists.berlios.de 
> > > [mailto:stgt-devel-bounces at lists.berlios.de] On Behalf Of 
> > > FUJITA Tomonori
> > > Sent: Sunday, August 06, 2006 11:05 AM
> > > To: Dan Bar Dov
> > > Cc: fujita.tomonori at lab.ntt.co.jp; mingz at ele.uri.edu; 
> > > stgt-devel at lists.berlios.de; tom at opengridcomputing.com
> > > Subject: Re: [Stgt-devel] User-mode iSER
> > > 
> > > From: "Dan Bar Dov" <danb at voltaire.com>
> > > Subject: Re: [Stgt-devel] User-mode iSER
> > > Date: Sun, 6 Aug 2006 10:59:36 +0300
> > > 
> > > >  
> > > > 
> > > > > -----Original Message-----
> > > > > From: stgt-devel-bounces at lists.berlios.de 
> > > > > [mailto:stgt-devel-bounces at lists.berlios.de] On Behalf Of 
> > > > > FUJITA Tomonori
> > > > > Sent: Thursday, August 03, 2006 6:20 PM
> > > > > To: Dan Bar Dov
> > > > > Cc: mingz at ele.uri.edu; fujita.tomonori at lab.ntt.co.jp; 
> > > > > stgt-devel at lists.berlios.de; tom at opengridcomputing.com
> > > > > Subject: Re: [Stgt-devel] User-mode iSER
> > > > > 
> > > > > From: "Dan Bar Dov" <danb at voltaire.com>
> > > > > Subject: Re: [Stgt-devel] User-mode iSER
> > > > > Date: Thu, 3 Aug 2006 11:34:33 +0300
> > > > > 
> > > > > >  
> > > > > > 
> > > > > > > -----Original Message-----
> > > > > > > From: stgt-devel-bounces at lists.berlios.de 
> > > > > > > [mailto:stgt-devel-bounces at lists.berlios.de] On Behalf Of 
> > > > > Ming Zhang
> > > > > > > Sent: Wednesday, August 02, 2006 5:20 PM
> > > > > > > To: Tom Tucker
> > > > > > > Cc: FUJITA Tomonori; stgt-devel at lists.berlios.de
> > > > > > > Subject: Re: [Stgt-devel] User-mode iSER
> > > > > > > 
> > > > > > > On Wed, 2006-08-02 at 09:01 -0500, Tom Tucker wrote:
> > > > > > > > [...snip...]
> > > > > > > > > > 
> > > > > > > > > > I think this is the area where we will need to get 
> > > > > > > fancy if we want
> > > > > > > > > > higher performance. To avoid the copy, we 
> would have to 
> > > > > > > migrate to
> > > > > > > > > > netchannels (if they every happen) or 
> implement our own 
> > > > > > > simple tear-away
> > > > > > > > > > buffer scheme on top of a socket. I think this is 
> > > > > > > phase-ii, however. 
> > > > > > > > > 
> > > > > > > > > ok, otherwise copy to user space and copy 
> back to kernel 
> > > > > > > for disk = low
> > > > > > > > > performance. yes, direct io can be used here, 
> but then u 
> > > > > > > lose whole
> > > > > > > > > cache benefits.
> > > > > > > > 
> > > > > > > > Can you elaborate on the loss of "whole cache 
> benefits". 
> > > > > > > 
> > > > > > > if you use a Linux box as a storage server, it will be 
> > > > > desired to use
> > > > > > > page cache as storage cache. though this will bring 
> > > data integrity
> > > > > > > issues, but so many people still want to have it 
> for specific
> > > > > > > applications.
> > > > > > > 
> > > > > > 
> > > > > > I don't think page cache as storage cache makes sense. 
> > > If you look
> > > > > > at the networked storage, you have caching on the initiator 
> > > > > (client) side
> > > > > > and you have caching on the target (storage server) 
> > > side. Storage
> > > > > > servers usually have write-behind cache, and the better 
> > > ones, have
> > > > > > it battery backed up for data integrity's sake. Storage 
> > > > > admins know to
> > > > > > shutd down write behind caching if it is not backed up.
> > > > > 
> > > > > Modern operating systems and applications (like file 
> > > systems) does not
> > > > > need help from battery-backed memory to enjoy 
> > > write-behind cache on
> > > > > SAN target devices for better performance without 
> data corruption
> > > > > risks. So page cache is always useful.
> > > > 
> > > > Since we are working on the SAN target, my point is we need 
> > > to know if
> > > > this serving target is write-behind or not. If our code 
> > > relies on page cache, 
> > > > it can not operate in any mode except write behind, 
> however, this is
> > > > an implicit write behind. We could provide a faster 
> > > response with an explicit 
> > > > write behind.
> > > 
> > > Sorry, I'm not sure what you mean. What does 'an implicit write
> > > behind' mean?
> >
> > It means that the target sends a response because from the target's
> > code point of view it wrote the data to storage, but since you use
> > page cache, the data didn't actually get to storage.  The target
> > code thinks it did a write- through, but it actually got a write
> > behind.
> 
> Why does the target code think that it did a write-though?
> 
> tgt thinks that it does a write-back. The initiators can use
> SYNCHRONIZE_CACHE, ordering write, etc for data integrity. And when
> the target code is asked to write the data to storage permanently, we
> can use fsync, msync, etc.

That would work.

Dan

> 
> 
> > An explicit write-behind would let the target send a response
> > *before* it wrote the data anywhere (even buffer cache). This
> > explicit write behind is therefore faster.
>