[Stgt-devel] User-mode iSER

[FUJITA Tomonori]

From: "Dan Bar Dov" <danb at voltaire.com>
Subject: RE: [Stgt-devel] User-mode iSER
Date: Sun, 6 Aug 2006 11:44:31 +0300

>  
> 
> > -----Original Message-----
> > From: stgt-devel-bounces at lists.berlios.de 
> > [mailto:stgt-devel-bounces at lists.berlios.de] On Behalf Of 
> > FUJITA Tomonori
> > Sent: Sunday, August 06, 2006 11:05 AM
> > To: Dan Bar Dov
> > Cc: fujita.tomonori at lab.ntt.co.jp; mingz at ele.uri.edu; 
> > stgt-devel at lists.berlios.de; tom at opengridcomputing.com
> > Subject: Re: [Stgt-devel] User-mode iSER
> > 
> > From: "Dan Bar Dov" <danb at voltaire.com>
> > Subject: Re: [Stgt-devel] User-mode iSER
> > Date: Sun, 6 Aug 2006 10:59:36 +0300
> > 
> > >  
> > > 
> > > > -----Original Message-----
> > > > From: stgt-devel-bounces at lists.berlios.de 
> > > > [mailto:stgt-devel-bounces at lists.berlios.de] On Behalf Of 
> > > > FUJITA Tomonori
> > > > Sent: Thursday, August 03, 2006 6:20 PM
> > > > To: Dan Bar Dov
> > > > Cc: mingz at ele.uri.edu; fujita.tomonori at lab.ntt.co.jp; 
> > > > stgt-devel at lists.berlios.de; tom at opengridcomputing.com
> > > > Subject: Re: [Stgt-devel] User-mode iSER
> > > > 
> > > > From: "Dan Bar Dov" <danb at voltaire.com>
> > > > Subject: Re: [Stgt-devel] User-mode iSER
> > > > Date: Thu, 3 Aug 2006 11:34:33 +0300
> > > > 
> > > > >  
> > > > > 
> > > > > > -----Original Message-----
> > > > > > From: stgt-devel-bounces at lists.berlios.de 
> > > > > > [mailto:stgt-devel-bounces at lists.berlios.de] On Behalf Of 
> > > > Ming Zhang
> > > > > > Sent: Wednesday, August 02, 2006 5:20 PM
> > > > > > To: Tom Tucker
> > > > > > Cc: FUJITA Tomonori; stgt-devel at lists.berlios.de
> > > > > > Subject: Re: [Stgt-devel] User-mode iSER
> > > > > > 
> > > > > > On Wed, 2006-08-02 at 09:01 -0500, Tom Tucker wrote:
> > > > > > > [...snip...]
> > > > > > > > > 
> > > > > > > > > I think this is the area where we will need to get 
> > > > > > fancy if we want
> > > > > > > > > higher performance. To avoid the copy, we would have to 
> > > > > > migrate to
> > > > > > > > > netchannels (if they every happen) or implement our own 
> > > > > > simple tear-away
> > > > > > > > > buffer scheme on top of a socket. I think this is 
> > > > > > phase-ii, however. 
> > > > > > > > 
> > > > > > > > ok, otherwise copy to user space and copy back to kernel 
> > > > > > for disk = low
> > > > > > > > performance. yes, direct io can be used here, but then u 
> > > > > > lose whole
> > > > > > > > cache benefits.
> > > > > > > 
> > > > > > > Can you elaborate on the loss of "whole cache benefits". 
> > > > > > 
> > > > > > if you use a Linux box as a storage server, it will be 
> > > > desired to use
> > > > > > page cache as storage cache. though this will bring 
> > data integrity
> > > > > > issues, but so many people still want to have it for specific
> > > > > > applications.
> > > > > > 
> > > > > 
> > > > > I don't think page cache as storage cache makes sense. 
> > If you look
> > > > > at the networked storage, you have caching on the initiator 
> > > > (client) side
> > > > > and you have caching on the target (storage server) 
> > side. Storage
> > > > > servers usually have write-behind cache, and the better 
> > ones, have
> > > > > it battery backed up for data integrity's sake. Storage 
> > > > admins know to
> > > > > shutd down write behind caching if it is not backed up.
> > > > 
> > > > Modern operating systems and applications (like file 
> > systems) does not
> > > > need help from battery-backed memory to enjoy 
> > write-behind cache on
> > > > SAN target devices for better performance without data corruption
> > > > risks. So page cache is always useful.
> > > 
> > > Since we are working on the SAN target, my point is we need 
> > to know if
> > > this serving target is write-behind or not. If our code 
> > relies on page cache, 
> > > it can not operate in any mode except write behind, however, this is
> > > an implicit write behind. We could provide a faster 
> > response with an explicit 
> > > write behind.
> > 
> > Sorry, I'm not sure what you mean. What does 'an implicit write
> > behind' mean?
>
> It means that the target sends a response because from the target's
> code point of view it wrote the data to storage, but since you use
> page cache, the data didn't actually get to storage.  The target
> code thinks it did a write- through, but it actually got a write
> behind.

Why does the target code think that it did a write-though?

tgt thinks that it does a write-back. The initiators can use
SYNCHRONIZE_CACHE, ordering write, etc for data integrity. And when
the target code is asked to write the data to storage permanently, we
can use fsync, msync, etc.


> An explicit write-behind would let the target send a response
> *before* it wrote the data anywhere (even buffer cache). This
> explicit write behind is therefore faster.