[Sheepdog] [PATCH] object cache: enable direct IO for cache object

[MORITA Kazutaka]

At Mon, 02 Apr 2012 01:04:15 +0800,
Liu Yuan wrote:
> 
> On 04/01/2012 08:56 PM, MORITA Kazutaka wrote:
> > At Sun,  1 Apr 2012 19:51:41 +0800, Liu Yuan wrote:
> >> 
> >> From: Liu Yuan <tailai.ly at taobao.com>
> >> 
> >> When sheep is launched with '-D' or '--directio' option, we will
> >> use direct IO for cache object too.
> > 
> > If you don't want to use a page cache for sheepdog object caches,
> > how about making O_DIRECT default?  I think you know, but O_DSYNC
> > doesn't means that sheep doesn't use a page cache.
> > 
> 
> Yes, but later I thought if it is better to let user choose whether
> use page cache or not. And I notice that our backend store uses O_DSYNC
> as default, so I use it too to follow the convention.
> 
> So the point is, is it good to use uniform IO flags for all the IO path
> (since our sheep cluster appear as a whole to be working as virtual disk
> for Guest)?
> To be frankly speaking, I am uncertain of it.

It is not good to use the same flag blindly.  O_DSYNC for normal I/Os
is mandatory to ensure that data is completely written to disks.  But,
as I said before, I think we don't need a sync flag for a cache.

> >> 
> >> Signed-off-by: Liu Yuan <tailai.ly at taobao.com> --- 
> >> sheep/object_cache.c |   18 ++++++++++++------ sheep/store.c
> >> |   28 +++++++--------------------- 2 files changed, 19
> >> insertions(+), 27 deletions(-)
> > 
> > This patch uses (O_DIRECT|O_DSYNC) for data object cache I/Os. Is
> > there any reason you want to add O_DSYNC to O_DIRECT?  As I said 
> > before, we don't need to flush cache data to disks.
> > 
> > In addition, I'd like to remove a O_DSYNC flag from vdi object cache 
> > I/Os.  It would boosts the performance of metadata updates.
> > 
> > So my suggestion is: - (O_DIRECT|O_RDWR) for data object cache I/Os -
> > O_RDWR for vdi object cache I/Os
> >
> 
> Looks sensible to me, if we can use different scheme for object cache and
> backend store.

If you don't mind disabling O_DSYNC for a object cache, please support
following flags:

 * for users who run VMs in Sheepdog cluster

   - (O_DIRECT|O_DSYNC|O_RDWR) for backend stores
   - (O_DIRECT|O_RDWR) for object caches

   It is because we want to use a host memory for guest VMs.

 * for users who run VMs outside Sheepdog cluster

   - (O_DIRECT|O_DSYNC|O_RDWR) for backend stores
   - only O_RDWR for object caches

   It is because we can use a memory on the storage node for disk
   caches of Sheepdog virtual disks.  If you don't mind, I'd like to
   make this option default since it shows the best storage
   performance.


Thanks,

Kazutaka