[sheepdog] [PATCH 00/12] introduce basic NFS server support

[Hitoshi Mitake]

At Wed, 29 Jan 2014 14:17:27 +0800,
Liu Yuan wrote:
> 
> On Wed, Jan 29, 2014 at 11:41:59AM +0900, Hitoshi Mitake wrote:
> > At Wed, 29 Jan 2014 04:19:00 +0800,
> > Liu Yuan wrote:
> > > 
> > > This patch set mainly introduce NFS server building blocks, that is
> > > 
> > > - NFS v3 transport by glibc's SUNRPC
> > > - preliminary NFS server
> > > 
> > > NFS protocol is seen as the most established shared storage protocol to provide
> > > POSIX file semantics and has rich client and server features. It was introduced
> > > to partition the UNIX file system into two parts, client that deals with APP and
> > > server side that deals with disks. NFS has a tremendous success both in system
> > > design (where Linux VFS root from, allowing different backend implementation
> > > such as ext4, xfs, btrfs, FUSE, etc.) and in business market. It is well received
> > > by uesrs, so there is no reason we don't start from NFS for a POSIX file
> > > abstraction (tree structure, rich system calls, ACL, etc)
> > > 
> > > I choose hyper volume as the base for NFS sever like our HTTP storage system.
> > > 
> > > For now, this patch set limit
> > > 
> > > - file size to 4M, hence directory is also 4M at most
> > > - inode take up the whole sd object (4M)
> > > - no file deletion and directory deletion support due to lack of dentry management
> > > - no link, symlink, rename support
> > > - no ACL
> > > - no nanosecond support
> > > 
> > > But we now support traditional UNIX file system structure and it is a working
> > > grund to play with. It even in this early stage support bash command like 'echo,
> > > cat, mkdir, touch, df ...'.
> > > 
> > > Usage:
> > >  # on host
> > >  $ ./configure --enable-nfs
> > >  # work around auth error for mount
> > >  $ echo 'OPTIONS="-w -i"' | sudo tee /etc/default/rpcbind
> > >  $ sudo service portmap restart
> > >  $ dog nfs create test # create a NFS v3 server (listen on nfsd and mountd port)
> > > 
> > > --------------------------------------------------------------------
> > >  # on client
> > >  # now on any client machine, make sure you install nfs-common at first
> > >  $ sudo mount o tcp,nfsvers=3 -t nfs server_ip:test /mnt
> > > 
> > > TODO:
> > >  - finish stubs
> > >  - add stat sub-command to 'dog nfs'
> > >  - add extent to file allocation
> > >  - add btree or hash based kv store to manage dentries
> > >  - implement a multi-threaded SUNRPC to take place of poor performance glibc RPC
> > >  - implement NFS v4
> > > 
> > > Liu Yuan (12):
> > >   sheep: fix some compile errors
> > >   sheep: introduce NFSv3 transport
> > >   sheep/nfs: add mount protocol stub
> > >   dog: add nfs command
> > >   util: add PTR_ERR helpers
> > >   sheep/nfs: add basic file system framework
> > >   sheep: clean up sd_inode btree error code
> > >   sheep/nfs: support lookup/create/setattr nfs operation
> > >   sheep/nfs: implement read/write nfs operation
> > >   sheep/nfs: add mkdir support
> > >   sheep: make stat_data_objs a generic function
> > >   sheep/nfs: implement nfs3_fsstat operation
> > > 
> > >  configure.ac             |   12 +
> > >  dog/Makefile.am          |    4 +
> > >  dog/dog.c                |    5 +
> > >  dog/dog.h                |    8 +-
> > >  dog/nfs.c                |   81 +++
> > >  dog/vdi.c                |   96 +--
> > >  include/compiler.h       |    2 +
> > >  include/internal_proto.h |    6 +
> > >  include/sheepdog_proto.h |    3 -
> > >  include/util.h           |   44 +-
> > >  lib/Makefile.am          |    3 +-
> > >  lib/sd.c                 |  112 +++
> > >  lib/sd_inode.c           |   22 +-
> > >  sheep/Makefile.am        |    6 +
> > >  sheep/http/http.h        |    2 +-
> > >  sheep/http/kv.c          |   35 +-
> > >  sheep/nfs/fs.c           |  379 ++++++++++
> > >  sheep/nfs/fs.h           |   73 ++
> > >  sheep/nfs/mount.c        |   80 +++
> > >  sheep/nfs/nfs.c          |  723 +++++++++++++++++++
> > >  sheep/nfs/nfs.h          | 1241 ++++++++++++++++++++++++++++++++
> > >  sheep/nfs/nfsd.c         |  245 +++++++
> > >  sheep/nfs/xdr.c          | 1795 ++++++++++++++++++++++++++++++++++++++++++++++
> > >  sheep/ops.c              |   23 +
> > >  sheep/sheep.c            |    4 +
> > >  sheep/sheep_priv.h       |   14 +-
> > >  sheep/trace/trace.h      |    8 -
> > >  sheep/vdi.c              |   30 +
> > >  28 files changed, 4897 insertions(+), 159 deletions(-)
> > >  create mode 100644 dog/nfs.c
> > >  create mode 100644 lib/sd.c
> > >  create mode 100644 sheep/nfs/fs.c
> > >  create mode 100644 sheep/nfs/fs.h
> > >  create mode 100644 sheep/nfs/mount.c
> > >  create mode 100644 sheep/nfs/nfs.c
> > >  create mode 100644 sheep/nfs/nfs.h
> > >  create mode 100644 sheep/nfs/nfsd.c
> > >  create mode 100644 sheep/nfs/xdr.c
> > 
> > I commented on 4th, 9th, and 11th patches and currently don't have
> > other objections on this patchset.
> > 
> > (To be honest, I have an objection that svc_dispatcher() should
> > delegate its procedures to main thread via register_event() and
> > eventfd for ease of maintenance. But current patchset doesn't affect
> > stability of sheepdog so it can be fixed later.)
> 
> I'm afraid nfs transport is much more complex than register a handler with a simple parameter
> like zk_watcher. There are thansands of lines to set up RPC service and very
> complex parameter and reply marshalling and unmarshalling. I guess we need severl thansoud
> lines to implement a svc_dispatcher like what you expect. Even we have such one,
> I don't think it is good to couple nfs server with sheep main loop because our
> main loop is very likely to be bottleneck and no obvious benefit from it.
> 
> Anyway, current svc_xxx from glibc is very low performance (single threaded),
> we need later implement our own sunrpc framework like linux kernel or glusterfs
> for pratical use of nfs server over sheepdog. Probably when we do it, we can
> make some use of our main loop infrastructure.

Yes, I think your current implementation is okay as a temporal thing.

Thanks,
Hitoshi