[sheepdog] [PATCH 00/12] introduce basic NFS server support

[Liu Yuan]

On Wed, Jan 29, 2014 at 11:41:59AM +0900, Hitoshi Mitake wrote:
> At Wed, 29 Jan 2014 04:19:00 +0800,
> Liu Yuan wrote:
> > 
> > This patch set mainly introduce NFS server building blocks, that is
> > 
> > - NFS v3 transport by glibc's SUNRPC
> > - preliminary NFS server
> > 
> > NFS protocol is seen as the most established shared storage protocol to provide
> > POSIX file semantics and has rich client and server features. It was introduced
> > to partition the UNIX file system into two parts, client that deals with APP and
> > server side that deals with disks. NFS has a tremendous success both in system
> > design (where Linux VFS root from, allowing different backend implementation
> > such as ext4, xfs, btrfs, FUSE, etc.) and in business market. It is well received
> > by uesrs, so there is no reason we don't start from NFS for a POSIX file
> > abstraction (tree structure, rich system calls, ACL, etc)
> > 
> > I choose hyper volume as the base for NFS sever like our HTTP storage system.
> > 
> > For now, this patch set limit
> > 
> > - file size to 4M, hence directory is also 4M at most
> > - inode take up the whole sd object (4M)
> > - no file deletion and directory deletion support due to lack of dentry management
> > - no link, symlink, rename support
> > - no ACL
> > - no nanosecond support
> > 
> > But we now support traditional UNIX file system structure and it is a working
> > grund to play with. It even in this early stage support bash command like 'echo,
> > cat, mkdir, touch, df ...'.
> > 
> > Usage:
> >  # on host
> >  $ ./configure --enable-nfs
> >  # work around auth error for mount
> >  $ echo 'OPTIONS="-w -i"' | sudo tee /etc/default/rpcbind
> >  $ sudo service portmap restart
> >  $ dog nfs create test # create a NFS v3 server (listen on nfsd and mountd port)
> > 
> > --------------------------------------------------------------------
> >  # on client
> >  # now on any client machine, make sure you install nfs-common at first
> >  $ sudo mount o tcp,nfsvers=3 -t nfs server_ip:test /mnt
> > 
> > TODO:
> >  - finish stubs
> >  - add stat sub-command to 'dog nfs'
> >  - add extent to file allocation
> >  - add btree or hash based kv store to manage dentries
> >  - implement a multi-threaded SUNRPC to take place of poor performance glibc RPC
> >  - implement NFS v4
> > 
> > Liu Yuan (12):
> >   sheep: fix some compile errors
> >   sheep: introduce NFSv3 transport
> >   sheep/nfs: add mount protocol stub
> >   dog: add nfs command
> >   util: add PTR_ERR helpers
> >   sheep/nfs: add basic file system framework
> >   sheep: clean up sd_inode btree error code
> >   sheep/nfs: support lookup/create/setattr nfs operation
> >   sheep/nfs: implement read/write nfs operation
> >   sheep/nfs: add mkdir support
> >   sheep: make stat_data_objs a generic function
> >   sheep/nfs: implement nfs3_fsstat operation
> > 
> >  configure.ac             |   12 +
> >  dog/Makefile.am          |    4 +
> >  dog/dog.c                |    5 +
> >  dog/dog.h                |    8 +-
> >  dog/nfs.c                |   81 +++
> >  dog/vdi.c                |   96 +--
> >  include/compiler.h       |    2 +
> >  include/internal_proto.h |    6 +
> >  include/sheepdog_proto.h |    3 -
> >  include/util.h           |   44 +-
> >  lib/Makefile.am          |    3 +-
> >  lib/sd.c                 |  112 +++
> >  lib/sd_inode.c           |   22 +-
> >  sheep/Makefile.am        |    6 +
> >  sheep/http/http.h        |    2 +-
> >  sheep/http/kv.c          |   35 +-
> >  sheep/nfs/fs.c           |  379 ++++++++++
> >  sheep/nfs/fs.h           |   73 ++
> >  sheep/nfs/mount.c        |   80 +++
> >  sheep/nfs/nfs.c          |  723 +++++++++++++++++++
> >  sheep/nfs/nfs.h          | 1241 ++++++++++++++++++++++++++++++++
> >  sheep/nfs/nfsd.c         |  245 +++++++
> >  sheep/nfs/xdr.c          | 1795 ++++++++++++++++++++++++++++++++++++++++++++++
> >  sheep/ops.c              |   23 +
> >  sheep/sheep.c            |    4 +
> >  sheep/sheep_priv.h       |   14 +-
> >  sheep/trace/trace.h      |    8 -
> >  sheep/vdi.c              |   30 +
> >  28 files changed, 4897 insertions(+), 159 deletions(-)
> >  create mode 100644 dog/nfs.c
> >  create mode 100644 lib/sd.c
> >  create mode 100644 sheep/nfs/fs.c
> >  create mode 100644 sheep/nfs/fs.h
> >  create mode 100644 sheep/nfs/mount.c
> >  create mode 100644 sheep/nfs/nfs.c
> >  create mode 100644 sheep/nfs/nfs.h
> >  create mode 100644 sheep/nfs/nfsd.c
> >  create mode 100644 sheep/nfs/xdr.c
> 
> I commented on 4th, 9th, and 11th patches and currently don't have
> other objections on this patchset.
> 
> (To be honest, I have an objection that svc_dispatcher() should
> delegate its procedures to main thread via register_event() and
> eventfd for ease of maintenance. But current patchset doesn't affect
> stability of sheepdog so it can be fixed later.)

I'm afraid nfs transport is much more complex than register a handler with a simple parameter
like zk_watcher. There are thansands of lines to set up RPC service and very
complex parameter and reply marshalling and unmarshalling. I guess we need severl thansoud
lines to implement a svc_dispatcher like what you expect. Even we have such one,
I don't think it is good to couple nfs server with sheep main loop because our
main loop is very likely to be bottleneck and no obvious benefit from it.

Anyway, current svc_xxx from glibc is very low performance (single threaded),
we need later implement our own sunrpc framework like linux kernel or glusterfs
for pratical use of nfs server over sheepdog. Probably when we do it, we can
make some use of our main loop infrastructure.

Thanks
Yuan