[sheepdog] [PATCH 0/9] introduce libsheepdog.so
Liu Yuan
namei.unix at gmail.com
Fri Apr 3 10:11:15 CEST 2015
On Fri, Apr 03, 2015 at 03:59:18PM +0900, Hitoshi Mitake wrote:
> At Fri, 3 Apr 2015 12:06:56 +0800,
> Liu Yuan wrote:
> >
> > On Fri, Apr 03, 2015 at 12:49:31PM +0900, Hitoshi Mitake wrote:
> > > At Fri, 3 Apr 2015 11:20:46 +0800,
> > > Liu Yuan wrote:
> > > >
> > > > From: Liu Yuan <liuyuan at cmss.chinamobile.com>
> > > >
> > > > Finally, we kick started, though it is far away from a complete implemention.
> > > >
> > > > This patch set introduces a shared library and for now it only supports simple
> > > > vdi operation. The vdi read/write operation would be the most difficult part of
> > > > shared library, which require a high-performance request handling. The other
> > > > adminsitrative operations, can be implemented on top of a single send-response
> > > > helper function, which send the sheep header and get the response.
> > > >
> > > > Currently, it only suppose 6 API:
> > > >
> > > > sd_{connect, disconnect, vdi_open, vdi_read, vdi_write, vdi_close}, but we can
> > > > extend it to other commands in the future. The hardest part of the extension is
> > > > not the implemention, but the API itself. We need more users inputs and
> > > > production experices to shape our future APIs.
> > > >
> > > > The core idea is the same as sbd's framework.
> > > >
> > > > User request
> > > > |
> > > > | aio control block (struct aiocb)
> > > > V
> > > > +----+----+
> > > > | | |
> > > > r1 r2 r3
> > > > | | |
> > > > V V V
> > > > obj obj obj
> > > >
> > > > User request is spit into several sheep requests which are performed on the
> > > > individual sheepdog object. All the sheep requests are queued and sent in a
> > > > async mode concurently. When all the sheep requests are done, aiocb will try
> > > > to user of the completion of the user request.
> > > >
> > > > For future possible other administritive opcodes, we provide a sync function,
> > > > which can run the opcode and get the response in a sync mode.
> > > >
> > > > note: "sudo make install" will install the api headers into /usr/include/sheepdog
> > > > and libsheepdog.so into /usr/lib/libsheepdog.so.
> > > >
> > > > A simple example of test.c to make use of libsheepdog.so:
> > > >
> > > > Suppose you have a cluster and a vdi named test.
> > > >
> > > > #include <stdio.h>
> > > > #include <stdlib.h>
> > > > #include <sheepdog/sheepdog.h>
> > > >
> > > > int main()
> > > > {
> > > > struct sd_cluster *c = sd_connect("127.0.0.1:7000");
> > > > struct sd_vdi *vdi;
> > > > char buf[512] = {};
> > > >
> > > > if (!c) {
> > > > fprintf(stderr, "failed to connect %m\n");
> > > > return -1;
> > > > }
> > > >
> > > > vdi = sd_vdi_open(c, "test");
> > > > if (!vdi) {
> > > > fprintf(stderr, "failed to open %m\n");
> > > > return -1;
> > > > }
> > > >
> > > > sd_vdi_write(vdi, "hello world!\n", 512, 0);
> > > > sd_vdi_read(vdi, buf, 512, 0);
> > > > sd_vdi_close(vdi);
> > > > printf("%s", buf);
> > > > return 0;
> > > > }
> > > >
> > > > compile:
> > > > $ gcc test.c -lsheepdog -lpthread
> > > >
> > > > ./a.out will show you "hello world!" if all goes well.
> > > >
> > > > Liu Yuan (8):
> > > > add sd_assert()
> > > > move functions used by libsheepdog.a out of util.c
> > > > sheep: dissociate utils.c from logger.c
> > > > move sheep specific headers out of sheepdog_proto.h
> > > > unralate list.h, util.h to compiler.h
> > > > shared lib: add the low level request handling framework
> > > > shared lib: implement vdi_{open, read, write, close}
> > > > shared lib: kick it run
> > > >
> > > > configure.ac | 14 +-
> > > > dog/common.c | 25 ++
> > > > dog/dog.h | 6 +
> > > > dog/treeview.c | 1 +
> > > > include/Makefile.am | 3 +-
> > > > include/compiler.h | 9 -
> > > > include/fec.h | 1 +
> > > > include/list.h | 2 -
> > > > include/logger.h | 11 +-
> > > > include/sheep.h | 14 +
> > > > include/sheepdog_proto.h | 16 +-
> > > > include/util.h | 69 +++--
> > > > include/work.h | 1 +
> > > > lib/Makefile.am | 17 +-
> > > > lib/common.c | 279 ++++++++++++++++++++
> > > > lib/event.c | 1 +
> > > > lib/fec.c | 15 +-
> > > > lib/logger.c | 6 +-
> > > > lib/sd_inode.c | 1 +
> > > > lib/shared/sheep.c | 642 ++++++++++++++++++++++++++++++++++++++++++++++
> > > > lib/shared/sheepdog.h | 74 ++++++
> > > > lib/shared/vdi.c | 185 +++++++++++++
> > > > lib/util.c | 285 +-------------------
> > > > lib/work.c | 2 +-
> > > > sheep/cluster.h | 1 +
> > > > sheep/cluster/corosync.c | 2 +-
> > > > sheep/cluster/shepherd.c | 4 +-
> > > > sheep/cluster/zookeeper.c | 2 +-
> > > > sheep/group.c | 6 +-
> > > > sheep/md.c | 4 +-
> > > > sheep/migrate.c | 2 +-
> > > > sheep/object_cache.c | 6 +-
> > > > sheep/ops.c | 6 +-
> > > > sheep/plain_store.c | 2 +-
> > > > sheep/request.c | 10 +-
> > > > sheep/sheep.c | 2 +-
> > > > sheep/sheep_priv.h | 1 +
> > > > sheep/trace/trace.c | 4 +-
> > > > sheep/vdi.c | 16 +-
> > > > sheepfs/core.c | 1 +
> > > > shepherd/shepherd.c | 1 +
> > > > 41 files changed, 1343 insertions(+), 406 deletions(-)
> > > > create mode 100644 lib/common.c
> > > > create mode 100644 lib/shared/sheep.c
> > > > create mode 100644 lib/shared/sheepdog.h
> > > > create mode 100644 lib/shared/vdi.c
> > >
> > > What is the motivation of this shared library?
> > >
> >
> > This is called by the users on the list for a long time. The motivation is for
> > third party softwares to make use of sheepdog without calling dog.
> >
> > For e.g, openstack cinder, nova and glance's driver can call libsheepdog.to to
> > implement its driver instead of calling dog for performance reason and kill some
> > workarounds that dog can't sovle.
> >
> > One of our internal requirements is to backup the sheepdog cluster into other
> > media. Calling dog to back up the whole cluster data is a pain in the ass for
> > performance reason and dog can't meet our needs because we have to send sheep
> > hdr by ourself to achieve some purposes. I don't think dog is a suggested approach
> > to accomplish the third party library demands, which want to integerate sheepdog
> > into their own systems.
> >
> > All the conterparts storage systems like glusterfs, ceph provide its own libraries
> > , such as librados. shared library for sheepdog is requested by users several
> > times on this list, if I remember well. The recent request is from Zhang Kai (kyle)
> > from zelin.io.
>
> OK, I'll review it later.
>
> But I suggest proposing design before working on implementation. For
> example, it is not obvious why dog and qemu-img cannot satisfy your
> backup requirements but libsheepdog can do.
It is obvious dog's rw is very bad at performance and every call to 'dog vdi read'
or 'dog vdi write' will send two cluster wide notification "LOCK and RELEASE",
which will increase znode hugely in zk server and influence sheepdog cluster a
lot. Anyway, this is not the point that we need libsheepdog.so, as you can guess,
it will enable other softwares to access storage provided by sheepdog. We want
to integrate sheepdog into our own software stack as some others do.
To create a client library allow many other use cases is why we need it.
As you can see, we discuss the design for years and no one bring it as code. It
is always better to patch a working system than talking on something non-exist.
Talk is cheap, so I show up the code.
Thanks,
Yuan
Thanks,
Yuan
More information about the sheepdog
mailing list