[Sheepdog] [PATCH v6 0/17] add a new store named 'farm'

[Liu Yuan]

Hi Kazum,

	I think the core of the patch set is stable, farm is ready to be
merged.

changes v6:
 - address Kazum's comments. Thanks!
	- remove compressed feature for sha1_file.
	- use global config to remember backend store.
	- sending name of store instread of index 
	- fix a bug in restore operation.
	- use macros for patch check.

changes v5:
 - address Chirstoph Hellwig's comments. Thanks.
	- check path length at the set-up
	- error handling setxattr
	- use sys/xattr.h
	- rename __trunk_entry to trunk_entry_incore
	- remove unnecessary warppers
	- add comments in the files
	- check ret of strtoull for ULLONG_MAX
 - fix autoconfig as Dongsu Park suggests. Thanks.
 - fix some farm operations in recovery stage
 - support dynamic backend set when formatting
 - transfer backend store for newly joined nodes.
 - abstract out sys vdi bitmap set operation

changes v4:
 - support automatic removal of stale object
 - refine the doc
 - test IO perf
 - drop slab implementation, use glibc malloc/free

changes v3:
 - add basic cluster-wide snapshot support
 - farm.c clean up and refactoring

changes v2:
 - add a doc
 - let farm_link write atomically

   "farm" is suppoed be merged for .4.0, the new kv-store which will
support

 - cluster wide snapshot. (both customized time and at the point of
epoch change)

 - node-wide data sharing for snapshot objects with the same content. so
snapshoting is very cheap operation.

 - support 'collie cluster restore snapshot' to restore fully to the state of
customized snapshots.

 - faster recovery

 - auto checksumed snapshot objects

 - no stale objects that are found in Simple Store implementation, that will
waste storage a lot.

 - practically the same performance as Simple Store.
        A simple qemu-io test on my laptop shows that
                write           read
        farm    14.840 MB/s     11.211 MB/s
        simple  14.842 MB/s     11.245 MB/s

   You can access the patches from farm branch:
	$git checkout -b farm origin/farm

todos:
 - consolidate snapshot feature.
 - more enchancements.

Thanks,
Yuan

Liu Yuan (17):
      sheep: hide some minor store layout aware operations
      store: add dynamic mechanism to chain the available backend stores.
      sheep: transfer store backend for newly joined node
      sheep: modify the configures to run farm.
      sheep: check object directory path at start-up
      farm: add sha1_file operations
      farm: add trunk object
      farm: remove stale object in backend store
      farm: add snapshot object
      sheep: add begin_recover() hook to store.
      sheep: add end_recover() hook to store.
      farm: the farm impelmentation proper
      farm: add a documentation for farm internals
      sheep: add cluster snapshot/restore support
      collie: enable cluster-wide snapshot command
      farm: add format() support
      store: abstract VDI bitmap set-up

 collie/cluster.c         |  219 +++++++++++++++-
 collie/collie.c          |    3 +
 configure.ac             |   14 +
 doc/farm-internal.txt    |  121 +++++++++
 include/sheep.h          |   12 +-
 include/sheepdog_proto.h |   13 +
 sheep/Makefile.am        |    8 +-
 sheep/farm/farm.c        |  644 ++++++++++++++++++++++++++++++++++++++++++++++
 sheep/farm/farm.h        |   79 ++++++
 sheep/farm/sha1_file.c   |  259 +++++++++++++++++++
 sheep/farm/snap.c        |  169 ++++++++++++
 sheep/farm/trunk.c       |  380 +++++++++++++++++++++++++++
 sheep/group.c            |   17 ++-
 sheep/ops.c              |   97 +++++++-
 sheep/sheep_priv.h       |   36 +++-
 sheep/simple_store.c     |   76 +++++-
 sheep/store.c            |  249 +++++++++++-------
 17 files changed, 2273 insertions(+), 123 deletions(-)