[Sheepdog] [PATCH v6 0/17] add a new store named 'farm'

[MORITA Kazutaka]

At Thu, 12 Jan 2012 21:37:11 +0800,
Liu Yuan wrote:
> 
> Hi Kazum,
> 
> 	I think the core of the patch set is stable, farm is ready to be
> merged.
> 
> changes v6:
>  - address Kazum's comments. Thanks!
> 	- remove compressed feature for sha1_file.
> 	- use global config to remember backend store.
> 	- sending name of store instread of index 
> 	- fix a bug in restore operation.
> 	- use macros for patch check.
> 
> changes v5:
>  - address Chirstoph Hellwig's comments. Thanks.
> 	- check path length at the set-up
> 	- error handling setxattr
> 	- use sys/xattr.h
> 	- rename __trunk_entry to trunk_entry_incore
> 	- remove unnecessary warppers
> 	- add comments in the files
> 	- check ret of strtoull for ULLONG_MAX
>  - fix autoconfig as Dongsu Park suggests. Thanks.
>  - fix some farm operations in recovery stage
>  - support dynamic backend set when formatting
>  - transfer backend store for newly joined nodes.
>  - abstract out sys vdi bitmap set operation
> 
> changes v4:
>  - support automatic removal of stale object
>  - refine the doc
>  - test IO perf
>  - drop slab implementation, use glibc malloc/free
> 
> changes v3:
>  - add basic cluster-wide snapshot support
>  - farm.c clean up and refactoring
> 
> changes v2:
>  - add a doc
>  - let farm_link write atomically
> 
>    "farm" is suppoed be merged for .4.0, the new kv-store which will
> support
> 
>  - cluster wide snapshot. (both customized time and at the point of
> epoch change)
> 
>  - node-wide data sharing for snapshot objects with the same content. so
> snapshoting is very cheap operation.
> 
>  - support 'collie cluster restore snapshot' to restore fully to the state of
> customized snapshots.
> 
>  - faster recovery
> 
>  - auto checksumed snapshot objects
> 
>  - no stale objects that are found in Simple Store implementation, that will
> waste storage a lot.
> 
>  - practically the same performance as Simple Store.
>         A simple qemu-io test on my laptop shows that
>                 write           read
>         farm    14.840 MB/s     11.211 MB/s
>         simple  14.842 MB/s     11.245 MB/s
> 
>    You can access the patches from farm branch:
> 	$git checkout -b farm origin/farm
> 
> todos:
>  - consolidate snapshot feature.
>  - more enchancements.
> 
> Thanks,
> Yuan
> 
> Liu Yuan (17):
>       sheep: hide some minor store layout aware operations
>       store: add dynamic mechanism to chain the available backend stores.
>       sheep: transfer store backend for newly joined node
>       sheep: modify the configures to run farm.
>       sheep: check object directory path at start-up
>       farm: add sha1_file operations
>       farm: add trunk object
>       farm: remove stale object in backend store
>       farm: add snapshot object
>       sheep: add begin_recover() hook to store.
>       sheep: add end_recover() hook to store.
>       farm: the farm impelmentation proper
>       farm: add a documentation for farm internals
>       sheep: add cluster snapshot/restore support
>       collie: enable cluster-wide snapshot command
>       farm: add format() support
>       store: abstract VDI bitmap set-up
> 
>  collie/cluster.c         |  219 +++++++++++++++-
>  collie/collie.c          |    3 +
>  configure.ac             |   14 +
>  doc/farm-internal.txt    |  121 +++++++++
>  include/sheep.h          |   12 +-
>  include/sheepdog_proto.h |   13 +
>  sheep/Makefile.am        |    8 +-
>  sheep/farm/farm.c        |  644 ++++++++++++++++++++++++++++++++++++++++++++++
>  sheep/farm/farm.h        |   79 ++++++
>  sheep/farm/sha1_file.c   |  259 +++++++++++++++++++
>  sheep/farm/snap.c        |  169 ++++++++++++
>  sheep/farm/trunk.c       |  380 +++++++++++++++++++++++++++
>  sheep/group.c            |   17 ++-
>  sheep/ops.c              |   97 +++++++-
>  sheep/sheep_priv.h       |   36 +++-
>  sheep/simple_store.c     |   76 +++++-
>  sheep/store.c            |  249 +++++++++++-------
>  17 files changed, 2273 insertions(+), 123 deletions(-)

Applied after fixing some coding style issues.

I think there are still some problems in this patchset, but
let's switch to a more incremental development instead of
sending a big patchset each time.

Thanks,

Kazutaka