[sheepdog] [PATCH v3 00/12] A new impelmentation of cluster snapshot

Kai Zhang kyle at zelin.io
Tue May 14 10:31:29 CEST 2013


*Usage*
# save all readonly objects to local path
# local path should created before this command
# local path could save more than one cluster snapshot
collie cluster snapshot save /localpath                                                                                                                                     

# list all cluster snapshot saved in local path
collie cluster snapshot list /localpath                                                                                                                                     

# load a snapshot to a cluster
# this will format cluster firstly
# option "-e" is used to specify snapshot if there are more than one
# default is to load the last snapshot if user doesn't specify
collie cluster snapshot load /localpath                                                                                                                                     

More details of usage and command outputs could be found in tests/030 which has been modified for new implementation.

*TODO*
1. compression of snapshot data in sha1 file                                                                                                                                
2. only read snapshot objects created after the latest cluster snapshot taken                                                                                               
3. reduce the size of sha1 file for better de-duplication                                                                                                                  
4. support for saving snapshot to other storage systems, including s3, hdfs, etc. 

Any suggestions would be greatly appreciated:)

Thanks,
Klye

On May 14, 2013, at 3:51 PM, Kai Zhang <kyle at zelin.io> wrote:

> v3: including 10/12, 11/12 and 12/12 which are missed in the last email.
> 
> v2: including new implementation of cluster snapshot
> 
> Current way of doing cluster snapshot is very powerful and has great performance.
> However it also has some drawbacks:
> 1. After a new node joins the cluster, all former snapshots are not available to be restored.
> 2. It is hard to backup cluster snapshot to an other storage system for disaster-tolerance.
> 3. It is hard to init a new cluster by loading another cluster's snapshot.
> 
> New idea is to move "farm" from sheep to collie and save cluster snapshot to a localpath.
> 
> New cluster snapshot retains all features from "farm", including:
> 1. object de-duplication
> 2. incremental store capability
> 
> In addition, it also provides ability of:
> 1. export cluster snapshot to other storage device for backup and disaster-tolerance
> 2. deploy new cluster by restore from one snapshot of other cluster
> 
> The first 4 patches are used to remove farm from sheep.
> The others are used to re-implement farm under collie.
> 
> Kai Zhang (12):
>  sheep: change default store driver from "farm" to "plain"
>  collie: remove snapshot from cluster subcommand
>  script: remove script/simple2farm
>  sheep: remove farm from sheep
>  collie/farm: implement object_rb_tree
>  collie/farm: implement sha1_file
>  collie/farm: implement snap object
>  collie/farm: impelement trunk object
>  sheep: add a flag to let notify_vdi_add set bitmap if needed
>  collie/farm: implement farm
>  collie: implement "collie cluster snapshot" subcommand
>  test: add tests/030 for cluster snapshot
> 
> collie/Makefile.am           |    4 +-
> collie/cluster.c             |  154 ++++++++++++++----------
> collie/farm/farm.c           |  252 ++++++++++++++++++++++++++++++++++++++
> collie/farm/farm.h           |   84 +++++++++++++
> collie/farm/object_rb_tree.c |  148 +++++++++++++++++++++++
> collie/farm/sha1_file.c      |  273 ++++++++++++++++++++++++++++++++++++++++++
> collie/farm/snap.c           |  150 +++++++++++++++++++++++
> collie/farm/trunk.c          |   75 ++++++++++++
> include/sheepdog_proto.h     |    7 +-
> script/simple2farm           |   51 --------
> sheep/Makefile.am            |   10 +-
> sheep/farm/farm.c            |  271 -----------------------------------------
> sheep/farm/farm.h            |   68 -----------
> sheep/farm/sha1_file.c       |  273 ------------------------------------------
> sheep/farm/snap.c            |  152 -----------------------
> sheep/farm/trunk.c           |  139 ---------------------
> sheep/ops.c                  |   65 +----------
> sheep/sheep_priv.h           |    6 +-
> sheep/store.c                |    1 -
> sheep/vdi.c                  |    1 +
> tests/001.out                |    2 +-
> tests/002.out                |    2 +-
> tests/003.out                |    2 +-
> tests/004.out                |    2 +-
> tests/005.out                |    2 +-
> tests/006.out                |    2 +-
> tests/007.out                |    4 +-
> tests/008.out                |    2 +-
> tests/009.out                |    2 +-
> tests/010.out                |    2 +-
> tests/013.out                |    2 +-
> tests/014.out                |    2 +-
> tests/015.out                |    2 +-
> tests/016.out                |    2 +-
> tests/017.out                |    2 +-
> tests/018.out                |    2 +-
> tests/019.out                |    2 +-
> tests/020.out                |    2 +-
> tests/021.out                |    2 +-
> tests/022.out                |    2 +-
> tests/023.out                |    2 +-
> tests/024.out                |    2 +-
> tests/025.out                |    2 +-
> tests/026.out                |    2 +-
> tests/027.out                |    2 +-
> tests/028.out                |    2 +-
> tests/029.out                |    2 +-
> tests/030                    |   77 +++++++++---
> tests/030.out                |   20 +++-
> tests/031.out                |    2 +-
> tests/032.out                |    2 +-
> tests/033.out                |    2 +-
> tests/034.out                |    2 +-
> tests/035.out                |    2 +-
> tests/036.out                |    2 +-
> tests/037.out                |    2 +-
> tests/038.out                |    2 +-
> tests/039.out                |    2 +-
> tests/040.out                |    2 +-
> tests/041.out                |    2 +-
> tests/042.out                |    2 +-
> tests/043.out                |    2 +-
> tests/044.out                |    2 +-
> tests/045.out                |    2 +-
> tests/046.out                |    2 +-
> tests/047.out                |    2 +-
> tests/048.out                |    2 +-
> tests/049.out                |    2 +-
> tests/050.out                |    2 +-
> tests/051.out                |    2 +-
> tests/052.out                |    2 +-
> tests/053.out                |    2 +-
> tests/054.out                |    2 +-
> tests/055.out                |    2 +-
> tests/056.out                |    2 +-
> tests/057.out                |    2 +-
> tests/058.out                |    2 +-
> tests/059.out                |    2 +-
> 78 files changed, 1223 insertions(+), 1172 deletions(-)
> create mode 100644 collie/farm/farm.c
> create mode 100644 collie/farm/farm.h
> create mode 100644 collie/farm/object_rb_tree.c
> create mode 100644 collie/farm/sha1_file.c
> create mode 100644 collie/farm/snap.c
> create mode 100644 collie/farm/trunk.c
> delete mode 100755 script/simple2farm
> delete mode 100644 sheep/farm/farm.c
> delete mode 100644 sheep/farm/farm.h
> delete mode 100644 sheep/farm/sha1_file.c
> delete mode 100644 sheep/farm/snap.c
> delete mode 100644 sheep/farm/trunk.c
> 




More information about the sheepdog mailing list