[sheepdog] What is cooking in master # 2013.8.9

[Liu Yuan]

On Fri, Aug 09, 2013 at 08:52:05PM +0800, Liu Yuan wrote:
> Hello sheepdog walkers,
> 
> This is the report of current status of development of sheepdog project.
> 
> I'll split the report into two part: user-visible, developer-visible from now on.
> 
> So what we are cooking right now in the development mailing list are:
> 
> User-visible changes:
> 
>  1 Release Cycle 
> 
>    Our release cycle was changed and now we'll release a stable version every 3
>    months, which is at the exact the same release day as QEMU. For example,
>    v0.7.0 will be released at 8.15, and v0.8.0 is scheduled at 11.15. Besides
>    release cycle, we'll mantain a stable branch of every release version, 
>    though how long the support for stable branch is not fixed yet. Current 
>    stable branch is 'stable-0.6', and after v0.7.0 released, 'stable-0.7' will
>    be created. At the same time, 'stable-0.6' is still going to be maintained.
> 
>    We have also get a new policy for versioning of release candicate, see
>    http://lists.wpkg.org/pipermail/sheepdog/2013-August/011435.html
> 
>  2 Cluster-wide snapshot is considered working, request for test
> 
>    We have moved 'farm' from sheep to collie as code base for new cluster-wide
>    snapshot. So now we only have 'plain' store left as store backend.
> 
>    features done: incremental backup, auto-deduplication, slicing for dedup
> 
>    For a simple test, it can achieve nearly 50% deduplication for images. More
>    info, see https://github.com/collie/sheepdog/wiki/Backend-Stores%2C-Object-Cache-and-Disk-Cache#cluster-backup
> 
>  3 Better recovery status from 'collie node recovery'
>    
>    For example, the new output is
>    $ collie node recovery
>       Id   Host:Port         V-Nodes       Zone       Progress
>        0   10.68.13.1:7000        64   17646602       56.3%
>        1   10.68.13.2:7000        64   34423818       37.2%
>        2   10.68.13.3:7000        64   51201034       18.1%
>        3   10.68.13.4:7000        64   67978250       68.9%
>        4   10.68.13.5:7000        64   84755466        3.9%
>        5   10.68.13.6:7000        64  101532682       52.6%
>        6   10.68.13.7:7000        64  118309898       77.9%
>        7   10.68.13.8:7000        64  135087114       38.9%
>        8   10.68.13.9:7000        64  151864330        8.7%
>        9   10.68.13.10:7000       64  168641546       97.1%
>       10   10.68.13.11:7000       64  185418762        1.1%
>       11   10.68.13.12:7000       64  202195978       25.6%
> 
>    And you can watch one node progress with a progress bar
>    $ collie node recovery --progress
>    99.7 % [==============================================>] 7047 / 7068
> 
>  4 Openstack Glance support is upstream-merged
>    
>    Now users can use sheepdog as the backend storage both for
>    Cinder(volume service) and Glance(image service)
> 
>    We are also in the effort to provide 'object storage' service on top of
>    sheepdog. We are planned to provide Openstack Swift compatible API first.
> 
>  5 More detailed help messages
>    
>    Users can get the elaberate help messages for specific sheep options with
>    sheep binary. For exmaple, if you don't know how to use cache, try
> 
>    $ sheep -w
>    sheep/sheep: option requires an argument -- 'w'
> Available arguments:
> 	size=: size of the cache in megabyes
> 	dir=: path to the location of the cache (default: $STORE/cache)
> 	directio: use directio mode for cache IO, if not specified use buffered IO
> 
> Example:
> 	$ sheep -w size=200000,dir=/my_ssd,directio ...
> This tries to use /my_ssd as the cache storage with 200G allocted to the
> cache in directio mode
> 
>    A detailed usage with example will be provided.
> 
>  6 Simplied restart of the cluster after crash
>  
>    In the past, we have to start many sheep twice if epoches of sheep get
>    inconsistent. Now we will only have at most two steps to relaunch the nodes
>    from a crashed state:
> 
>    A. start sheep on each node one by one, after all the nodes are up again, the
>       cluster will be working. That is all. But
> 
>    B. If some node(s) is physically down and never have a chance to get up, try
>       $ collie cluster recover force
> 	  The cluster will be working again. Done.
> 
>  7 Qeury the cache infomation on the node
> 
>    A new command is added to get the information of the cache usage
> usage:
>  $ collie vdi cache info # Get the cache informatin of the node
> 
> Example:
> yliu at ubuntu-precise:~/sheepdog$ collie/collie vdi cache info
> Name    Tag     Total   Dirty   Clean
> test            88 MB   68 MB   20 MB
> data            88 MB   0.0 MB  88 MB
> 
> Cache size 200 MB, used 176 MB
> 
>  8 Deb package support
> 
>    Now we can make deb or rpm pakcage on our own by a simple command
>    $ make deb # for debian-based system package
>    $ make rpm # for redhat-based system pakcage
> 
 
One candy I forgot to mention is that auto reconnection from VM to sheep daemon
is finished, but the code will be merged for next release of QEMU. We'll provide
the patched next-released stable v1.6.0 of QEMU with reconnection support at

git at github.com:collie/qemu.git

This qemu repo is served as a temporary place when our patches for QEMU aren't
merged yet in the official QEMU.

Thanks
Yuan