[sheepdog-users] [Sheepdog Announcement] Erasure coding is fully functional with Sheepdog now

[Liu Yuan]

Hello all,

Apologies if this mail sounds annoying to you.

Sheepdog is a distributed object storage system for QEMU VM and RESTful 
services (in progress).

Openstack users can make use of Sheepdog as Cinder and Glance storage as of now 
and Swift compatitable API is working in progress.

Yuan will introduce sheepdog in the openstack Hong Kong summit and slides are
already available at
http://www.slideshare.net/multics/sheepdog-yet-another-all-inone-storage-for-openstack-27402520

You can see more info about this talk at
http://openstacksummitnovember2013.sched.org/event/706dc3952a8917aa74998e047d015e6a#.UmZnNYYW31E

Erasure coding is now seamlessly functional with all other features such as
snapshot/clone/cluster-wide snapshot/multi-disk/auto-healing e.c with following
characteristics:

1 Data is erasure coded automatically while being written to sheepdog storage, no extra operations.
2 Support random read/write, in-place-update, misaligned read/write
3 Support to run any type VM images or attach as a vdisk of a VM
4 User-defined coding scheme on a VDI basis
5 Better read/write performance compared to fully replication scheme
6 A single cluster can both support replicated VDI or erasure-coded VDI

You can get more info at

https://github.com/sheepdog/sheepdog/wiki for a general wiki
https://github.com/sheepdog/sheepdog/wiki/Erasure-Code-Support for erasure coding

Anyone who is interested can give it a try if you are confortable with command
line:

 $ git clone https://github.com/sheepdog/sheepdog.git

For a 10 minutes quick start with a single machine, you can try:
(Assume you are debian based system)

# Compile from the source

 $ sudo apt-get install liburcu-dev
 $ git clone git://github.com/sheepdog/sheepdog.git
 $ cd sheepdog
 $ ./autogen.sh; ./configure --disable-corosync
 $ make; sudo make install

# Create a 6 node cluster with local driver
 $ for i in `seq 0 5`; do sheep /tmp/store$i -n -c local -z $i -p 700$i;done
 $ dog cluster format

# Create a replicated thin-provisioned 20G volume with 3 copies
 $ dog vdi create -c 3 replica 20G
 $ dog vdi list # show vdi list
 $ dog node info # show node information
 $ dog cluster info # show cluster infomation

# Create a erasure-coded (4 data strips and 2 parity strips) 20G volume
 $ dog vdi create -c 4:2 erasure 20G

# Now you should have 2 vdi created
 $ dog vdi list

# You can install OS on these volumes with upstream QEMU
 $ qemu-system-x86_64 -m 1024 --enable-kvm \
   -drive file=sheepdog:erasure,if=virtio -cdrom path_to_your_iso

# or attach the volumes with existant VM
 $ qemu-system-x86_64 -m 1024 --enable-kvm \
   -drive file=your_image,if=virtio -drive file=sheepdog:erasure,if=virtio

# Take a live disk-only snapshot of running VM
 $ dog vdi snapshot -s tag erasure

# Mount the volume(vdi) to local storage file system
 $ sheepfs dir
 $ echo erasure > dir/vdi/mount
# then you can do whatever with the mounted file at dir/vdi/volume/erasure

Have fun

And feedbacks are always welcome.

Thanks
Yuan