[sheepdog-users] monitor cluster to avoid corruption

[Liu Yuan]

On 12/15/2012 06:44 PM, Liu Yuan wrote:
>> To monitor when the cluster is getting full, we have
>> >   collie node info
>> > 
>> > It's pretty easy if we have nodes all with the same amount of space,
>> > we just have to look at the 'Total' percentage or any of the disk.
>> > It gets more difficult when we have different node sizes.
>> > 
>> > Here is an example, after I've been writing 512M (formated with 2 copies)
>> > ---
>> > collie  node info
>> > Id      Size    Used    Use%
>> >  0      982 MB  196 MB   19%
>> >  1      982 MB  160 MB   16%
>> >  2      982 MB  204 MB   20%
>> >  3      10.0 GB 528 MB    5%
>> > Total   13 GB   1.1 GB    8%
>> > Total virtual image size        10 GB
>> > ---
>> > 
>> > And here is the same cluster after I've been writing data till filling
>> > up all the available space.
>> > ---
>> > fino in fondo
>> > collie  node info
>> > Id      Size    Used    Use%
>> >  0      982 MB  980 MB   99%
>> >  1      982 MB  796 MB   81%
>> >  2      982 MB  952 MB   96%
>> >  3      10.0 GB 2.5 GB   25%
>> > Total   13 GB   5.2 GB   40%
>> > Total virtual image size        10 GB
>> > ---
>> > 
>> > *Obviously we can't look at the 'Total' percentage to understand when
>> > the cluster is getting full.*
> With nodes of different size, sheep just try its best to balance the
> data over all nodes. Sheepdog internally use a hash function to store
> objects on to nodes and this is kind of hash collision problem. We have
> make use of virtual nodes to mitigate this problem and it works well
> with multiple images(The more the better). But for a single VM, I think
> it fails its goal. Well, single VM usage isn't practical.
> 
> Please try to test a more practical case, for e.g, to run dozens of VMs.
> If data aren't balanced well, we need to fix sheep then.
> 

I tried a single image case, it looks to me that sheep works well.

yliu at K55VM-ubuntu:~/sheepdog$ collie/collie vdi list
  Name        Id    Size    Used  Shared    Creation time   VDI id  Copies  Tag
  data         1  1.0 GB  1.0 GB  0.0 MB 2012-12-15 18:49   a34a05     1              
yliu at K55VM-ubuntu:~/sheepdog$ collie/collie node info
Id	Size	Used	Use%
 0	100 MB	48 MB	 48%
 1	100 MB	84 MB	 84%
 2	100 MB	76 MB	 76%
 3	1000 MB	820 MB	 82%
Total	1.3 GB	1.0 GB	 79%

Total virtual image size	1.0 GB

node 3 share about x10 storage compared to node 1,2,3. The problem is that 
those disks with smaller space will be much more sensitive to the hash collision.

So the best tip we suggest for heteromerous disks is, don't put a disk too 
smaller than others into the cluster.

Thanks,
Yuan