[sheepdog-users] Sheepdog design architecture

[Liu Yuan]

On Thu, Mar 27, 2014 at 01:27:06PM +0000, Andrew J. Hobbs wrote:
> This would be a good place for clarification on the recovery process 
> when sheepdog is working in md mode.  Does the sheepdog client only 
> attempt to recover the lost data to the remaining drives, or does it 
> trigger a full blown recovery process?  Or is there a threshhold?  IE: 
> if the available remaining capacity would hold the recovered data do 
> that otherwise trigger a recovery/rebalance?

For default, only a local recovery(data recovered on the remaining drivers from
other nodes) is triggered. The purpose of local recovery is to reduce the
recovery negative effect to other nodes.

Note that local recovery won't update the weight of local node. But we can
update the weight of local node by manual command 'dog cluster reweight'.

So if you want a full blown recovery, you can issue 'dog cluster reweight'.

> It's something of a maximization problem.  We want to minimize the 
> amount of time running in a degraded state.  That includes failure 
> states, recovery states, and rebuild states (on the RAID side) as all 
> adversely affect performance.  What percentage of recoveries are 
> triggered by disk failures as opposed to other issues (wiring guy 
> unplugging a switch by accident, true story)?

I also interested in the general percentage but I think it highly depends
on your environment. Anyway, a full blown distributed recovery should be much
faster than a local recovery.

>
> 
> Perhaps the best situation is a hybrid.  IE split the arrays into two 
> RAID 5s of 8 disks each and running sheepdog md on top of them.  But 
> without numbers, it's just a guess.
> 
> Gives me an idea though.  Think I'm going to instrument my cluster a bit 
> to try an get some aggregate numbers during a recovery, both on node 
> failure and disk failure on md nodes.

I'm looking forward to your feedbacks of your test.

Thanks
Yuan