[sheepdog] read/write during recovery

[Dietmar Maurer]

> > Well, I never considered such error scenarios. I thought we can simply
> > reject reads during recovery, but that is not the case.
> >
> > The journal does not contain 'all' object data (only the pieces
> > written), so you can never do a successful read. You need to wait until data
> is recovered.
> 
> But wait. Maybe we can force the gateway node to write the whole object if
> needed?

To be more specific:

While a node is in recovery, but obj is not yet recovered, we simply 
return 'SD_RES_NEED_FULL_DATA' for partial writes.

The gateway node then reads the obj data, and finally writes all 4MB 
to the target node (which can update the object immediately).

I am not sure what implication that has on performance, but I can see
some positive side effect when object cache is enabled, because it is likely
that the whole object is already cached.