[Sheepdog] support object recovery

[MORITA Kazutaka]

On Thu, Jan 21, 2010 at 8:32 PM, Piavlo <piavka at cs.bgu.ac.il> wrote:
> The first thing that bothers me is than one I start 3 sheepdogs (no mkfs
> made yet) but I see the following
> fire-srv2 ~ # ls -la  /sheepdog/0/*
> /sheepdog/0/epoch:
> total 0
> drwxr-x--- 1 root root 48 2010-01-21 08:30 .
> drwxr-x--- 1 root root 16 2010-01-21 08:30 ..
> -rw-r----- 1 root root 32 2010-01-21 08:30 00000001
> -rw-r----- 1 root root 64 2010-01-21 08:30 00000002
> -rw-r----- 1 root root 96 2010-01-21 08:30 00000003
>
> /sheepdog/0/obj:
> total 0
> drwxr-x--- 1 root root 48 2010-01-21 08:30 .
> drwxr-x--- 1 root root 16 2010-01-21 08:30 ..
> drwxr-x--- 1 root root  0 2010-01-21 08:30 00000001
> drwxr-x--- 1 root root  0 2010-01-21 08:30 00000002
> drwxr-x--- 1 root root  0 2010-01-21 08:30 00000003
> fire-srv2 ~ #
>
> fire-srv3 ~ # ls -la /sheepdog/0/*
> /sheepdog/0/epoch:
> total 0
> drwxr-x--- 1 root root 48 2010-01-21 08:30 .
> drwxr-x--- 1 root root 16 2010-01-21 08:30 ..
> -rw-r----- 1 root root 32 2010-01-21 08:30 00000001
> -rw-r----- 1 root root 64 2010-01-21 08:30 00000002
> -rw-r----- 1 root root 96 2010-01-21 08:30 00000003
>
> /sheepdog/0/obj:
> total 0
> drwxr-x--- 1 root root 48 2010-01-21 08:30 .
> drwxr-x--- 1 root root 16 2010-01-21 08:30 ..
> drwxr-x--- 1 root root  0 2010-01-21 08:30 00000001
> drwxr-x--- 1 root root  0 2010-01-21 08:30 00000002
> drwxr-x--- 1 root root  0 2010-01-21 08:30 00000003
> fire-srv3 ~ #
>
> fire-srv4 ~ # ls -la /sheepdog/0/*
> /sheepdog/0/epoch:
> total 0
> drwxr-x--- 1 root root 32 2010-01-21 08:30 .
> drwxr-x--- 1 root root 16 2010-01-21 08:30 ..
> -rw-r----- 1 root root 64 2010-01-21 08:30 00000002
> -rw-r----- 1 root root 96 2010-01-21 08:30 00000003
>
> /sheepdog/0/obj:
> total 0
> drwxr-x--- 1 root root 32 2010-01-21 08:30 .
> drwxr-x--- 1 root root 16 2010-01-21 08:30 ..
> drwxr-x--- 1 root root  0 2010-01-21 08:30 00000002
> drwxr-x--- 1 root root  0 2010-01-21 08:30 00000003
> fire-srv4 ~ #
>
> The sheepdogs were started in the following order
> fire-srv2,fire-srv3,fire-srv4
> Why is does fire-srv4 missing 00000001 and what meanings do these files
> have?

`epoch' means the version of sheepdog node map.
In your case,

epoch 1: {fire-srv2}
epoch 2: {fire-srv2, fire-srv3}
epoch 3: {fire-srv2, fire-srv3, fire-srv4}

Collies are logging the information about sheepdog nodes in the epoch directory.
When fire-srv4 are joined to sheepdog nodes, the epoch is already 2,
so  00000001 is not logged at fire-srv4.

> After
> fire-srv2 ~ #   shepherd mkfs --copies=2
> fire-srv2 ~ #
>
> /sheepdog/0/vdi/ is created only on fire-srv3 fire-srv4
> I don't know if the /sheepdog/0/vdi/ can be called an image metadata -
> if yes this means the metadata is will always reside on these two nodes
> only.

Yes, vdi directory is like metadata, which we calld `super object' before.
Currently, the redundancy of vdi directory is same as objects redundancy.
Perhaps, we should change it because vdi directory is more important than
data objects.

How to decide the nodes to store vdi directory is depends on consistent hashing,
so the stored nodes may change if you add or remove nodes.

> Also all image blocks get stored under the  /sheepdog/0/obj/00000003/
> directory - so what is the purpose of /sheepdog/0/obj/0000000{1,2}/ ?
> and what are those /sheepdog/0/epoch/0000000{1,2,3}/?

The current epoch is 3, so objects are stored in obj/00000003 directory.
You didn't create any object when the epoch is 1 or 2, so there is no object
in obj/0000000{1,2}/.

If you create objects when epoch is 1 or 2, and current epoch is 3,
objects in obj/0000000{1,2}/ are moved to obj/00000003 little by little
to store objects based on new node map.

The past epoch directories is not used if they don't have any objects,
so collie should clean up them.

Thanks,

Kazutaka Morita