[Sheepdog] Question about get_obj_list()

[MORITA Kazutaka]

At Wed, 14 Sep 2011 14:32:39 +0800,
Liu Yuan wrote:
> 
> Hi,
>      I am writing something that can gets object distribution stat for 
> specified image like
> 
>      dev at taobao:~/sheepdog$ collie/collie vdi object tailai.ly --stat
>      node                     number of objects
>      192.168.0.1:7000        96
>      192.168.0.2:7000        95
>      192.168.0.3:7000        97
>      ....

Sounds nice. :)

> 
>      In the process, I found a bug in get_obj_list(), which would result 
> in sheep aborting  when handling SD_OP_GET_OBJ_LIST. I traced and found 
> the culprit was 'buf' that was used to serve as a buffer for object 
> list, zalloced from sheep's heap. The problem is, the metadata that 
> gcc's malloc implementation reserved for 'buf' would sometimes get 
> corrupted and following 'free(buf)' would cause
> 
>      *** glibc detected *** sheep/sheep: double free or corruption (out)
> 
> or similar problem and sheep process terminated.
> 
>      From my personal understanding of the code, get_obj_list() serves 
> to return a list of *targeted* objects to the requester. The 
> patch[sheep: remove object list file] changed its logic a bit, and there 
> is a loop that
> iterates from  epoch 1 to epoch n, to merge all the object it finds.
> 
>     I am not sure which line of code overrun the 'buf', but when I 
> remove the for loop, and just return object list
> from one targeted epoch, I have no longer seen the problem.

Oops, I'll take a look at this issue.

> 
>      So my question is, what is idea behind the for loop? Because 
> SD_OP_GET_OBJ_LIST request is served when
> the node is active (agree on the epoch that other nodes can see), so the 
> targeted epoch exists for sure when serving the request. Actually, old 
> objects in the old epoch need to be cleaned up in my opinion. So why bother
> searching and get list from them? I think they are simply stale hardlinks.

It is because to handle multiple node failure.  If Sheepdog increments
a epoch number before finishing object recovery, the latest epoch
directory will not have all objects it should have.  To handle this
problem, in the previous Sheepdog, we created a object list file just
after updating epoch, but there were some problem about it.  I think
the simplest way to create a correct object list is searching all the
object in the Sheepdog cluster, and this is the reason of looping.


Thanks,

Kazutaka