[Sheepdog] [PATCH 0/2] object list testing script

[levin]

The patch is to reduce the count of IO in get_obj_list() when the cluster
is recovering, by adding
a object list cache which implemented by hash table, so it can read the
object list from the cache,
instead of reading the disk to get the object list, which makes it much
faster, and I modified the
get_objlist() hook function of farm and simple_store to make them return
the current full object list in
the disk no matter which epoch the object belongs to, we always keep the
full object list in the cache, when
create_and_write_obj() creates a new object ,we write the new oid to the
cache to make it up to date.

In my testing environment，there's 2G data in the cluster, without the
patch, it takes 69344.645 us for
get_obj_list() to execute which is measured by "collie debug trace" , with
the patch it takes at most
15.655 us.

On Thursday, March 1, 2012, Li Wenpeng wrote:

> I wrote a bash script to test the performance of object list cache:
>
> #!/bin/sh
>
> pkill -9 sheep
> rm /root/sheepdog/store/* -rf
> for i in 0 1 2 3 4 5 6 7 8; do
>    echo "initializing sheep on port 700$i"
>    sheep/sheep -d /root/sheepdog/store/$i -z $i -p 700$i;
>    sleep 1;
> done
> collie/collie cluster format -c 3 -b "farm"
> qemu-img create -f raw sheepdog:debian 2G
> qemu-io -c "write -P 0x1 0 10M" sheepdog:debian
>
> ps aux | grep "sheep.*1$" | awk '{print $2}' | xargs kill -9
> sleep 2
>
> qemu-io -c "write -P 0x1 0 1000M" sheepdog:debian
> qemu-io -c "write -P 0x1 1000M 1000M" sheepdog:debian
>
> collie/collie debug trace -e
>
> ps aux | grep "sheep.*5$" | awk '{print $2}' | xargs kill -9
>
> sheep/sheep -d /root/sheepdog/store/14/ -z 14 -p 7014;
>
> sleep 600
>
> collie/collie debug trace -t
> collie/collie debug trace -C > trace.txt
>
>
> Before the patch is applied, it takes tens of thousands of microseconds
> to execute the function get_obj_list() in store.c, while with the patch
> it only takes dozens of microseconds.
>
> The object list cache avoids too much IO in get_obj_list(), so it runs
> much faster and saves resource.
>
>

-- 
levin
basiccoder.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wpkg.org/pipermail/sheepdog/attachments/20120301/4e0be202/attachment-0003.html>