[sheepdog] [Sheepdog] Sheepdog 0.4.0 schedule and plan

Shawn Moore smmoore at gmail.com
Wed May 16 21:16:02 CEST 2012


On Mon, May 14, 2012 at 9:50 PM, Liu Yuan <namei.unix at gmail.com> wrote:
> On 05/15/2012 01:31 AM, Shawn Moore wrote:
>
>> If I had to list them in order of importance for us, it's:
>>
>> 1. reclaim unused objects automatically
> I want to mention that Farm doesn't have this issue.
>
> Kazum and I have tried to address this issue for Simple store, but
> unfortunately no a simple yet efficient method coming out. This problem
> is more difficult than it looks, due to independent node recovering and
> it is very hard to reach agreement for a distributed system without
> simple mechanism. (need paxos?)
>
> Thanks,
> Yuan

Below is my testing about the reclaiming of space in farm.  This was
tested with latest source pulled today.  Feel free to rename the
thread if you want to.

# collie cluster format -b farm -c 1 -H

# collie node info
Id	Size	Used	Use%
 0	63 GB	0.0 MB	  0%
 1	63 GB	0.0 MB	  0%
...
42	63 GB	0.0 MB	  0%
43	63 GB	0.0 MB	  0%
Total	2.7 TB	0.2 MB	  0%

Total virtual image size	0.0 MB

# collie vdi create -P test01 128M

# collie vdi list
  Name        Id    Size    Used  Shared    Creation time   VDI id  Tag
  test01       1  128 MB  128 MB  0.0 MB 2012-05-16 14:30   40687a

# - LIST ALL OBJECTS -
[1] 14:32:57 [SUCCESS] 192.168.1.101
[2] 14:32:57 [SUCCESS] 192.168.1.102
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000018
[3] 14:32:57 [SUCCESS] 192.168.1.103
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000019
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a0000001d
[4] 14:32:57 [SUCCESS] 192.168.1.104
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000000
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000016
-rw-r----- 1 root root 4198968 May 16 14:30 /sheep/obj/8040687a00000000
[5] 14:32:57 [SUCCESS] 192.168.1.105
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000012
[6] 14:32:57 [SUCCESS] 192.168.1.106
[7] 14:32:57 [SUCCESS] 192.168.1.107
[8] 14:32:57 [SUCCESS] 192.168.1.108
[9] 14:32:57 [SUCCESS] 192.168.1.109
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000006
[10] 14:32:57 [SUCCESS] 192.168.1.110
[11] 14:32:57 [SUCCESS] 192.168.1.111
[12] 14:32:57 [SUCCESS] 192.168.1.112
[13] 14:32:57 [SUCCESS] 192.168.1.113
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000007
[14] 14:32:57 [SUCCESS] 192.168.1.114
[15] 14:32:57 [SUCCESS] 192.168.1.115
[16] 14:32:57 [SUCCESS] 192.168.1.116
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000011
[17] 14:32:57 [SUCCESS] 192.168.1.117
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000002
[18] 14:32:57 [SUCCESS] 192.168.1.118
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a0000000c
[19] 14:32:57 [SUCCESS] 192.168.1.119
[20] 14:32:57 [SUCCESS] 192.168.1.120
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000013
[21] 14:32:57 [SUCCESS] 192.168.1.121
[22] 14:32:57 [SUCCESS] 192.168.1.122
[23] 14:32:57 [SUCCESS] 192.168.1.124
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a0000000e
[24] 14:32:57 [SUCCESS] 192.168.1.125
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a0000001c
[25] 14:32:57 [SUCCESS] 192.168.1.126
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a0000001b
[26] 14:32:57 [SUCCESS] 192.168.1.127
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000009
[27] 14:32:57 [SUCCESS] 192.168.1.128
[28] 14:32:57 [SUCCESS] 192.168.1.129
[29] 14:32:57 [SUCCESS] 192.168.1.131
[30] 14:32:57 [SUCCESS] 192.168.1.134
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a0000000d
[31] 14:32:57 [SUCCESS] 192.168.1.133
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a0000000f
[32] 14:32:57 [SUCCESS] 192.168.1.135
[33] 14:32:57 [SUCCESS] 192.168.1.136
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000005
[34] 14:32:58 [SUCCESS] 192.168.1.123
[35] 14:32:58 [SUCCESS] 192.168.1.132
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000004
[36] 14:32:58 [SUCCESS] 192.168.1.130
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000008
[37] 14:32:58 [SUCCESS] 192.168.1.138
[38] 14:32:58 [SUCCESS] 192.168.1.139
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a0000000a
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a0000000b
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000017
[39] 14:32:58 [SUCCESS] 192.168.1.137
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000001
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a0000001f
[40] 14:32:58 [SUCCESS] 192.168.1.140
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000003
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000015
[41] 14:32:58 [SUCCESS] 192.168.1.141
[42] 14:32:58 [SUCCESS] 192.168.1.142
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000010
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a0000001a
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a0000001e
[43] 14:32:58 [SUCCESS] 192.168.1.143
[44] 14:32:58 [SUCCESS] 192.168.1.144
-rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000014

Add up all these sizes and you get 132 MB, which is 128MB (data) plus
4MB for 8040687a00000000 (not sure what is? Index?) file.

Now if I delete the VDI I'm left with the 8040687a00000000 file, but
all else is gone.
[1] 14:36:18 [SUCCESS] 192.168.1.101
[2] 14:36:18 [SUCCESS] 192.168.1.102
[3] 14:36:18 [SUCCESS] 192.168.1.103
[4] 14:36:18 [SUCCESS] 192.168.1.104
-rw-r----- 1 root root 4198968 May 16 14:36 /sheep/obj/8040687a00000000
[5] 14:36:18 [SUCCESS] 192.168.1.105
...
[42] 14:36:18 [SUCCESS] 192.168.1.143
[43] 14:36:18 [SUCCESS] 192.168.1.144
[44] 14:36:23 [SUCCESS] 192.168.1.137

We can live with this, just wasted 4MB per deleted VDI unless it is
re-created later with the same VDI name, then it's re-used.

But when you involve snapshots, you can end up with a lot of left over
"80" files that add up at 4MB a piece per deleted snapshot.

If you create test01 again you get:
  Name        Id    Size    Used  Shared    Creation time   VDI id  Tag
  test01       1  128 MB  128 MB  0.0 MB 2012-05-16 14:38   40687a

Now create 100 snapshots:
s test01 1 134217728 134217728 0 1337193484 40687a SNAP_0
s test01 2 134217728 0 134217728 1337193559 40687b SNAP_1
...
s test01 99 134217728 0 134217728 1337193613 4068dc SNAP_98
s test01 100 134217728 0 134217728 1337193613 4068dd SNAP_99

All the VDI id's increment by hex 1.  When you delete these, you are left with:
= test01 101 134217728 0 134217728 1337193614 4068de

Even after you delete the VDI:
# collie vdi list
  Name        Id    Size    Used  Shared    Creation time   VDI id  Tag

You are left with:
[1] 14:45:57 [SUCCESS] 192.168.1.101
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688500000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068a600000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068c300000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068cb00000000
[2] 14:45:57 [SUCCESS] 192.168.1.102
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688e00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068ba00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068d400000000
[3] 14:45:57 [SUCCESS] 192.168.1.103
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068ab00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068c800000000
[4] 14:45:57 [SUCCESS] 192.168.1.104
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040687a00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689a00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068a500000000
[5] 14:45:57 [SUCCESS] 192.168.1.105
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068a200000000
[6] 14:45:57 [SUCCESS] 192.168.1.106
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689b00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068c700000000
-rw-r----- 1 root root 4198968 May 16 14:45 /sheep/obj/804068de00000000
[7] 14:45:57 [SUCCESS] 192.168.1.107
[8] 14:45:57 [SUCCESS] 192.168.1.108
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688300000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068a900000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068af00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068b600000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068d300000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068db00000000
[9] 14:45:57 [SUCCESS] 192.168.1.109
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040687e00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068b400000000
[10] 14:45:57 [SUCCESS] 192.168.1.110
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068cf00000000
[11] 14:45:57 [SUCCESS] 192.168.1.111
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040687d00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688f00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068bc00000000
[12] 14:45:57 [SUCCESS] 192.168.1.112
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689400000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068ce00000000
[13] 14:45:57 [SUCCESS] 192.168.1.113
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068a100000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068b200000000
[14] 14:45:57 [SUCCESS] 192.168.1.114
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040687f00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688000000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689f00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068a800000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068d100000000
[15] 14:45:57 [SUCCESS] 192.168.1.115
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688200000000
[16] 14:45:57 [SUCCESS] 192.168.1.116
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688900000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068b900000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068bf00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068c600000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068c900000000
[17] 14:45:57 [SUCCESS] 192.168.1.117
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040687c00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689700000000
[18] 14:45:57 [SUCCESS] 192.168.1.118
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068c200000000
[19] 14:45:57 [SUCCESS] 192.168.1.119
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068a000000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068a300000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068c500000000
[20] 14:45:57 [SUCCESS] 192.168.1.120
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688100000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688b00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689500000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689900000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068a400000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068b700000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068d500000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068da00000000
[21] 14:45:57 [SUCCESS] 192.168.1.121
[22] 14:45:57 [SUCCESS] 192.168.1.122
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068c000000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068c400000000
[23] 14:45:57 [SUCCESS] 192.168.1.123
[24] 14:45:57 [SUCCESS] 192.168.1.124
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040687b00000000
[25] 14:45:57 [SUCCESS] 192.168.1.125
[26] 14:45:57 [SUCCESS] 192.168.1.126
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689200000000
[27] 14:45:57 [SUCCESS] 192.168.1.127
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068aa00000000
[28] 14:45:57 [SUCCESS] 192.168.1.128
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689e00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068bd00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068dc00000000
[29] 14:45:57 [SUCCESS] 192.168.1.129
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689600000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068b500000000
[30] 14:45:57 [SUCCESS] 192.168.1.130
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068b000000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068ca00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068d800000000
[31] 14:45:57 [SUCCESS] 192.168.1.131
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688400000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688d00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068ae00000000
[32] 14:45:57 [SUCCESS] 192.168.1.133
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689000000000
[33] 14:45:57 [SUCCESS] 192.168.1.132
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688600000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688800000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688a00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689800000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068ad00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068be00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068c100000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068d600000000
[34] 14:45:57 [SUCCESS] 192.168.1.134
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689300000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068d000000000
[35] 14:45:57 [SUCCESS] 192.168.1.135
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688700000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068b300000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068cc00000000
[36] 14:45:57 [SUCCESS] 192.168.1.138
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068d900000000
[37] 14:45:57 [SUCCESS] 192.168.1.137
[38] 14:45:57 [SUCCESS] 192.168.1.136
[39] 14:45:57 [SUCCESS] 192.168.1.139
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688c00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068a700000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068ac00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068b800000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068dd00000000
[40] 14:45:57 [SUCCESS] 192.168.1.140
[41] 14:45:57 [SUCCESS] 192.168.1.141
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689100000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689d00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068d200000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068d700000000
[42] 14:45:57 [SUCCESS] 192.168.1.142
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689c00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068b100000000
[43] 14:45:57 [SUCCESS] 192.168.1.143
[44] 14:45:57 [SUCCESS] 192.168.1.144
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068bb00000000
-rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068cd00000000

Which is ~404MB left over.  Now this is with just one VDI and copies
set to 1.  In our real usage, we want 3 copies.  Currently we take 4
snapshots per hour (*/15 min) per VDI with ZFS.  With 4 snaps per hour
per day per year and copies set to 3 this equals
"(((4*24)*365)*3)/1024" ~ 102.6 GB per VDI per year wasted.  We
currently have what will be 140 VDI's.

Even if you delete all snapshots, and then the main VDI, when you
re-create the VDI it re-uses the initial 80 file, but new snapshots do
not re-use snapshot 80 files.  So I create 1 VDI and 100 snapshots.
After deleting all snapshots and the VDI itself, I was left with 101
80 files.  Then I re-create the VDI and keep the same number of 80
files (101).  Now I create 20 new snapshots and the number of 80 files
goes to 121.  I then delete all snapshots and VDI and I am still left
with 121 total files.

I have not found a way to reclaim this space without formatting the
cluster.  If I just delete a "80" file I get:
# collie vdi list
  Name        Id    Size    Used  Shared    Creation time   VDI id  Tag
Failed to read object 8040687a00000000 No object found
Failed to read inode header

Regards,
Shawn



More information about the sheepdog mailing list