[sheepdog] [PATCH v2] sheep/recovery: multi-threading recovery process
Hitoshi Mitake
mitake.hitoshi at gmail.com
Thu Jan 30 03:33:27 CET 2014
At Thu, 30 Jan 2014 10:17:16 +0800,
Liu Yuan wrote:
>
> On Wed, Jan 29, 2014 at 05:19:09PM +0900, Hitoshi Mitake wrote:
> > At Wed, 29 Jan 2014 16:14:56 +0800,
> > Liu Yuan wrote:
> > >
> > > On Wed, Jan 29, 2014 at 05:01:52PM +0900, Hitoshi Mitake wrote:
> > > > At Wed, 29 Jan 2014 15:53:57 +0800,
> > > > Liu Yuan wrote:
> > > > >
> > > > > On Wed, Jan 29, 2014 at 03:32:34PM +0800, Liu Yuan wrote:
> > > > > > On Wed, Jan 29, 2014 at 04:28:35PM +0900, Hitoshi Mitake wrote:
> > > > > > > At Tue, 28 Jan 2014 18:01:42 +0800,
> > > > > > > Liu Yuan wrote:
> > > > > > > >
> > > > > > > > Rationale for multi-threaded recovery:
> > > > > > > >
> > > > > > > > 1. If one node is added, we find that all the VMs on other nodes will get
> > > > > > > > noticeably affected until 50% data is transferred to the new node.
> > > > > > > >
> > > > > > > > 2. For node failure, we might not have problems of running VM but the
> > > > > > > > recovery process boost will benefit IO operation of VM with less
> > > > > > > > chances to be blocked for write and also improve reliability.
> > > > > > > >
> > > > > > > > 3. For disk failure in node, this is similar to adding a node. All
> > > > > > > > the data on the broken disk will be recovered on other disks in
> > > > > > > > this node. Speedy recoery not only improve data reliability but
> > > > > > > > also cause less writing blocking on the lost data.
> > > > > > > >
> > > > > > > > Our oid scheduling algorithm is intact and simply add multi-threading onto top
> > > > > > > > of current recovery algorithm with minimal changes.
> > > > > > > >
> > > > > > > > - we still have ->oids array to denote oids to be recovered
> > > > > > > > - we start up 2 * nr_disks threads for recovery
> > > > > > > > - the tricky part is that we need to wait all the running threads to
> > > > > > > > completion before start next recovery events for multiple nodes/disks events
> > > > > > > >
> > > > > > > > This patch passes "./check -g md -md" on my local box
> > > > > > >
> > > > > > > On my box, at least 32 and 33 failed. I'm seeking the root cause now
> > > > > > > but this patch seems to be a little bit dangerous.
> > > > > > >
> > > > > >
> > > > > > Yes, this shouldn't go to stable-0.8, but master is okay and at least we need to
> > > > > > pass all the tests before it can goto master.
> > > > >
> > > > > 32 and 33 isn't md-ready tests. Please use
> > > > >
> > > > > ./check -g md -md
> > > > >
> > > > > to test this patch.
> > > >
> > > > 32 and 33 are tests for recovery. So I think we shouldn't exclude them
> > > > for testing your patch, no?
> > >
> > > I run 'sudo tests/functional/check 32 33' several times and no failure.
> >
> > On my environment, 32 and 33 fail every time I run (sudo DRIVER=local
> > ./check 32 33).
> >
> > Below is 32.out.bad and 33.out.bad:
> >
> > 32.out.bad:
> > QA output created by 032
> > using backend plain store
> > 9c7766570b3be3aff2724f587c2f4107 -
> > STORE/1/obj/807c2b2500000000
> > STORE/2/obj/807c2b2500000000
> > STORE/4/obj/807c2b2500000000
> > STORE/5/obj/807c2b2500000000
> > STORE/6/obj/807c2b2500000000
> > STORE/7/obj/807c2b2500000000
> > STORE/0/obj/007c2b2500000000
> > STORE/1/obj/007c2b2500000000
> > STORE/4/obj/007c2b2500000000
> > STORE/5/obj/007c2b2500000000
> > STORE/6/obj/007c2b2500000000
> > STORE/7/obj/007c2b2500000000
> > STORE/0/obj/007c2b2500000001
> > STORE/1/obj/007c2b2500000001
> > STORE/2/obj/007c2b2500000001
> > STORE/5/obj/007c2b2500000001
> > STORE/6/obj/007c2b2500000001
> > STORE/7/obj/007c2b2500000001
> > STORE/0/obj/007c2b2500000002
> > STORE/2/obj/007c2b2500000002
> > STORE/4/obj/007c2b2500000002
> > STORE/5/obj/007c2b2500000002
> > STORE/6/obj/007c2b2500000002
> > STORE/7/obj/007c2b2500000002
> > STORE/0/obj/007c2b2500000003
> > STORE/1/obj/007c2b2500000003
> > STORE/2/obj/007c2b2500000003
> > STORE/5/obj/007c2b2500000003
> > STORE/6/obj/007c2b2500000003
> > STORE/7/obj/007c2b2500000003
> > STORE/0/obj/007c2b2500000004
> > STORE/1/obj/007c2b2500000004
> > STORE/2/obj/007c2b2500000004
> > STORE/4/obj/007c2b2500000004
> > STORE/5/obj/007c2b2500000004
> > STORE/6/obj/007c2b2500000004
> > STORE/1/obj/007c2b2500000005
> > STORE/2/obj/007c2b2500000005
> > STORE/4/obj/007c2b2500000005
> > STORE/5/obj/007c2b2500000005
> > STORE/6/obj/007c2b2500000005
> > STORE/7/obj/007c2b2500000005
> > STORE/0/obj/007c2b2500000006
> > STORE/1/obj/007c2b2500000006
> > STORE/4/obj/007c2b2500000006
> > STORE/5/obj/007c2b2500000006
> > STORE/6/obj/007c2b2500000006
> > STORE/7/obj/007c2b2500000006
> > STORE/0/obj/007c2b2500000007
> > STORE/1/obj/007c2b2500000007
> > STORE/2/obj/007c2b2500000007
> > STORE/4/obj/007c2b2500000007
> > STORE/5/obj/007c2b2500000007
> > STORE/6/obj/007c2b2500000007
> > STORE/0/obj/007c2b2500000008
> > STORE/1/obj/007c2b2500000008
> > STORE/2/obj/007c2b2500000008
> > STORE/4/obj/007c2b2500000008
> > STORE/5/obj/007c2b2500000008
> > STORE/6/obj/007c2b2500000008
> > STORE/1/obj/007c2b2500000009
> > STORE/2/obj/007c2b2500000009
> > STORE/4/obj/007c2b2500000009
> > STORE/5/obj/007c2b2500000009
> > STORE/6/obj/007c2b2500000009
> > STORE/7/obj/007c2b2500000009
> > STORE/0/obj/007c2b250000000a
> > STORE/1/obj/007c2b250000000a
> > STORE/2/obj/007c2b250000000a
> > STORE/4/obj/007c2b250000000a
> > STORE/5/obj/007c2b250000000a
> > STORE/6/obj/007c2b250000000a
> > STORE/0/obj/007c2b250000000b
> > STORE/1/obj/007c2b250000000b
> > STORE/2/obj/007c2b250000000b
> > STORE/4/obj/007c2b250000000b
> > STORE/5/obj/007c2b250000000b
> > STORE/6/obj/007c2b250000000b
> > STORE/0/obj/007c2b250000000c
> > STORE/1/obj/007c2b250000000c
> > STORE/4/obj/007c2b250000000c
> > STORE/5/obj/007c2b250000000c
> > STORE/6/obj/007c2b250000000c
> > STORE/7/obj/007c2b250000000c
> > STORE/1/obj/007c2b250000000d
> > STORE/2/obj/007c2b250000000d
> > STORE/3/obj/007c2b250000000d
> > STORE/4/obj/007c2b250000000d
> > STORE/5/obj/007c2b250000000d
> > STORE/6/obj/007c2b250000000d
> > STORE/7/obj/007c2b250000000d
> > STORE/1/obj/007c2b250000000e
> > STORE/2/obj/007c2b250000000e
> > STORE/4/obj/007c2b250000000e
> > STORE/5/obj/007c2b250000000e
> > STORE/6/obj/007c2b250000000e
> > STORE/7/obj/007c2b250000000e
> > STORE/1/obj/007c2b250000000f
> > STORE/2/obj/007c2b250000000f
> > STORE/4/obj/007c2b250000000f
> > STORE/5/obj/007c2b250000000f
> > STORE/6/obj/007c2b250000000f
> > STORE/7/obj/007c2b250000000f
> > STORE/0/obj/007c2b2500000010
> > STORE/2/obj/007c2b2500000010
> > STORE/4/obj/007c2b2500000010
> > STORE/5/obj/007c2b2500000010
> > STORE/6/obj/007c2b2500000010
> > STORE/7/obj/007c2b2500000010
> > STORE/1/obj/007c2b2500000011
> > STORE/2/obj/007c2b2500000011
> > STORE/4/obj/007c2b2500000011
> > STORE/5/obj/007c2b2500000011
> > STORE/6/obj/007c2b2500000011
> > STORE/7/obj/007c2b2500000011
> > STORE/1/obj/007c2b2500000012
> > STORE/2/obj/007c2b2500000012
> > STORE/4/obj/007c2b2500000012
> > STORE/5/obj/007c2b2500000012
> > STORE/6/obj/007c2b2500000012
> > STORE/7/obj/007c2b2500000012
> > STORE/0/obj/007c2b2500000013
> > STORE/1/obj/007c2b2500000013
> > STORE/2/obj/007c2b2500000013
> > STORE/4/obj/007c2b2500000013
> > STORE/5/obj/007c2b2500000013
> > STORE/7/obj/007c2b2500000013
> > STORE/0/obj/007c2b2500000014
> > STORE/1/obj/007c2b2500000014
> > STORE/2/obj/007c2b2500000014
> > STORE/4/obj/007c2b2500000014
> > STORE/5/obj/007c2b2500000014
> > STORE/6/obj/007c2b2500000014
> > STORE/0/obj/007c2b2500000015
> > STORE/1/obj/007c2b2500000015
> > STORE/2/obj/007c2b2500000015
> > STORE/4/obj/007c2b2500000015
> > STORE/6/obj/007c2b2500000015
> > STORE/7/obj/007c2b2500000015
> > STORE/0/obj/007c2b2500000016
> > STORE/1/obj/007c2b2500000016
> > STORE/3/obj/007c2b2500000016
> > STORE/4/obj/007c2b2500000016
> > STORE/5/obj/007c2b2500000016
> > STORE/6/obj/007c2b2500000016
> > STORE/7/obj/007c2b2500000016
> > STORE/0/obj/007c2b2500000017
> > STORE/1/obj/007c2b2500000017
> > STORE/2/obj/007c2b2500000017
> > STORE/4/obj/007c2b2500000017
> > STORE/5/obj/007c2b2500000017
> > STORE/6/obj/007c2b2500000017
> > STORE/1/obj/007c2b2500000018
> > STORE/2/obj/007c2b2500000018
> > STORE/4/obj/007c2b2500000018
> > STORE/5/obj/007c2b2500000018
> > STORE/6/obj/007c2b2500000018
> > STORE/7/obj/007c2b2500000018
> >
> >
> >
> >
> >
> >
> >
> > 007c2b2500000000.5
> > 007c2b2500000001.5
> > 007c2b2500000002.5
> > 007c2b2500000003.5
> > 007c2b2500000004.5
> > 007c2b2500000005.5
> > 007c2b2500000006.5
> > 007c2b2500000007.5
> > 007c2b2500000008.5
> > 007c2b2500000009.5
> > 007c2b250000000a.5
> > 007c2b250000000b.5
> > 007c2b250000000c.5
> > 007c2b250000000d.5
> > 007c2b250000000e.5
> > 007c2b250000000f.5
> > 007c2b2500000010.5
> > 007c2b2500000011.5
> > 007c2b2500000012.5
> > 007c2b2500000013.5
> > 007c2b2500000014.5
> > 007c2b2500000015.5
> > 007c2b2500000016.5
> > 007c2b2500000017.5
> > 007c2b2500000018.5
> > 807c2b2500000000.5
>
> For 032, I think this is timing issue because all the other output is good.
> I have no idea why you have so many xxxx.5. There is no directory for it?
> it should be in .stale directory.
I executed 032 and the bad output differed from previous one:
QA output created by 032
using backend plain store
9c7766570b3be3aff2724f587c2f4107 -
STORE/0/obj/807c2b2500000000
STORE/2/obj/807c2b2500000000
STORE/4/obj/807c2b2500000000
STORE/5/obj/807c2b2500000000
STORE/6/obj/807c2b2500000000
STORE/7/obj/807c2b2500000000
STORE/0/obj/007c2b2500000000
STORE/2/obj/007c2b2500000000
STORE/4/obj/007c2b2500000000
STORE/5/obj/007c2b2500000000
STORE/6/obj/007c2b2500000000
STORE/7/obj/007c2b2500000000
STORE/0/obj/007c2b2500000001
STORE/2/obj/007c2b2500000001
STORE/4/obj/007c2b2500000001
STORE/5/obj/007c2b2500000001
STORE/6/obj/007c2b2500000001
STORE/7/obj/007c2b2500000001
STORE/0/obj/007c2b2500000002
STORE/2/obj/007c2b2500000002
STORE/4/obj/007c2b2500000002
STORE/5/obj/007c2b2500000002
STORE/6/obj/007c2b2500000002
STORE/7/obj/007c2b2500000002
STORE/0/obj/007c2b2500000003
STORE/1/obj/007c2b2500000003
STORE/2/obj/007c2b2500000003
STORE/4/obj/007c2b2500000003
STORE/5/obj/007c2b2500000003
STORE/6/obj/007c2b2500000003
STORE/7/obj/007c2b2500000003
STORE/0/obj/007c2b2500000004
STORE/2/obj/007c2b2500000004
STORE/4/obj/007c2b2500000004
STORE/5/obj/007c2b2500000004
STORE/6/obj/007c2b2500000004
STORE/7/obj/007c2b2500000004
STORE/0/obj/007c2b2500000005
STORE/2/obj/007c2b2500000005
STORE/4/obj/007c2b2500000005
STORE/5/obj/007c2b2500000005
STORE/6/obj/007c2b2500000005
STORE/7/obj/007c2b2500000005
STORE/0/obj/007c2b2500000006
STORE/2/obj/007c2b2500000006
STORE/4/obj/007c2b2500000006
STORE/5/obj/007c2b2500000006
STORE/6/obj/007c2b2500000006
STORE/7/obj/007c2b2500000006
STORE/0/obj/007c2b2500000007
STORE/2/obj/007c2b2500000007
STORE/4/obj/007c2b2500000007
STORE/5/obj/007c2b2500000007
STORE/6/obj/007c2b2500000007
STORE/7/obj/007c2b2500000007
STORE/0/obj/007c2b2500000008
STORE/2/obj/007c2b2500000008
STORE/4/obj/007c2b2500000008
STORE/5/obj/007c2b2500000008
STORE/6/obj/007c2b2500000008
STORE/7/obj/007c2b2500000008
STORE/0/obj/007c2b2500000009
STORE/2/obj/007c2b2500000009
STORE/4/obj/007c2b2500000009
STORE/5/obj/007c2b2500000009
STORE/6/obj/007c2b2500000009
STORE/7/obj/007c2b2500000009
STORE/0/obj/007c2b250000000a
STORE/1/obj/007c2b250000000a
STORE/2/obj/007c2b250000000a
STORE/4/obj/007c2b250000000a
STORE/5/obj/007c2b250000000a
STORE/6/obj/007c2b250000000a
STORE/7/obj/007c2b250000000a
STORE/0/obj/007c2b250000000b
STORE/2/obj/007c2b250000000b
STORE/4/obj/007c2b250000000b
STORE/5/obj/007c2b250000000b
STORE/6/obj/007c2b250000000b
STORE/7/obj/007c2b250000000b
STORE/0/obj/007c2b250000000c
STORE/2/obj/007c2b250000000c
STORE/4/obj/007c2b250000000c
STORE/5/obj/007c2b250000000c
STORE/6/obj/007c2b250000000c
STORE/7/obj/007c2b250000000c
STORE/0/obj/007c2b250000000d
STORE/1/obj/007c2b250000000d
STORE/2/obj/007c2b250000000d
STORE/3/obj/007c2b250000000d
STORE/4/obj/007c2b250000000d
STORE/5/obj/007c2b250000000d
STORE/6/obj/007c2b250000000d
STORE/7/obj/007c2b250000000d
STORE/0/obj/007c2b250000000e
STORE/2/obj/007c2b250000000e
STORE/4/obj/007c2b250000000e
STORE/5/obj/007c2b250000000e
STORE/6/obj/007c2b250000000e
STORE/7/obj/007c2b250000000e
STORE/0/obj/007c2b250000000f
STORE/2/obj/007c2b250000000f
STORE/4/obj/007c2b250000000f
STORE/5/obj/007c2b250000000f
STORE/6/obj/007c2b250000000f
STORE/7/obj/007c2b250000000f
STORE/0/obj/007c2b2500000010
STORE/2/obj/007c2b2500000010
STORE/4/obj/007c2b2500000010
STORE/5/obj/007c2b2500000010
STORE/6/obj/007c2b2500000010
STORE/7/obj/007c2b2500000010
STORE/0/obj/007c2b2500000011
STORE/2/obj/007c2b2500000011
STORE/4/obj/007c2b2500000011
STORE/5/obj/007c2b2500000011
STORE/6/obj/007c2b2500000011
STORE/7/obj/007c2b2500000011
STORE/0/obj/007c2b2500000012
STORE/2/obj/007c2b2500000012
STORE/4/obj/007c2b2500000012
STORE/5/obj/007c2b2500000012
STORE/6/obj/007c2b2500000012
STORE/7/obj/007c2b2500000012
STORE/0/obj/007c2b2500000013
STORE/2/obj/007c2b2500000013
STORE/4/obj/007c2b2500000013
STORE/5/obj/007c2b2500000013
STORE/6/obj/007c2b2500000013
STORE/7/obj/007c2b2500000013
STORE/0/obj/007c2b2500000014
STORE/2/obj/007c2b2500000014
STORE/4/obj/007c2b2500000014
STORE/5/obj/007c2b2500000014
STORE/6/obj/007c2b2500000014
STORE/7/obj/007c2b2500000014
STORE/0/obj/007c2b2500000015
STORE/2/obj/007c2b2500000015
STORE/4/obj/007c2b2500000015
STORE/5/obj/007c2b2500000015
STORE/6/obj/007c2b2500000015
STORE/7/obj/007c2b2500000015
STORE/0/obj/007c2b2500000016
STORE/1/obj/007c2b2500000016
STORE/2/obj/007c2b2500000016
STORE/3/obj/007c2b2500000016
STORE/4/obj/007c2b2500000016
STORE/5/obj/007c2b2500000016
STORE/6/obj/007c2b2500000016
STORE/7/obj/007c2b2500000016
STORE/0/obj/007c2b2500000017
STORE/2/obj/007c2b2500000017
STORE/4/obj/007c2b2500000017
STORE/5/obj/007c2b2500000017
STORE/6/obj/007c2b2500000017
STORE/7/obj/007c2b2500000017
STORE/0/obj/007c2b2500000018
STORE/2/obj/007c2b2500000018
STORE/4/obj/007c2b2500000018
STORE/5/obj/007c2b2500000018
STORE/6/obj/007c2b2500000018
STORE/7/obj/007c2b2500000018
007c2b2500000000.5
007c2b2500000000.7
007c2b2500000001.5
007c2b2500000001.7
007c2b2500000002.5
007c2b2500000002.7
007c2b2500000003.5
007c2b2500000003.7
007c2b2500000004.5
007c2b2500000004.7
007c2b2500000005.5
007c2b2500000005.7
007c2b2500000006.5
007c2b2500000006.7
007c2b2500000007.5
007c2b2500000007.7
007c2b2500000008.5
007c2b2500000008.7
007c2b2500000009.5
007c2b2500000009.7
007c2b250000000a.5
007c2b250000000a.7
007c2b250000000b.5
007c2b250000000b.7
007c2b250000000c.5
007c2b250000000c.7
007c2b250000000d.5
007c2b250000000d.7
007c2b250000000e.5
007c2b250000000e.7
007c2b250000000f.5
007c2b250000000f.7
007c2b2500000010.5
007c2b2500000010.7
007c2b2500000011.5
007c2b2500000011.7
007c2b2500000012.5
007c2b2500000012.7
007c2b2500000013.5
007c2b2500000013.7
007c2b2500000014.5
007c2b2500000014.7
007c2b2500000015.5
007c2b2500000015.7
007c2b2500000016.5
007c2b2500000016.7
007c2b2500000017.5
007c2b2500000017.7
007c2b2500000018.5
007c2b2500000018.7
807c2b2500000000.5
807c2b2500000000.7
STORE/0/obj/.stale:
STORE/1/obj/.stale:
STORE/2/obj/.stale:
STORE/3/obj/.stale:
STORE/4/obj/.stale:
STORE/5/obj/.stale:
STORE/6/obj/.stale:
STORE/7/obj/.stale:
9c7766570b3be3aff2724f587c2f4107 -
This is an output of "tree -a /tmp/sheepdog/032"
/tmp/sheepdog/032
├── 0
│ ├── config
│ ├── epoch
│ │ ├── 00000001
│ │ ├── 00000002
│ │ ├── 00000003
│ │ ├── 00000004
│ │ ├── 00000005
│ │ ├── 00000006
│ │ ├── 00000007
│ │ ├── 00000008
│ │ ├── 00000009
│ │ ├── 00000010
│ │ ├── 00000011
│ │ ├── 00000012
│ │ ├── 00000013
│ │ ├── 00000014
│ │ ├── 00000015
│ │ ├── 00000016
│ │ └── 00000017
│ ├── lock
│ ├── obj
│ │ ├── .stale
│ │ ├── 007c2b2500000000
│ │ ├── 007c2b2500000001
│ │ ├── 007c2b2500000002
│ │ ├── 007c2b2500000003
│ │ ├── 007c2b2500000004
│ │ ├── 007c2b2500000005
│ │ ├── 007c2b2500000006
│ │ ├── 007c2b2500000007
│ │ ├── 007c2b2500000008
│ │ ├── 007c2b2500000009
│ │ ├── 007c2b250000000a
│ │ ├── 007c2b250000000b
│ │ ├── 007c2b250000000c
│ │ ├── 007c2b250000000d
│ │ ├── 007c2b250000000e
│ │ ├── 007c2b250000000f
│ │ ├── 007c2b2500000010
│ │ ├── 007c2b2500000011
│ │ ├── 007c2b2500000012
│ │ ├── 007c2b2500000013
│ │ ├── 007c2b2500000014
│ │ ├── 007c2b2500000015
│ │ ├── 007c2b2500000016
│ │ ├── 007c2b2500000017
│ │ ├── 007c2b2500000018
│ │ └── 807c2b2500000000
│ ├── sheep.log
│ └── sock
├── 1
│ ├── config
│ ├── core
│ ├── epoch
│ │ ├── 00000001
│ │ ├── 00000002
│ │ ├── 00000003
│ │ ├── 00000004
│ │ ├── 00000005
│ │ ├── 00000006
│ │ ├── 00000007
│ │ ├── 00000009
│ │ ├── 00000010
│ │ ├── 00000011
│ │ ├── 00000012
│ │ └── 00000013
│ ├── gdb.txt
│ ├── lock
│ ├── obj
│ │ ├── .stale
│ │ │ ├── 007c2b2500000000.7
│ │ │ ├── 007c2b2500000001.7
│ │ │ ├── 007c2b2500000002.7
│ │ │ ├── 007c2b2500000003.7
│ │ │ ├── 007c2b2500000004.7
│ │ │ ├── 007c2b2500000005.7
│ │ │ ├── 007c2b2500000006.7
│ │ │ ├── 007c2b2500000007.7
│ │ │ ├── 007c2b2500000008.7
│ │ │ ├── 007c2b2500000009.7
│ │ │ ├── 007c2b250000000a.7
│ │ │ ├── 007c2b250000000b.7
│ │ │ ├── 007c2b250000000c.7
│ │ │ ├── 007c2b250000000d.7
│ │ │ ├── 007c2b250000000e.7
│ │ │ ├── 007c2b250000000f.7
│ │ │ ├── 007c2b2500000010.7
│ │ │ ├── 007c2b2500000011.7
│ │ │ ├── 007c2b2500000012.7
│ │ │ ├── 007c2b2500000013.7
│ │ │ ├── 007c2b2500000014.7
│ │ │ ├── 007c2b2500000015.7
│ │ │ ├── 007c2b2500000016.7
│ │ │ ├── 007c2b2500000017.7
│ │ │ ├── 007c2b2500000018.7
│ │ │ └── 807c2b2500000000.7
│ │ ├── 007c2b2500000003
│ │ ├── 007c2b250000000a
│ │ ├── 007c2b250000000d
│ │ └── 007c2b2500000016
│ ├── sheep.log
│ └── sock
├── 2
│ ├── config
│ ├── epoch
│ │ ├── 00000001
│ │ ├── 00000002
│ │ ├── 00000003
│ │ ├── 00000004
│ │ ├── 00000005
│ │ ├── 00000006
│ │ ├── 00000010
│ │ ├── 00000011
│ │ ├── 00000012
│ │ ├── 00000013
│ │ ├── 00000014
│ │ ├── 00000015
│ │ ├── 00000016
│ │ └── 00000017
│ ├── lock
│ ├── obj
│ │ ├── .stale
│ │ ├── 007c2b2500000000
│ │ ├── 007c2b2500000001
│ │ ├── 007c2b2500000002
│ │ ├── 007c2b2500000003
│ │ ├── 007c2b2500000004
│ │ ├── 007c2b2500000005
│ │ ├── 007c2b2500000006
│ │ ├── 007c2b2500000007
│ │ ├── 007c2b2500000008
│ │ ├── 007c2b2500000009
│ │ ├── 007c2b250000000a
│ │ ├── 007c2b250000000b
│ │ ├── 007c2b250000000c
│ │ ├── 007c2b250000000d
│ │ ├── 007c2b250000000e
│ │ ├── 007c2b250000000f
│ │ ├── 007c2b2500000010
│ │ ├── 007c2b2500000011
│ │ ├── 007c2b2500000012
│ │ ├── 007c2b2500000013
│ │ ├── 007c2b2500000014
│ │ ├── 007c2b2500000015
│ │ ├── 007c2b2500000016
│ │ ├── 007c2b2500000017
│ │ ├── 007c2b2500000018
│ │ └── 807c2b2500000000
│ ├── sheep.log
│ └── sock
├── 3
│ ├── config
│ ├── core
│ ├── epoch
│ │ ├── 00000001
│ │ ├── 00000002
│ │ ├── 00000003
│ │ ├── 00000004
│ │ ├── 00000005
│ │ ├── 00000012
│ │ ├── 00000013
│ │ ├── 00000014
│ │ └── 00000015
│ ├── gdb.txt
│ ├── lock
│ ├── obj
│ │ ├── .stale
│ │ │ ├── 007c2b2500000000.5
│ │ │ ├── 007c2b2500000001.5
│ │ │ ├── 007c2b2500000002.5
│ │ │ ├── 007c2b2500000003.5
│ │ │ ├── 007c2b2500000004.5
│ │ │ ├── 007c2b2500000005.5
│ │ │ ├── 007c2b2500000006.5
│ │ │ ├── 007c2b2500000007.5
│ │ │ ├── 007c2b2500000008.5
│ │ │ ├── 007c2b2500000009.5
│ │ │ ├── 007c2b250000000a.5
│ │ │ ├── 007c2b250000000b.5
│ │ │ ├── 007c2b250000000c.5
│ │ │ ├── 007c2b250000000d.5
│ │ │ ├── 007c2b250000000e.5
│ │ │ ├── 007c2b250000000f.5
│ │ │ ├── 007c2b2500000010.5
│ │ │ ├── 007c2b2500000011.5
│ │ │ ├── 007c2b2500000012.5
│ │ │ ├── 007c2b2500000013.5
│ │ │ ├── 007c2b2500000014.5
│ │ │ ├── 007c2b2500000015.5
│ │ │ ├── 007c2b2500000016.5
│ │ │ ├── 007c2b2500000017.5
│ │ │ ├── 007c2b2500000018.5
│ │ │ └── 807c2b2500000000.5
│ │ ├── 007c2b250000000d
│ │ └── 007c2b2500000016
│ ├── sheep.log
│ └── sock
├── 4
│ ├── config
│ ├── epoch
│ │ ├── 00000001
│ │ ├── 00000002
│ │ ├── 00000003
│ │ ├── 00000004
│ │ ├── 00000011
│ │ ├── 00000012
│ │ ├── 00000013
│ │ ├── 00000014
│ │ ├── 00000015
│ │ ├── 00000016
│ │ └── 00000017
│ ├── lock
│ ├── obj
│ │ ├── .stale
│ │ ├── 007c2b2500000000
│ │ ├── 007c2b2500000001
│ │ ├── 007c2b2500000002
│ │ ├── 007c2b2500000003
│ │ ├── 007c2b2500000004
│ │ ├── 007c2b2500000005
│ │ ├── 007c2b2500000006
│ │ ├── 007c2b2500000007
│ │ ├── 007c2b2500000008
│ │ ├── 007c2b2500000009
│ │ ├── 007c2b250000000a
│ │ ├── 007c2b250000000b
│ │ ├── 007c2b250000000c
│ │ ├── 007c2b250000000d
│ │ ├── 007c2b250000000e
│ │ ├── 007c2b250000000f
│ │ ├── 007c2b2500000010
│ │ ├── 007c2b2500000011
│ │ ├── 007c2b2500000012
│ │ ├── 007c2b2500000013
│ │ ├── 007c2b2500000014
│ │ ├── 007c2b2500000015
│ │ ├── 007c2b2500000016
│ │ ├── 007c2b2500000017
│ │ ├── 007c2b2500000018
│ │ └── 807c2b2500000000
│ ├── sheep.log
│ └── sock
├── 5
│ ├── config
│ ├── epoch
│ │ ├── 00000001
│ │ ├── 00000002
│ │ ├── 00000003
│ │ ├── 00000013
│ │ ├── 00000014
│ │ ├── 00000015
│ │ ├── 00000016
│ │ └── 00000017
│ ├── lock
│ ├── obj
│ │ ├── .stale
│ │ ├── 007c2b2500000000
│ │ ├── 007c2b2500000001
│ │ ├── 007c2b2500000002
│ │ ├── 007c2b2500000003
│ │ ├── 007c2b2500000004
│ │ ├── 007c2b2500000005
│ │ ├── 007c2b2500000006
│ │ ├── 007c2b2500000007
│ │ ├── 007c2b2500000008
│ │ ├── 007c2b2500000009
│ │ ├── 007c2b250000000a
│ │ ├── 007c2b250000000b
│ │ ├── 007c2b250000000c
│ │ ├── 007c2b250000000d
│ │ ├── 007c2b250000000e
│ │ ├── 007c2b250000000f
│ │ ├── 007c2b2500000010
│ │ ├── 007c2b2500000011
│ │ ├── 007c2b2500000012
│ │ ├── 007c2b2500000013
│ │ ├── 007c2b2500000014
│ │ ├── 007c2b2500000015
│ │ ├── 007c2b2500000016
│ │ ├── 007c2b2500000017
│ │ ├── 007c2b2500000018
│ │ └── 807c2b2500000000
│ ├── sheep.log
│ └── sock
├── 6
│ ├── config
│ ├── epoch
│ │ ├── 00000001
│ │ ├── 00000002
│ │ ├── 00000014
│ │ ├── 00000015
│ │ ├── 00000016
│ │ └── 00000017
│ ├── lock
│ ├── obj
│ │ ├── .stale
│ │ ├── 007c2b2500000000
│ │ ├── 007c2b2500000001
│ │ ├── 007c2b2500000002
│ │ ├── 007c2b2500000003
│ │ ├── 007c2b2500000004
│ │ ├── 007c2b2500000005
│ │ ├── 007c2b2500000006
│ │ ├── 007c2b2500000007
│ │ ├── 007c2b2500000008
│ │ ├── 007c2b2500000009
│ │ ├── 007c2b250000000a
│ │ ├── 007c2b250000000b
│ │ ├── 007c2b250000000c
│ │ ├── 007c2b250000000d
│ │ ├── 007c2b250000000e
│ │ ├── 007c2b250000000f
│ │ ├── 007c2b2500000010
│ │ ├── 007c2b2500000011
│ │ ├── 007c2b2500000012
│ │ ├── 007c2b2500000013
│ │ ├── 007c2b2500000014
│ │ ├── 007c2b2500000015
│ │ ├── 007c2b2500000016
│ │ ├── 007c2b2500000017
│ │ ├── 007c2b2500000018
│ │ └── 807c2b2500000000
│ ├── sheep.log
│ └── sock
└── 7
├── config
├── epoch
│ ├── 00000001
│ ├── 00000015
│ ├── 00000016
│ └── 00000017
├── lock
├── obj
│ ├── .stale
│ ├── 007c2b2500000000
│ ├── 007c2b2500000001
│ ├── 007c2b2500000002
│ ├── 007c2b2500000003
│ ├── 007c2b2500000004
│ ├── 007c2b2500000005
│ ├── 007c2b2500000006
│ ├── 007c2b2500000007
│ ├── 007c2b2500000008
│ ├── 007c2b2500000009
│ ├── 007c2b250000000a
│ ├── 007c2b250000000b
│ ├── 007c2b250000000c
│ ├── 007c2b250000000d
│ ├── 007c2b250000000e
│ ├── 007c2b250000000f
│ ├── 007c2b2500000010
│ ├── 007c2b2500000011
│ ├── 007c2b2500000012
│ ├── 007c2b2500000013
│ ├── 007c2b2500000014
│ ├── 007c2b2500000015
│ ├── 007c2b2500000016
│ ├── 007c2b2500000017
│ ├── 007c2b2500000018
│ └── 807c2b2500000000
├── sheep.log
└── sock
32 directories, 331 files
Thanks,
Hitoshi
More information about the sheepdog
mailing list