[sheepdog] [PATCH v2] sheep/recovery: multi-threading recovery process

Hitoshi Mitake mitake.hitoshi at gmail.com
Thu Jan 30 06:42:56 CET 2014


At Thu, 30 Jan 2014 10:44:06 +0800,
Liu Yuan wrote:
> 
> On Thu, Jan 30, 2014 at 11:33:27AM +0900, Hitoshi Mitake wrote:
> > At Thu, 30 Jan 2014 10:17:16 +0800,
> > Liu Yuan wrote:
> > > 
> > > On Wed, Jan 29, 2014 at 05:19:09PM +0900, Hitoshi Mitake wrote:
> > > > At Wed, 29 Jan 2014 16:14:56 +0800,
> > > > Liu Yuan wrote:
> > > > > 
> > > > > On Wed, Jan 29, 2014 at 05:01:52PM +0900, Hitoshi Mitake wrote:
> > > > > > At Wed, 29 Jan 2014 15:53:57 +0800,
> > > > > > Liu Yuan wrote:
> > > > > > > 
> > > > > > > On Wed, Jan 29, 2014 at 03:32:34PM +0800, Liu Yuan wrote:
> > > > > > > > On Wed, Jan 29, 2014 at 04:28:35PM +0900, Hitoshi Mitake wrote:
> > > > > > > > > At Tue, 28 Jan 2014 18:01:42 +0800,
> > > > > > > > > Liu Yuan wrote:
> > > > > > > > > > 
> > > > > > > > > > Rationale for multi-threaded recovery:
> > > > > > > > > > 
> > > > > > > > > > 1. If one node is added, we find that all the VMs on other nodes will get
> > > > > > > > > >    noticeably affected until 50% data is transferred to the new node.
> > > > > > > > > > 
> > > > > > > > > > 2. For node failure, we might not have problems of running VM but the
> > > > > > > > > >    recovery process boost will benefit IO operation of VM with less
> > > > > > > > > >    chances to be blocked for write and also improve reliability.
> > > > > > > > > > 
> > > > > > > > > > 3. For disk failure in node, this is similar to adding a node. All
> > > > > > > > > >    the data on the broken disk will be recovered on other disks in
> > > > > > > > > >    this node. Speedy recoery not only improve data reliability but
> > > > > > > > > >    also cause less writing blocking on the lost data.
> > > > > > > > > > 
> > > > > > > > > > Our oid scheduling algorithm is intact and simply add multi-threading onto top
> > > > > > > > > > of current recovery algorithm with minimal changes.
> > > > > > > > > > 
> > > > > > > > > > - we still have ->oids array to denote oids to be recovered
> > > > > > > > > > - we start up 2 * nr_disks threads for recovery
> > > > > > > > > > - the tricky part is that we need to wait all the running threads to
> > > > > > > > > >   completion before start next recovery events for multiple nodes/disks events
> > > > > > > > > > 
> > > > > > > > > > This patch passes "./check -g md -md" on my local box
> > > > > > > > > 
> > > > > > > > > On my box, at least 32 and 33 failed. I'm seeking the root cause now
> > > > > > > > > but this patch seems to be a little bit dangerous.
> > > > > > > > > 
> > > > > > > > 
> > > > > > > > Yes, this shouldn't go to stable-0.8, but master is okay and at least we need to
> > > > > > > > pass all the tests before it can goto master.
> > > > > > > 
> > > > > > > 32 and 33 isn't md-ready tests. Please use 
> > > > > > > 
> > > > > > > ./check -g md -md
> > > > > > > 
> > > > > > > to test this patch.
> > > > > > 
> > > > > > 32 and 33 are tests for recovery. So I think we shouldn't exclude them
> > > > > > for testing your patch, no?
> > > > > 
> > > > > I run 'sudo tests/functional/check 32 33' several times and no failure.
> > > > 
> > > > On my environment, 32 and 33 fail every time I run (sudo DRIVER=local
> > > > ./check 32 33).
> > > > 
> > > > Below is 32.out.bad and 33.out.bad:
> > > > 
> > > > 32.out.bad:
> > > > QA output created by 032
> > > > using backend plain store
> > > > 9c7766570b3be3aff2724f587c2f4107  -
> > > > STORE/1/obj/807c2b2500000000
> > > > STORE/2/obj/807c2b2500000000
> > > > STORE/4/obj/807c2b2500000000
> > > > STORE/5/obj/807c2b2500000000
> > > > STORE/6/obj/807c2b2500000000
> > > > STORE/7/obj/807c2b2500000000
> > > > STORE/0/obj/007c2b2500000000
> > > > STORE/1/obj/007c2b2500000000
> > > > STORE/4/obj/007c2b2500000000
> > > > STORE/5/obj/007c2b2500000000
> > > > STORE/6/obj/007c2b2500000000
> > > > STORE/7/obj/007c2b2500000000
> > > > STORE/0/obj/007c2b2500000001
> > > > STORE/1/obj/007c2b2500000001
> > > > STORE/2/obj/007c2b2500000001
> > > > STORE/5/obj/007c2b2500000001
> > > > STORE/6/obj/007c2b2500000001
> > > > STORE/7/obj/007c2b2500000001
> > > > STORE/0/obj/007c2b2500000002
> > > > STORE/2/obj/007c2b2500000002
> > > > STORE/4/obj/007c2b2500000002
> > > > STORE/5/obj/007c2b2500000002
> > > > STORE/6/obj/007c2b2500000002
> > > > STORE/7/obj/007c2b2500000002
> > > > STORE/0/obj/007c2b2500000003
> > > > STORE/1/obj/007c2b2500000003
> > > > STORE/2/obj/007c2b2500000003
> > > > STORE/5/obj/007c2b2500000003
> > > > STORE/6/obj/007c2b2500000003
> > > > STORE/7/obj/007c2b2500000003
> > > > STORE/0/obj/007c2b2500000004
> > > > STORE/1/obj/007c2b2500000004
> > > > STORE/2/obj/007c2b2500000004
> > > > STORE/4/obj/007c2b2500000004
> > > > STORE/5/obj/007c2b2500000004
> > > > STORE/6/obj/007c2b2500000004
> > > > STORE/1/obj/007c2b2500000005
> > > > STORE/2/obj/007c2b2500000005
> > > > STORE/4/obj/007c2b2500000005
> > > > STORE/5/obj/007c2b2500000005
> > > > STORE/6/obj/007c2b2500000005
> > > > STORE/7/obj/007c2b2500000005
> > > > STORE/0/obj/007c2b2500000006
> > > > STORE/1/obj/007c2b2500000006
> > > > STORE/4/obj/007c2b2500000006
> > > > STORE/5/obj/007c2b2500000006
> > > > STORE/6/obj/007c2b2500000006
> > > > STORE/7/obj/007c2b2500000006
> > > > STORE/0/obj/007c2b2500000007
> > > > STORE/1/obj/007c2b2500000007
> > > > STORE/2/obj/007c2b2500000007
> > > > STORE/4/obj/007c2b2500000007
> > > > STORE/5/obj/007c2b2500000007
> > > > STORE/6/obj/007c2b2500000007
> > > > STORE/0/obj/007c2b2500000008
> > > > STORE/1/obj/007c2b2500000008
> > > > STORE/2/obj/007c2b2500000008
> > > > STORE/4/obj/007c2b2500000008
> > > > STORE/5/obj/007c2b2500000008
> > > > STORE/6/obj/007c2b2500000008
> > > > STORE/1/obj/007c2b2500000009
> > > > STORE/2/obj/007c2b2500000009
> > > > STORE/4/obj/007c2b2500000009
> > > > STORE/5/obj/007c2b2500000009
> > > > STORE/6/obj/007c2b2500000009
> > > > STORE/7/obj/007c2b2500000009
> > > > STORE/0/obj/007c2b250000000a
> > > > STORE/1/obj/007c2b250000000a
> > > > STORE/2/obj/007c2b250000000a
> > > > STORE/4/obj/007c2b250000000a
> > > > STORE/5/obj/007c2b250000000a
> > > > STORE/6/obj/007c2b250000000a
> > > > STORE/0/obj/007c2b250000000b
> > > > STORE/1/obj/007c2b250000000b
> > > > STORE/2/obj/007c2b250000000b
> > > > STORE/4/obj/007c2b250000000b
> > > > STORE/5/obj/007c2b250000000b
> > > > STORE/6/obj/007c2b250000000b
> > > > STORE/0/obj/007c2b250000000c
> > > > STORE/1/obj/007c2b250000000c
> > > > STORE/4/obj/007c2b250000000c
> > > > STORE/5/obj/007c2b250000000c
> > > > STORE/6/obj/007c2b250000000c
> > > > STORE/7/obj/007c2b250000000c
> > > > STORE/1/obj/007c2b250000000d
> > > > STORE/2/obj/007c2b250000000d
> > > > STORE/3/obj/007c2b250000000d
> > > > STORE/4/obj/007c2b250000000d
> > > > STORE/5/obj/007c2b250000000d
> > > > STORE/6/obj/007c2b250000000d
> > > > STORE/7/obj/007c2b250000000d
> > > > STORE/1/obj/007c2b250000000e
> > > > STORE/2/obj/007c2b250000000e
> > > > STORE/4/obj/007c2b250000000e
> > > > STORE/5/obj/007c2b250000000e
> > > > STORE/6/obj/007c2b250000000e
> > > > STORE/7/obj/007c2b250000000e
> > > > STORE/1/obj/007c2b250000000f
> > > > STORE/2/obj/007c2b250000000f
> > > > STORE/4/obj/007c2b250000000f
> > > > STORE/5/obj/007c2b250000000f
> > > > STORE/6/obj/007c2b250000000f
> > > > STORE/7/obj/007c2b250000000f
> > > > STORE/0/obj/007c2b2500000010
> > > > STORE/2/obj/007c2b2500000010
> > > > STORE/4/obj/007c2b2500000010
> > > > STORE/5/obj/007c2b2500000010
> > > > STORE/6/obj/007c2b2500000010
> > > > STORE/7/obj/007c2b2500000010
> > > > STORE/1/obj/007c2b2500000011
> > > > STORE/2/obj/007c2b2500000011
> > > > STORE/4/obj/007c2b2500000011
> > > > STORE/5/obj/007c2b2500000011
> > > > STORE/6/obj/007c2b2500000011
> > > > STORE/7/obj/007c2b2500000011
> > > > STORE/1/obj/007c2b2500000012
> > > > STORE/2/obj/007c2b2500000012
> > > > STORE/4/obj/007c2b2500000012
> > > > STORE/5/obj/007c2b2500000012
> > > > STORE/6/obj/007c2b2500000012
> > > > STORE/7/obj/007c2b2500000012
> > > > STORE/0/obj/007c2b2500000013
> > > > STORE/1/obj/007c2b2500000013
> > > > STORE/2/obj/007c2b2500000013
> > > > STORE/4/obj/007c2b2500000013
> > > > STORE/5/obj/007c2b2500000013
> > > > STORE/7/obj/007c2b2500000013
> > > > STORE/0/obj/007c2b2500000014
> > > > STORE/1/obj/007c2b2500000014
> > > > STORE/2/obj/007c2b2500000014
> > > > STORE/4/obj/007c2b2500000014
> > > > STORE/5/obj/007c2b2500000014
> > > > STORE/6/obj/007c2b2500000014
> > > > STORE/0/obj/007c2b2500000015
> > > > STORE/1/obj/007c2b2500000015
> > > > STORE/2/obj/007c2b2500000015
> > > > STORE/4/obj/007c2b2500000015
> > > > STORE/6/obj/007c2b2500000015
> > > > STORE/7/obj/007c2b2500000015
> > > > STORE/0/obj/007c2b2500000016
> > > > STORE/1/obj/007c2b2500000016
> > > > STORE/3/obj/007c2b2500000016
> > > > STORE/4/obj/007c2b2500000016
> > > > STORE/5/obj/007c2b2500000016
> > > > STORE/6/obj/007c2b2500000016
> > > > STORE/7/obj/007c2b2500000016
> > > > STORE/0/obj/007c2b2500000017
> > > > STORE/1/obj/007c2b2500000017
> > > > STORE/2/obj/007c2b2500000017
> > > > STORE/4/obj/007c2b2500000017
> > > > STORE/5/obj/007c2b2500000017
> > > > STORE/6/obj/007c2b2500000017
> > > > STORE/1/obj/007c2b2500000018
> > > > STORE/2/obj/007c2b2500000018
> > > > STORE/4/obj/007c2b2500000018
> > > > STORE/5/obj/007c2b2500000018
> > > > STORE/6/obj/007c2b2500000018
> > > > STORE/7/obj/007c2b2500000018
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 007c2b2500000000.5
> > > > 007c2b2500000001.5
> > > > 007c2b2500000002.5
> > > > 007c2b2500000003.5
> > > > 007c2b2500000004.5
> > > > 007c2b2500000005.5
> > > > 007c2b2500000006.5
> > > > 007c2b2500000007.5
> > > > 007c2b2500000008.5
> > > > 007c2b2500000009.5
> > > > 007c2b250000000a.5
> > > > 007c2b250000000b.5
> > > > 007c2b250000000c.5
> > > > 007c2b250000000d.5
> > > > 007c2b250000000e.5
> > > > 007c2b250000000f.5
> > > > 007c2b2500000010.5
> > > > 007c2b2500000011.5
> > > > 007c2b2500000012.5
> > > > 007c2b2500000013.5
> > > > 007c2b2500000014.5
> > > > 007c2b2500000015.5
> > > > 007c2b2500000016.5
> > > > 007c2b2500000017.5
> > > > 007c2b2500000018.5
> > > > 807c2b2500000000.5
> > > 
> > > For 032, I think this is timing issue because all the other output is good. 
> > > I have no idea why you have so many xxxx.5. There is no directory for it?
> > > it should be in .stale directory.
> > 
> > I executed 032 and the bad output differed from previous one:
> > 
> > QA output created by 032
> > using backend plain store
> > 9c7766570b3be3aff2724f587c2f4107  -
> > STORE/0/obj/807c2b2500000000
> > STORE/2/obj/807c2b2500000000
> > STORE/4/obj/807c2b2500000000
> > STORE/5/obj/807c2b2500000000
> > STORE/6/obj/807c2b2500000000
> > STORE/7/obj/807c2b2500000000
> > STORE/0/obj/007c2b2500000000
> > STORE/2/obj/007c2b2500000000
> > STORE/4/obj/007c2b2500000000
> > STORE/5/obj/007c2b2500000000
> > STORE/6/obj/007c2b2500000000
> > STORE/7/obj/007c2b2500000000
> > STORE/0/obj/007c2b2500000001
> > STORE/2/obj/007c2b2500000001
> > STORE/4/obj/007c2b2500000001
> > STORE/5/obj/007c2b2500000001
> > STORE/6/obj/007c2b2500000001
> > STORE/7/obj/007c2b2500000001
> > STORE/0/obj/007c2b2500000002
> > STORE/2/obj/007c2b2500000002
> > STORE/4/obj/007c2b2500000002
> > STORE/5/obj/007c2b2500000002
> > STORE/6/obj/007c2b2500000002
> > STORE/7/obj/007c2b2500000002
> > STORE/0/obj/007c2b2500000003
> > STORE/1/obj/007c2b2500000003
> > STORE/2/obj/007c2b2500000003
> > STORE/4/obj/007c2b2500000003
> > STORE/5/obj/007c2b2500000003
> > STORE/6/obj/007c2b2500000003
> > STORE/7/obj/007c2b2500000003
> > STORE/0/obj/007c2b2500000004
> > STORE/2/obj/007c2b2500000004
> > STORE/4/obj/007c2b2500000004
> > STORE/5/obj/007c2b2500000004
> > STORE/6/obj/007c2b2500000004
> > STORE/7/obj/007c2b2500000004
> > STORE/0/obj/007c2b2500000005
> > STORE/2/obj/007c2b2500000005
> > STORE/4/obj/007c2b2500000005
> > STORE/5/obj/007c2b2500000005
> > STORE/6/obj/007c2b2500000005
> > STORE/7/obj/007c2b2500000005
> > STORE/0/obj/007c2b2500000006
> > STORE/2/obj/007c2b2500000006
> > STORE/4/obj/007c2b2500000006
> > STORE/5/obj/007c2b2500000006
> > STORE/6/obj/007c2b2500000006
> > STORE/7/obj/007c2b2500000006
> > STORE/0/obj/007c2b2500000007
> > STORE/2/obj/007c2b2500000007
> > STORE/4/obj/007c2b2500000007
> > STORE/5/obj/007c2b2500000007
> > STORE/6/obj/007c2b2500000007
> > STORE/7/obj/007c2b2500000007
> > STORE/0/obj/007c2b2500000008
> > STORE/2/obj/007c2b2500000008
> > STORE/4/obj/007c2b2500000008
> > STORE/5/obj/007c2b2500000008
> > STORE/6/obj/007c2b2500000008
> > STORE/7/obj/007c2b2500000008
> > STORE/0/obj/007c2b2500000009
> > STORE/2/obj/007c2b2500000009
> > STORE/4/obj/007c2b2500000009
> > STORE/5/obj/007c2b2500000009
> > STORE/6/obj/007c2b2500000009
> > STORE/7/obj/007c2b2500000009
> > STORE/0/obj/007c2b250000000a
> > STORE/1/obj/007c2b250000000a
> > STORE/2/obj/007c2b250000000a
> > STORE/4/obj/007c2b250000000a
> > STORE/5/obj/007c2b250000000a
> > STORE/6/obj/007c2b250000000a
> > STORE/7/obj/007c2b250000000a
> > STORE/0/obj/007c2b250000000b
> > STORE/2/obj/007c2b250000000b
> > STORE/4/obj/007c2b250000000b
> > STORE/5/obj/007c2b250000000b
> > STORE/6/obj/007c2b250000000b
> > STORE/7/obj/007c2b250000000b
> > STORE/0/obj/007c2b250000000c
> > STORE/2/obj/007c2b250000000c
> > STORE/4/obj/007c2b250000000c
> > STORE/5/obj/007c2b250000000c
> > STORE/6/obj/007c2b250000000c
> > STORE/7/obj/007c2b250000000c
> > STORE/0/obj/007c2b250000000d
> > STORE/1/obj/007c2b250000000d
> > STORE/2/obj/007c2b250000000d
> > STORE/3/obj/007c2b250000000d
> > STORE/4/obj/007c2b250000000d
> > STORE/5/obj/007c2b250000000d
> > STORE/6/obj/007c2b250000000d
> > STORE/7/obj/007c2b250000000d
> > STORE/0/obj/007c2b250000000e
> > STORE/2/obj/007c2b250000000e
> > STORE/4/obj/007c2b250000000e
> > STORE/5/obj/007c2b250000000e
> > STORE/6/obj/007c2b250000000e
> > STORE/7/obj/007c2b250000000e
> > STORE/0/obj/007c2b250000000f
> > STORE/2/obj/007c2b250000000f
> > STORE/4/obj/007c2b250000000f
> > STORE/5/obj/007c2b250000000f
> > STORE/6/obj/007c2b250000000f
> > STORE/7/obj/007c2b250000000f
> > STORE/0/obj/007c2b2500000010
> > STORE/2/obj/007c2b2500000010
> > STORE/4/obj/007c2b2500000010
> > STORE/5/obj/007c2b2500000010
> > STORE/6/obj/007c2b2500000010
> > STORE/7/obj/007c2b2500000010
> > STORE/0/obj/007c2b2500000011
> > STORE/2/obj/007c2b2500000011
> > STORE/4/obj/007c2b2500000011
> > STORE/5/obj/007c2b2500000011
> > STORE/6/obj/007c2b2500000011
> > STORE/7/obj/007c2b2500000011
> > STORE/0/obj/007c2b2500000012
> > STORE/2/obj/007c2b2500000012
> > STORE/4/obj/007c2b2500000012
> > STORE/5/obj/007c2b2500000012
> > STORE/6/obj/007c2b2500000012
> > STORE/7/obj/007c2b2500000012
> > STORE/0/obj/007c2b2500000013
> > STORE/2/obj/007c2b2500000013
> > STORE/4/obj/007c2b2500000013
> > STORE/5/obj/007c2b2500000013
> > STORE/6/obj/007c2b2500000013
> > STORE/7/obj/007c2b2500000013
> > STORE/0/obj/007c2b2500000014
> > STORE/2/obj/007c2b2500000014
> > STORE/4/obj/007c2b2500000014
> > STORE/5/obj/007c2b2500000014
> > STORE/6/obj/007c2b2500000014
> > STORE/7/obj/007c2b2500000014
> > STORE/0/obj/007c2b2500000015
> > STORE/2/obj/007c2b2500000015
> > STORE/4/obj/007c2b2500000015
> > STORE/5/obj/007c2b2500000015
> > STORE/6/obj/007c2b2500000015
> > STORE/7/obj/007c2b2500000015
> > STORE/0/obj/007c2b2500000016
> > STORE/1/obj/007c2b2500000016
> > STORE/2/obj/007c2b2500000016
> > STORE/3/obj/007c2b2500000016
> > STORE/4/obj/007c2b2500000016
> > STORE/5/obj/007c2b2500000016
> > STORE/6/obj/007c2b2500000016
> > STORE/7/obj/007c2b2500000016
> > STORE/0/obj/007c2b2500000017
> > STORE/2/obj/007c2b2500000017
> > STORE/4/obj/007c2b2500000017
> > STORE/5/obj/007c2b2500000017
> > STORE/6/obj/007c2b2500000017
> > STORE/7/obj/007c2b2500000017
> > STORE/0/obj/007c2b2500000018
> > STORE/2/obj/007c2b2500000018
> > STORE/4/obj/007c2b2500000018
> > STORE/5/obj/007c2b2500000018
> > STORE/6/obj/007c2b2500000018
> > STORE/7/obj/007c2b2500000018
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 007c2b2500000000.5
> > 007c2b2500000000.7
> > 007c2b2500000001.5
> > 007c2b2500000001.7
> > 007c2b2500000002.5
> > 007c2b2500000002.7
> > 007c2b2500000003.5
> > 007c2b2500000003.7
> > 007c2b2500000004.5
> > 007c2b2500000004.7
> > 007c2b2500000005.5
> > 007c2b2500000005.7
> > 007c2b2500000006.5
> > 007c2b2500000006.7
> > 007c2b2500000007.5
> > 007c2b2500000007.7
> > 007c2b2500000008.5
> > 007c2b2500000008.7
> > 007c2b2500000009.5
> > 007c2b2500000009.7
> > 007c2b250000000a.5
> > 007c2b250000000a.7
> > 007c2b250000000b.5
> > 007c2b250000000b.7
> > 007c2b250000000c.5
> > 007c2b250000000c.7
> > 007c2b250000000d.5
> > 007c2b250000000d.7
> > 007c2b250000000e.5
> > 007c2b250000000e.7
> > 007c2b250000000f.5
> > 007c2b250000000f.7
> > 007c2b2500000010.5
> > 007c2b2500000010.7
> > 007c2b2500000011.5
> > 007c2b2500000011.7
> > 007c2b2500000012.5
> > 007c2b2500000012.7
> > 007c2b2500000013.5
> > 007c2b2500000013.7
> > 007c2b2500000014.5
> > 007c2b2500000014.7
> > 007c2b2500000015.5
> > 007c2b2500000015.7
> > 007c2b2500000016.5
> > 007c2b2500000016.7
> > 007c2b2500000017.5
> > 007c2b2500000017.7
> > 007c2b2500000018.5
> > 007c2b2500000018.7
> > 807c2b2500000000.5
> > 807c2b2500000000.7
> > STORE/0/obj/.stale:
> > STORE/1/obj/.stale:
> > STORE/2/obj/.stale:
> > STORE/3/obj/.stale:
> > STORE/4/obj/.stale:
> > STORE/5/obj/.stale:
> > STORE/6/obj/.stale:
> > STORE/7/obj/.stale:
> > 9c7766570b3be3aff2724f587c2f4107  -
> > 
> > This is an output of "tree -a /tmp/sheepdog/032"
> > 
> > /tmp/sheepdog/032
> > ├── 0
> > │   ├── config
> > │   ├── epoch
> > │   │   ├── 00000001
> > │   │   ├── 00000002
> > │   │   ├── 00000003
> > │   │   ├── 00000004
> > │   │   ├── 00000005
> > │   │   ├── 00000006
> > │   │   ├── 00000007
> > │   │   ├── 00000008
> > │   │   ├── 00000009
> > │   │   ├── 00000010
> > │   │   ├── 00000011
> > │   │   ├── 00000012
> > │   │   ├── 00000013
> > │   │   ├── 00000014
> > │   │   ├── 00000015
> > │   │   ├── 00000016
> > │   │   └── 00000017
> > │   ├── lock
> > │   ├── obj
> > │   │   ├── .stale
> > │   │   ├── 007c2b2500000000
> > │   │   ├── 007c2b2500000001
> > │   │   ├── 007c2b2500000002
> > │   │   ├── 007c2b2500000003
> > │   │   ├── 007c2b2500000004
> > │   │   ├── 007c2b2500000005
> > │   │   ├── 007c2b2500000006
> > │   │   ├── 007c2b2500000007
> > │   │   ├── 007c2b2500000008
> > │   │   ├── 007c2b2500000009
> > │   │   ├── 007c2b250000000a
> > │   │   ├── 007c2b250000000b
> > │   │   ├── 007c2b250000000c
> > │   │   ├── 007c2b250000000d
> > │   │   ├── 007c2b250000000e
> > │   │   ├── 007c2b250000000f
> > │   │   ├── 007c2b2500000010
> > │   │   ├── 007c2b2500000011
> > │   │   ├── 007c2b2500000012
> > │   │   ├── 007c2b2500000013
> > │   │   ├── 007c2b2500000014
> > │   │   ├── 007c2b2500000015
> > │   │   ├── 007c2b2500000016
> > │   │   ├── 007c2b2500000017
> > │   │   ├── 007c2b2500000018
> > │   │   └── 807c2b2500000000
> > │   ├── sheep.log
> > │   └── sock
> > ├── 1
> > │   ├── config
> > │   ├── core
> > │   ├── epoch
> > │   │   ├── 00000001
> > │   │   ├── 00000002
> > │   │   ├── 00000003
> > │   │   ├── 00000004
> > │   │   ├── 00000005
> > │   │   ├── 00000006
> > │   │   ├── 00000007
> > │   │   ├── 00000009
> > │   │   ├── 00000010
> > │   │   ├── 00000011
> > │   │   ├── 00000012
> > │   │   └── 00000013
> > │   ├── gdb.txt
> > │   ├── lock
> > │   ├── obj
> > │   │   ├── .stale
> > │   │   │   ├── 007c2b2500000000.7
> > │   │   │   ├── 007c2b2500000001.7
> > │   │   │   ├── 007c2b2500000002.7
> > │   │   │   ├── 007c2b2500000003.7
> > │   │   │   ├── 007c2b2500000004.7
> > │   │   │   ├── 007c2b2500000005.7
> > │   │   │   ├── 007c2b2500000006.7
> > │   │   │   ├── 007c2b2500000007.7
> > │   │   │   ├── 007c2b2500000008.7
> > │   │   │   ├── 007c2b2500000009.7
> > │   │   │   ├── 007c2b250000000a.7
> > │   │   │   ├── 007c2b250000000b.7
> > │   │   │   ├── 007c2b250000000c.7
> > │   │   │   ├── 007c2b250000000d.7
> > │   │   │   ├── 007c2b250000000e.7
> > │   │   │   ├── 007c2b250000000f.7
> > │   │   │   ├── 007c2b2500000010.7
> > │   │   │   ├── 007c2b2500000011.7
> > │   │   │   ├── 007c2b2500000012.7
> > │   │   │   ├── 007c2b2500000013.7
> > │   │   │   ├── 007c2b2500000014.7
> > │   │   │   ├── 007c2b2500000015.7
> > │   │   │   ├── 007c2b2500000016.7
> > │   │   │   ├── 007c2b2500000017.7
> > │   │   │   ├── 007c2b2500000018.7
> > │   │   │   └── 807c2b2500000000.7
> > │   │   ├── 007c2b2500000003
> > │   │   ├── 007c2b250000000a
> > │   │   ├── 007c2b250000000d
> > │   │   └── 007c2b2500000016
> > │   ├── sheep.log
> > │   └── sock
> > ├── 2
> > │   ├── config
> > │   ├── epoch
> > │   │   ├── 00000001
> > │   │   ├── 00000002
> > │   │   ├── 00000003
> > │   │   ├── 00000004
> > │   │   ├── 00000005
> > │   │   ├── 00000006
> > │   │   ├── 00000010
> > │   │   ├── 00000011
> > │   │   ├── 00000012
> > │   │   ├── 00000013
> > │   │   ├── 00000014
> > │   │   ├── 00000015
> > │   │   ├── 00000016
> > │   │   └── 00000017
> > │   ├── lock
> > │   ├── obj
> > │   │   ├── .stale
> > │   │   ├── 007c2b2500000000
> > │   │   ├── 007c2b2500000001
> > │   │   ├── 007c2b2500000002
> > │   │   ├── 007c2b2500000003
> > │   │   ├── 007c2b2500000004
> > │   │   ├── 007c2b2500000005
> > │   │   ├── 007c2b2500000006
> > │   │   ├── 007c2b2500000007
> > │   │   ├── 007c2b2500000008
> > │   │   ├── 007c2b2500000009
> > │   │   ├── 007c2b250000000a
> > │   │   ├── 007c2b250000000b
> > │   │   ├── 007c2b250000000c
> > │   │   ├── 007c2b250000000d
> > │   │   ├── 007c2b250000000e
> > │   │   ├── 007c2b250000000f
> > │   │   ├── 007c2b2500000010
> > │   │   ├── 007c2b2500000011
> > │   │   ├── 007c2b2500000012
> > │   │   ├── 007c2b2500000013
> > │   │   ├── 007c2b2500000014
> > │   │   ├── 007c2b2500000015
> > │   │   ├── 007c2b2500000016
> > │   │   ├── 007c2b2500000017
> > │   │   ├── 007c2b2500000018
> > │   │   └── 807c2b2500000000
> > │   ├── sheep.log
> > │   └── sock
> > ├── 3
> > │   ├── config
> > │   ├── core
> > │   ├── epoch
> > │   │   ├── 00000001
> > │   │   ├── 00000002
> > │   │   ├── 00000003
> > │   │   ├── 00000004
> > │   │   ├── 00000005
> > │   │   ├── 00000012
> > │   │   ├── 00000013
> > │   │   ├── 00000014
> > │   │   └── 00000015
> > │   ├── gdb.txt
> > │   ├── lock
> > │   ├── obj
> > │   │   ├── .stale
> > │   │   │   ├── 007c2b2500000000.5
> > │   │   │   ├── 007c2b2500000001.5
> > │   │   │   ├── 007c2b2500000002.5
> > │   │   │   ├── 007c2b2500000003.5
> > │   │   │   ├── 007c2b2500000004.5
> > │   │   │   ├── 007c2b2500000005.5
> > │   │   │   ├── 007c2b2500000006.5
> > │   │   │   ├── 007c2b2500000007.5
> > │   │   │   ├── 007c2b2500000008.5
> > │   │   │   ├── 007c2b2500000009.5
> > │   │   │   ├── 007c2b250000000a.5
> > │   │   │   ├── 007c2b250000000b.5
> > │   │   │   ├── 007c2b250000000c.5
> > │   │   │   ├── 007c2b250000000d.5
> > │   │   │   ├── 007c2b250000000e.5
> > │   │   │   ├── 007c2b250000000f.5
> > │   │   │   ├── 007c2b2500000010.5
> > │   │   │   ├── 007c2b2500000011.5
> > │   │   │   ├── 007c2b2500000012.5
> > │   │   │   ├── 007c2b2500000013.5
> > │   │   │   ├── 007c2b2500000014.5
> > │   │   │   ├── 007c2b2500000015.5
> > │   │   │   ├── 007c2b2500000016.5
> > │   │   │   ├── 007c2b2500000017.5
> > │   │   │   ├── 007c2b2500000018.5
> > │   │   │   └── 807c2b2500000000.5
> > │   │   ├── 007c2b250000000d
> > │   │   └── 007c2b2500000016
> > │   ├── sheep.log
> > │   └── sock
> > ├── 4
> > │   ├── config
> > │   ├── epoch
> > │   │   ├── 00000001
> > │   │   ├── 00000002
> > │   │   ├── 00000003
> > │   │   ├── 00000004
> > │   │   ├── 00000011
> > │   │   ├── 00000012
> > │   │   ├── 00000013
> > │   │   ├── 00000014
> > │   │   ├── 00000015
> > │   │   ├── 00000016
> > │   │   └── 00000017
> > │   ├── lock
> > │   ├── obj
> > │   │   ├── .stale
> > │   │   ├── 007c2b2500000000
> > │   │   ├── 007c2b2500000001
> > │   │   ├── 007c2b2500000002
> > │   │   ├── 007c2b2500000003
> > │   │   ├── 007c2b2500000004
> > │   │   ├── 007c2b2500000005
> > │   │   ├── 007c2b2500000006
> > │   │   ├── 007c2b2500000007
> > │   │   ├── 007c2b2500000008
> > │   │   ├── 007c2b2500000009
> > │   │   ├── 007c2b250000000a
> > │   │   ├── 007c2b250000000b
> > │   │   ├── 007c2b250000000c
> > │   │   ├── 007c2b250000000d
> > │   │   ├── 007c2b250000000e
> > │   │   ├── 007c2b250000000f
> > │   │   ├── 007c2b2500000010
> > │   │   ├── 007c2b2500000011
> > │   │   ├── 007c2b2500000012
> > │   │   ├── 007c2b2500000013
> > │   │   ├── 007c2b2500000014
> > │   │   ├── 007c2b2500000015
> > │   │   ├── 007c2b2500000016
> > │   │   ├── 007c2b2500000017
> > │   │   ├── 007c2b2500000018
> > │   │   └── 807c2b2500000000
> > │   ├── sheep.log
> > │   └── sock
> > ├── 5
> > │   ├── config
> > │   ├── epoch
> > │   │   ├── 00000001
> > │   │   ├── 00000002
> > │   │   ├── 00000003
> > │   │   ├── 00000013
> > │   │   ├── 00000014
> > │   │   ├── 00000015
> > │   │   ├── 00000016
> > │   │   └── 00000017
> > │   ├── lock
> > │   ├── obj
> > │   │   ├── .stale
> > │   │   ├── 007c2b2500000000
> > │   │   ├── 007c2b2500000001
> > │   │   ├── 007c2b2500000002
> > │   │   ├── 007c2b2500000003
> > │   │   ├── 007c2b2500000004
> > │   │   ├── 007c2b2500000005
> > │   │   ├── 007c2b2500000006
> > │   │   ├── 007c2b2500000007
> > │   │   ├── 007c2b2500000008
> > │   │   ├── 007c2b2500000009
> > │   │   ├── 007c2b250000000a
> > │   │   ├── 007c2b250000000b
> > │   │   ├── 007c2b250000000c
> > │   │   ├── 007c2b250000000d
> > │   │   ├── 007c2b250000000e
> > │   │   ├── 007c2b250000000f
> > │   │   ├── 007c2b2500000010
> > │   │   ├── 007c2b2500000011
> > │   │   ├── 007c2b2500000012
> > │   │   ├── 007c2b2500000013
> > │   │   ├── 007c2b2500000014
> > │   │   ├── 007c2b2500000015
> > │   │   ├── 007c2b2500000016
> > │   │   ├── 007c2b2500000017
> > │   │   ├── 007c2b2500000018
> > │   │   └── 807c2b2500000000
> > │   ├── sheep.log
> > │   └── sock
> > ├── 6
> > │   ├── config
> > │   ├── epoch
> > │   │   ├── 00000001
> > │   │   ├── 00000002
> > │   │   ├── 00000014
> > │   │   ├── 00000015
> > │   │   ├── 00000016
> > │   │   └── 00000017
> > │   ├── lock
> > │   ├── obj
> > │   │   ├── .stale
> > │   │   ├── 007c2b2500000000
> > │   │   ├── 007c2b2500000001
> > │   │   ├── 007c2b2500000002
> > │   │   ├── 007c2b2500000003
> > │   │   ├── 007c2b2500000004
> > │   │   ├── 007c2b2500000005
> > │   │   ├── 007c2b2500000006
> > │   │   ├── 007c2b2500000007
> > │   │   ├── 007c2b2500000008
> > │   │   ├── 007c2b2500000009
> > │   │   ├── 007c2b250000000a
> > │   │   ├── 007c2b250000000b
> > │   │   ├── 007c2b250000000c
> > │   │   ├── 007c2b250000000d
> > │   │   ├── 007c2b250000000e
> > │   │   ├── 007c2b250000000f
> > │   │   ├── 007c2b2500000010
> > │   │   ├── 007c2b2500000011
> > │   │   ├── 007c2b2500000012
> > │   │   ├── 007c2b2500000013
> > │   │   ├── 007c2b2500000014
> > │   │   ├── 007c2b2500000015
> > │   │   ├── 007c2b2500000016
> > │   │   ├── 007c2b2500000017
> > │   │   ├── 007c2b2500000018
> > │   │   └── 807c2b2500000000
> > │   ├── sheep.log
> > │   └── sock
> > └── 7
> >     ├── config
> >     ├── epoch
> >     │   ├── 00000001
> >     │   ├── 00000015
> >     │   ├── 00000016
> >     │   └── 00000017
> >     ├── lock
> >     ├── obj
> >     │   ├── .stale
> >     │   ├── 007c2b2500000000
> >     │   ├── 007c2b2500000001
> >     │   ├── 007c2b2500000002
> >     │   ├── 007c2b2500000003
> >     │   ├── 007c2b2500000004
> >     │   ├── 007c2b2500000005
> >     │   ├── 007c2b2500000006
> >     │   ├── 007c2b2500000007
> >     │   ├── 007c2b2500000008
> >     │   ├── 007c2b2500000009
> >     │   ├── 007c2b250000000a
> >     │   ├── 007c2b250000000b
> >     │   ├── 007c2b250000000c
> >     │   ├── 007c2b250000000d
> >     │   ├── 007c2b250000000e
> >     │   ├── 007c2b250000000f
> >     │   ├── 007c2b2500000010
> >     │   ├── 007c2b2500000011
> >     │   ├── 007c2b2500000012
> >     │   ├── 007c2b2500000013
> >     │   ├── 007c2b2500000014
> >     │   ├── 007c2b2500000015
> >     │   ├── 007c2b2500000016
> >     │   ├── 007c2b2500000017
> >     │   ├── 007c2b2500000018
> >     │   └── 807c2b2500000000
> >     ├── sheep.log
> >     └── sock
> > 
> > 32 directories, 331 files
> 
> Can you try this patch for 032 and rerun test?
> 
> diff --git a/tests/functional/032 b/tests/functional/032
> index d902a30..4b65c5d 100755
> --- a/tests/functional/032
> +++ b/tests/functional/032
> @@ -34,6 +34,7 @@ for i in `seq 1 7`; do
>      _start_sheep $i
>  done
>  
> +_wait_for_sheep 8
>  _wait_for_sheep_recovery 0
>  
>  ls $STORE/*/obj/807c2b2500000000 | _filter_store | sort
> 

I tested it and 032 can be passed now.

But the problem of 033 isn't solved yet.

Thanks,
Hitoshi



More information about the sheepdog mailing list