[sheepdog] [PATCH v2] sheep/recovery: multi-threading recovery process
Hitoshi Mitake
mitake.hitoshi at gmail.com
Thu Jan 30 02:11:43 CET 2014
At Thu, 30 Jan 2014 02:53:27 +0800,
Liu Yuan wrote:
>
> On Wed, Jan 29, 2014 at 06:04:15PM +0900, Hitoshi Mitake wrote:
> > At Wed, 29 Jan 2014 17:38:34 +0900,
> > Hitoshi Mitake wrote:
> > >
> > > At Wed, 29 Jan 2014 16:29:03 +0800,
> > > Liu Yuan wrote:
> > > >
> > > > On Wed, Jan 29, 2014 at 05:19:09PM +0900, Hitoshi Mitake wrote:
> > > > > At Wed, 29 Jan 2014 16:14:56 +0800,
> > > > > Liu Yuan wrote:
> > > > > >
> > > > > > On Wed, Jan 29, 2014 at 05:01:52PM +0900, Hitoshi Mitake wrote:
> > > > > > > At Wed, 29 Jan 2014 15:53:57 +0800,
> > > > > > > Liu Yuan wrote:
> > > > > > > >
> > > > > > > > On Wed, Jan 29, 2014 at 03:32:34PM +0800, Liu Yuan wrote:
> > > > > > > > > On Wed, Jan 29, 2014 at 04:28:35PM +0900, Hitoshi Mitake wrote:
> > > > > > > > > > At Tue, 28 Jan 2014 18:01:42 +0800,
> > > > > > > > > > Liu Yuan wrote:
> > > > > > > > > > >
> > > > > > > > > > > Rationale for multi-threaded recovery:
> > > > > > > > > > >
> > > > > > > > > > > 1. If one node is added, we find that all the VMs on other nodes will get
> > > > > > > > > > > noticeably affected until 50% data is transferred to the new node.
> > > > > > > > > > >
> > > > > > > > > > > 2. For node failure, we might not have problems of running VM but the
> > > > > > > > > > > recovery process boost will benefit IO operation of VM with less
> > > > > > > > > > > chances to be blocked for write and also improve reliability.
> > > > > > > > > > >
> > > > > > > > > > > 3. For disk failure in node, this is similar to adding a node. All
> > > > > > > > > > > the data on the broken disk will be recovered on other disks in
> > > > > > > > > > > this node. Speedy recoery not only improve data reliability but
> > > > > > > > > > > also cause less writing blocking on the lost data.
> > > > > > > > > > >
> > > > > > > > > > > Our oid scheduling algorithm is intact and simply add multi-threading onto top
> > > > > > > > > > > of current recovery algorithm with minimal changes.
> > > > > > > > > > >
> > > > > > > > > > > - we still have ->oids array to denote oids to be recovered
> > > > > > > > > > > - we start up 2 * nr_disks threads for recovery
> > > > > > > > > > > - the tricky part is that we need to wait all the running threads to
> > > > > > > > > > > completion before start next recovery events for multiple nodes/disks events
> > > > > > > > > > >
> > > > > > > > > > > This patch passes "./check -g md -md" on my local box
> > > > > > > > > >
> > > > > > > > > > On my box, at least 32 and 33 failed. I'm seeking the root cause now
> > > > > > > > > > but this patch seems to be a little bit dangerous.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Yes, this shouldn't go to stable-0.8, but master is okay and at least we need to
> > > > > > > > > pass all the tests before it can goto master.
> > > > > > > >
> > > > > > > > 32 and 33 isn't md-ready tests. Please use
> > > > > > > >
> > > > > > > > ./check -g md -md
> > > > > > > >
> > > > > > > > to test this patch.
> > > > > > >
> > > > > > > 32 and 33 are tests for recovery. So I think we shouldn't exclude them
> > > > > > > for testing your patch, no?
> > > > > >
> > > > > > I run 'sudo tests/functional/check 32 33' several times and no failure.
> > > > >
> > > > > On my environment, 32 and 33 fail every time I run (sudo DRIVER=local
> > > > > ./check 32 33).
> > > > >
> > > > > Below is 32.out.bad and 33.out.bad:
> > > > >
> > > > > 32.out.bad:
> > > > > QA output created by 032
> > > > > using backend plain store
> > > > > 9c7766570b3be3aff2724f587c2f4107 -
> > > > > STORE/1/obj/807c2b2500000000
> > > > > STORE/2/obj/807c2b2500000000
> > > > > STORE/4/obj/807c2b2500000000
> > > > > STORE/5/obj/807c2b2500000000
> > > > > STORE/6/obj/807c2b2500000000
> > > > > STORE/7/obj/807c2b2500000000
> > > > > STORE/0/obj/007c2b2500000000
> > > > > STORE/1/obj/007c2b2500000000
> > > > > STORE/4/obj/007c2b2500000000
> > > > > STORE/5/obj/007c2b2500000000
> > > > > STORE/6/obj/007c2b2500000000
> > > > > STORE/7/obj/007c2b2500000000
> > > > > STORE/0/obj/007c2b2500000001
> > > > > STORE/1/obj/007c2b2500000001
> > > > > STORE/2/obj/007c2b2500000001
> > > > > STORE/5/obj/007c2b2500000001
> > > > > STORE/6/obj/007c2b2500000001
> > > > > STORE/7/obj/007c2b2500000001
> > > > > STORE/0/obj/007c2b2500000002
> > > > > STORE/2/obj/007c2b2500000002
> > > > > STORE/4/obj/007c2b2500000002
> > > > > STORE/5/obj/007c2b2500000002
> > > > > STORE/6/obj/007c2b2500000002
> > > > > STORE/7/obj/007c2b2500000002
> > > > > STORE/0/obj/007c2b2500000003
> > > > > STORE/1/obj/007c2b2500000003
> > > > > STORE/2/obj/007c2b2500000003
> > > > > STORE/5/obj/007c2b2500000003
> > > > > STORE/6/obj/007c2b2500000003
> > > > > STORE/7/obj/007c2b2500000003
> > > > > STORE/0/obj/007c2b2500000004
> > > > > STORE/1/obj/007c2b2500000004
> > > > > STORE/2/obj/007c2b2500000004
> > > > > STORE/4/obj/007c2b2500000004
> > > > > STORE/5/obj/007c2b2500000004
> > > > > STORE/6/obj/007c2b2500000004
> > > > > STORE/1/obj/007c2b2500000005
> > > > > STORE/2/obj/007c2b2500000005
> > > > > STORE/4/obj/007c2b2500000005
> > > > > STORE/5/obj/007c2b2500000005
> > > > > STORE/6/obj/007c2b2500000005
> > > > > STORE/7/obj/007c2b2500000005
> > > > > STORE/0/obj/007c2b2500000006
> > > > > STORE/1/obj/007c2b2500000006
> > > > > STORE/4/obj/007c2b2500000006
> > > > > STORE/5/obj/007c2b2500000006
> > > > > STORE/6/obj/007c2b2500000006
> > > > > STORE/7/obj/007c2b2500000006
> > > > > STORE/0/obj/007c2b2500000007
> > > > > STORE/1/obj/007c2b2500000007
> > > > > STORE/2/obj/007c2b2500000007
> > > > > STORE/4/obj/007c2b2500000007
> > > > > STORE/5/obj/007c2b2500000007
> > > > > STORE/6/obj/007c2b2500000007
> > > > > STORE/0/obj/007c2b2500000008
> > > > > STORE/1/obj/007c2b2500000008
> > > > > STORE/2/obj/007c2b2500000008
> > > > > STORE/4/obj/007c2b2500000008
> > > > > STORE/5/obj/007c2b2500000008
> > > > > STORE/6/obj/007c2b2500000008
> > > > > STORE/1/obj/007c2b2500000009
> > > > > STORE/2/obj/007c2b2500000009
> > > > > STORE/4/obj/007c2b2500000009
> > > > > STORE/5/obj/007c2b2500000009
> > > > > STORE/6/obj/007c2b2500000009
> > > > > STORE/7/obj/007c2b2500000009
> > > > > STORE/0/obj/007c2b250000000a
> > > > > STORE/1/obj/007c2b250000000a
> > > > > STORE/2/obj/007c2b250000000a
> > > > > STORE/4/obj/007c2b250000000a
> > > > > STORE/5/obj/007c2b250000000a
> > > > > STORE/6/obj/007c2b250000000a
> > > > > STORE/0/obj/007c2b250000000b
> > > > > STORE/1/obj/007c2b250000000b
> > > > > STORE/2/obj/007c2b250000000b
> > > > > STORE/4/obj/007c2b250000000b
> > > > > STORE/5/obj/007c2b250000000b
> > > > > STORE/6/obj/007c2b250000000b
> > > > > STORE/0/obj/007c2b250000000c
> > > > > STORE/1/obj/007c2b250000000c
> > > > > STORE/4/obj/007c2b250000000c
> > > > > STORE/5/obj/007c2b250000000c
> > > > > STORE/6/obj/007c2b250000000c
> > > > > STORE/7/obj/007c2b250000000c
> > > > > STORE/1/obj/007c2b250000000d
> > > > > STORE/2/obj/007c2b250000000d
> > > > > STORE/3/obj/007c2b250000000d
> > > > > STORE/4/obj/007c2b250000000d
> > > > > STORE/5/obj/007c2b250000000d
> > > > > STORE/6/obj/007c2b250000000d
> > > > > STORE/7/obj/007c2b250000000d
> > > > > STORE/1/obj/007c2b250000000e
> > > > > STORE/2/obj/007c2b250000000e
> > > > > STORE/4/obj/007c2b250000000e
> > > > > STORE/5/obj/007c2b250000000e
> > > > > STORE/6/obj/007c2b250000000e
> > > > > STORE/7/obj/007c2b250000000e
> > > > > STORE/1/obj/007c2b250000000f
> > > > > STORE/2/obj/007c2b250000000f
> > > > > STORE/4/obj/007c2b250000000f
> > > > > STORE/5/obj/007c2b250000000f
> > > > > STORE/6/obj/007c2b250000000f
> > > > > STORE/7/obj/007c2b250000000f
> > > > > STORE/0/obj/007c2b2500000010
> > > > > STORE/2/obj/007c2b2500000010
> > > > > STORE/4/obj/007c2b2500000010
> > > > > STORE/5/obj/007c2b2500000010
> > > > > STORE/6/obj/007c2b2500000010
> > > > > STORE/7/obj/007c2b2500000010
> > > > > STORE/1/obj/007c2b2500000011
> > > > > STORE/2/obj/007c2b2500000011
> > > > > STORE/4/obj/007c2b2500000011
> > > > > STORE/5/obj/007c2b2500000011
> > > > > STORE/6/obj/007c2b2500000011
> > > > > STORE/7/obj/007c2b2500000011
> > > > > STORE/1/obj/007c2b2500000012
> > > > > STORE/2/obj/007c2b2500000012
> > > > > STORE/4/obj/007c2b2500000012
> > > > > STORE/5/obj/007c2b2500000012
> > > > > STORE/6/obj/007c2b2500000012
> > > > > STORE/7/obj/007c2b2500000012
> > > > > STORE/0/obj/007c2b2500000013
> > > > > STORE/1/obj/007c2b2500000013
> > > > > STORE/2/obj/007c2b2500000013
> > > > > STORE/4/obj/007c2b2500000013
> > > > > STORE/5/obj/007c2b2500000013
> > > > > STORE/7/obj/007c2b2500000013
> > > > > STORE/0/obj/007c2b2500000014
> > > > > STORE/1/obj/007c2b2500000014
> > > > > STORE/2/obj/007c2b2500000014
> > > > > STORE/4/obj/007c2b2500000014
> > > > > STORE/5/obj/007c2b2500000014
> > > > > STORE/6/obj/007c2b2500000014
> > > > > STORE/0/obj/007c2b2500000015
> > > > > STORE/1/obj/007c2b2500000015
> > > > > STORE/2/obj/007c2b2500000015
> > > > > STORE/4/obj/007c2b2500000015
> > > > > STORE/6/obj/007c2b2500000015
> > > > > STORE/7/obj/007c2b2500000015
> > > > > STORE/0/obj/007c2b2500000016
> > > > > STORE/1/obj/007c2b2500000016
> > > > > STORE/3/obj/007c2b2500000016
> > > > > STORE/4/obj/007c2b2500000016
> > > > > STORE/5/obj/007c2b2500000016
> > > > > STORE/6/obj/007c2b2500000016
> > > > > STORE/7/obj/007c2b2500000016
> > > > > STORE/0/obj/007c2b2500000017
> > > > > STORE/1/obj/007c2b2500000017
> > > > > STORE/2/obj/007c2b2500000017
> > > > > STORE/4/obj/007c2b2500000017
> > > > > STORE/5/obj/007c2b2500000017
> > > > > STORE/6/obj/007c2b2500000017
> > > > > STORE/1/obj/007c2b2500000018
> > > > > STORE/2/obj/007c2b2500000018
> > > > > STORE/4/obj/007c2b2500000018
> > > > > STORE/5/obj/007c2b2500000018
> > > > > STORE/6/obj/007c2b2500000018
> > > > > STORE/7/obj/007c2b2500000018
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > 007c2b2500000000.5
> > > > > 007c2b2500000001.5
> > > > > 007c2b2500000002.5
> > > > > 007c2b2500000003.5
> > > > > 007c2b2500000004.5
> > > > > 007c2b2500000005.5
> > > > > 007c2b2500000006.5
> > > > > 007c2b2500000007.5
> > > > > 007c2b2500000008.5
> > > > > 007c2b2500000009.5
> > > > > 007c2b250000000a.5
> > > > > 007c2b250000000b.5
> > > > > 007c2b250000000c.5
> > > > > 007c2b250000000d.5
> > > > > 007c2b250000000e.5
> > > > > 007c2b250000000f.5
> > > > > 007c2b2500000010.5
> > > > > 007c2b2500000011.5
> > > > > 007c2b2500000012.5
> > > > > 007c2b2500000013.5
> > > > > 007c2b2500000014.5
> > > > > 007c2b2500000015.5
> > > > > 007c2b2500000016.5
> > > > > 007c2b2500000017.5
> > > > > 007c2b2500000018.5
> > > > > 807c2b2500000000.5
> > > > > STORE/0/obj/.stale:
> > > > > STORE/1/obj/.stale:
> > > > > STORE/2/obj/.stale:
> > > > > STORE/3/obj/.stale:
> > > > > STORE/4/obj/.stale:
> > > > > STORE/5/obj/.stale:
> > > > > STORE/6/obj/.stale:
> > > > > STORE/7/obj/.stale:
> > > > > 9c7766570b3be3aff2724f587c2f4107 -
> > > > >
> > > > >
> > > > >
> > > > > 33.out.bad:
> > > > > QA output created by 033
> > > > > using backend plain store
> > > > > 9c7766570b3be3aff2724f587c2f4107 -
> > > > > should have 2, but have 1 sheep
> > > > >
> > > >
> > > > I am also using local driver and haven't met a single failure yet. Could you try
> > > > make clean; then test?
> > > >
> > > > Seems that 33 has core? if yes, it would be easy to debug.
> > >
> > > Ah yes, I could find the core file. I'll look at it later.
> > >
> > > BTW, below is a tail of the dead sheep's log:
> > >
> > > Jan 29 17:33:15 DEBUG [rw 18759] fetch_object_list(971) 14
> > > Jan 29 17:33:15 DEBUG [rw 18759] prepare_object_list(1039) go to the next recovery
> > > Jan 29 17:33:15 DEBUG [main] run_next_rw(680) running threads nr 0
> > > Jan 29 17:33:15 EMERG [rw 18759] crash_handler(267) sheep exits unexpectedly (Segmentation fault).
> > > Jan 29 17:33:15 DEBUG [io 18806] do_process_work(1393) a1, 0, 7
> > > Jan 29 17:33:15 EMERG [rw 18759] sd_backtrace(817) sheep.c:269: crash_handler
> > > Jan 29 17:33:15 EMERG [rw 18759] sd_backtrace(831) /lib/x86_64-linux-gnu/libpthread.so.0(+0xf02f) [0x7f5dcd64302f]
> > > Jan 29 17:33:15 EMERG [rw 18759] sd_backtrace(817) work.c:336: worker_routine
> > > Jan 29 17:33:15 EMERG [rw 18759] sd_backtrace(831) /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b4f) [0x7f5dcd63ab4f]
> > > Jan 29 17:33:15 EMERG [rw 18759] sd_backtrace(831) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6c) [0x7f5dcccf7a7c]
> >
> > I forgot to mention. I could reproduce the problem and above log even
> > after cleaning and rebuild.
>
> I think besides core file, there is gdb.txt, can you paste it here?
OK, this is gdb.txt produced on my environment.
Thanks,
Hitoshi
==
== Jan 30 10:07:26
== program: /home/mitake/github/sheepdog.git/sheep/sheep
== command: thread apply all where full
==
Thread 17 (Thread 0x7f8b52fce700 (LWP 6717)):
#0 0x00007f8b62f076bd in sendmsg () at ../sysdeps/unix/syscall-template.S:82
No locals.
#1 0x0000000000422a15 in do_write (max_count=4294967295, epoch=0, need_retry=0, len=4194352, msg=0x7f8b52fcc730, sockfd=22) at net.c:267
ret = <optimized out>
repeat = -1
#2 send_req (sockfd=22, hdr=hdr at entry=0x7f8b52fcc7c0, data=<optimized out>, wlen=4194304, need_retry=need_retry at entry=0, epoch=epoch at entry=0, max_count=max_count at entry=4294967295) at net.c:316
ret = <optimized out>
msg = {msg_name = 0x0, msg_namelen = 0, msg_iov = 0x7f8b52fcc710, msg_iovlen = 2, msg_control = 0x0, msg_controllen = 0, msg_flags = 0}
iov = {{iov_base = 0x7f8b52fcc7c0, iov_len = 48}, {iov_base = 0x7f8b4c3c0000, iov_len = 4194304}}
__func__ = "send_req"
#3 0x0000000000408e87 in tx_work (work=0x167cc70) at request.c:792
ci = 0x167cc00
ret = <optimized out>
conn = 0x167cc00
rsp = {proto_ver = 0 '\000', opcode = 164 '\244', flags = 0, epoch = 4, id = 0, data_length = 4194304, {result = 0, obj = {__pad = 0, copies = 0 '\000', reserved = "\000\000", offset = 0}, vdi = {__pad = 0, rsvd = 0, vdi_id = 0, attr_id = 0, copies = 0 '\000', reserved = "\000\000"}, node = {__pad = 0, nr_nodes = 0, __reserved = {0, 0}, store_size = 0, store_free = 0}, hash = {__pad1 = 0, __pad2 = 0, digest = '\000' <repeats 19 times>}, __pad = {0, 0, 0, 0, 0, 0, 0, 0}}}
req = <optimized out>
data = <optimized out>
__func__ = "tx_work"
#4 0x00000000004273eb in worker_routine (arg=0x1665340) at work.c:348
wi = 0x1665340
work = 0x167cc70
tid = 6717
__func__ = "worker_routine"
#5 0x00007f8b62effb50 in start_thread (arg=<optimized out>) at pthread_create.c:304
__res = <optimized out>
pd = 0x7f8b52fce700
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140236369487616, 2993067626498392730, 140236637118880, 140236369488320, 140236641488960, 7, -2981621700169792870, -2981586388538588518}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#6 0x00007f8b625bca7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
No locals.
#7 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 16 (Thread 0x7f8b527cd700 (LWP 6718)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
No locals.
#1 0x000000000042739b in sd_cond_wait (mutex=<optimized out>, cond=0x16655f8) at ../include/util.h:371
No locals.
#2 worker_routine (arg=0x1665580) at work.c:337
wi = 0x1665580
work = <optimized out>
tid = 6718
__func__ = "worker_routine"
#3 0x00007f8b62effb50 in start_thread (arg=<optimized out>) at pthread_create.c:304
__res = <optimized out>
pd = 0x7f8b527cd700
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140236361094912, 2993067626498392730, 140236637118880, 140236361095616, 140236641488960, 7, -2981620600121294182, -2981586388538588518}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#4 0x00007f8b625bca7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
No locals.
#5 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 15 (Thread 0x7f8b51fcc700 (LWP 6719)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
No locals.
#1 0x000000000042739b in sd_cond_wait (mutex=<optimized out>, cond=0x1665838) at ../include/util.h:371
No locals.
#2 worker_routine (arg=0x16657c0) at work.c:337
wi = 0x16657c0
work = <optimized out>
tid = 6719
__func__ = "worker_routine"
#3 0x00007f8b62effb50 in start_thread (arg=<optimized out>) at pthread_create.c:304
__res = <optimized out>
pd = 0x7f8b51fcc700
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140236352702208, 2993067626498392730, 140236637118880, 140236352702912, 140236641488960, 7, -2981628296165817702, -2981586388538588518}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#4 0x00007f8b625bca7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
No locals.
#5 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 14 (Thread 0x7f8b517cb700 (LWP 6720)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
No locals.
#1 0x000000000042739b in sd_cond_wait (mutex=<optimized out>, cond=0x1665a78) at ../include/util.h:371
No locals.
#2 worker_routine (arg=0x1665a00) at work.c:337
wi = 0x1665a00
work = <optimized out>
tid = 6720
__func__ = "worker_routine"
#3 0x00007f8b62effb50 in start_thread (arg=<optimized out>) at pthread_create.c:304
__res = <optimized out>
pd = 0x7f8b517cb700
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140236344309504, 2993067626498392730, 140236637118880, 140236344310208, 140236641488960, 7, -2981627196117319014, -2981586388538588518}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#4 0x00007f8b625bca7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
No locals.
#5 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 13 (Thread 0x7f8b50fca700 (LWP 6721)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
No locals.
#1 0x000000000042739b in sd_cond_wait (mutex=<optimized out>, cond=0x1665cb8) at ../include/util.h:371
No locals.
#2 worker_routine (arg=0x1665c40) at work.c:337
wi = 0x1665c40
work = <optimized out>
tid = 6721
__func__ = "worker_routine"
#3 0x00007f8b62effb50 in start_thread (arg=<optimized out>) at pthread_create.c:304
__res = <optimized out>
pd = 0x7f8b50fca700
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140236335916800, 2993067626498392730, 140236637118880, 140236335917504, 140236641488960, 7, -2981626096068820326, -2981586388538588518}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#4 0x00007f8b625bca7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
No locals.
#5 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 12 (Thread 0x7f8b507c9700 (LWP 6722)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
No locals.
#1 0x000000000042739b in sd_cond_wait (mutex=<optimized out>, cond=0x1665ef8) at ../include/util.h:371
No locals.
#2 worker_routine (arg=0x1665e80) at work.c:337
wi = 0x1665e80
work = <optimized out>
tid = 6722
__func__ = "worker_routine"
#3 0x00007f8b62effb50 in start_thread (arg=<optimized out>) at pthread_create.c:304
__res = <optimized out>
pd = 0x7f8b507c9700
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140236327524096, 2993067626498392730, 140236637118880, 140236327524800, 140236641488960, 7, -2981624996020321638, -2981586388538588518}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#4 0x00007f8b625bca7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
No locals.
#5 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 11 (Thread 0x7f8b4ffc8700 (LWP 6723)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
No locals.
#1 0x000000000042739b in sd_cond_wait (mutex=<optimized out>, cond=0x1666138) at ../include/util.h:371
No locals.
#2 worker_routine (arg=0x16660c0) at work.c:337
wi = 0x16660c0
work = <optimized out>
tid = 6723
__func__ = "worker_routine"
#3 0x00007f8b62effb50 in start_thread (arg=<optimized out>) at pthread_create.c:304
__res = <optimized out>
pd = 0x7f8b4ffc8700
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140236319131392, 2993067626498392730, 140236637118880, 140236319132096, 140236641488960, 7, -2981632692064845158, -2981586388538588518}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#4 0x00007f8b625bca7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
No locals.
#5 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 10 (Thread 0x7f8b4f7c7700 (LWP 6724)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
No locals.
#1 0x000000000042739b in sd_cond_wait (mutex=<optimized out>, cond=0x1672718) at ../include/util.h:371
No locals.
#2 worker_routine (arg=0x16726a0) at work.c:337
wi = 0x16726a0
work = <optimized out>
tid = 6724
__func__ = "worker_routine"
#3 0x00007f8b62effb50 in start_thread (arg=<optimized out>) at pthread_create.c:304
__res = <optimized out>
pd = 0x7f8b4f7c7700
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140236310738688, 2993067626498392730, 140236637118880, 140236310739392, 140236641488960, 7, -2981631592016346470, -2981586388538588518}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#4 0x00007f8b625bca7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
No locals.
#5 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 9 (Thread 0x7f8b4efc6700 (LWP 6725)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
No locals.
#1 0x000000000042739b in sd_cond_wait (mutex=<optimized out>, cond=0x1672958) at ../include/util.h:371
No locals.
#2 worker_routine (arg=0x16728e0) at work.c:337
wi = 0x16728e0
work = <optimized out>
tid = 6725
__func__ = "worker_routine"
#3 0x00007f8b62effb50 in start_thread (arg=<optimized out>) at pthread_create.c:304
__res = <optimized out>
pd = 0x7f8b4efc6700
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140236302345984, 2993067626498392730, 140236637118880, 140236302346688, 140236641488960, 7, -2981630491967847782, -2981586388538588518}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#4 0x00007f8b625bca7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
No locals.
#5 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 8 (Thread 0x7f8b4e3c4700 (LWP 7024)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
No locals.
#1 0x000000000042739b in sd_cond_wait (mutex=<optimized out>, cond=0x1665a78) at ../include/util.h:371
No locals.
#2 worker_routine (arg=0x1665a00) at work.c:337
wi = 0x1665a00
work = <optimized out>
tid = 7024
__func__ = "worker_routine"
#3 0x00007f8b62effb50 in start_thread (arg=<optimized out>) at pthread_create.c:304
__res = <optimized out>
pd = 0x7f8b4e3c4700
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140236289754880, 2993067626498392730, 140236637118880, 140236289755584, 140236641488960, 7, -2981629941138292070, -2981586388538588518}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#4 0x00007f8b625bca7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
No locals.
#5 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 7 (Thread 0x7f8b4d7c2700 (LWP 7034)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
No locals.
#1 0x000000000042739b in sd_cond_wait (mutex=<optimized out>, cond=0x16653b8) at ../include/util.h:371
No locals.
#2 worker_routine (arg=0x1665340) at work.c:337
wi = 0x1665340
work = <optimized out>
tid = 7034
__func__ = "worker_routine"
#3 0x00007f8b62effb50 in start_thread (arg=<optimized out>) at pthread_create.c:304
__res = <optimized out>
pd = 0x7f8b4d7c2700
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140236277163776, 2993067626498392730, 140236637118880, 140236277164480, 140236641488960, 7, -2981635987378503014, -2981586388538588518}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#4 0x00007f8b625bca7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
No locals.
#5 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 6 (Thread 0x7f8b47fff700 (LWP 7035)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
No locals.
#1 0x000000000042739b in sd_cond_wait (mutex=<optimized out>, cond=0x1665838) at ../include/util.h:371
No locals.
#2 worker_routine (arg=0x16657c0) at work.c:337
wi = 0x16657c0
work = <optimized out>
tid = 7035
__func__ = "worker_routine"
#3 0x00007f8b62effb50 in start_thread (arg=<optimized out>) at pthread_create.c:304
__res = <optimized out>
pd = 0x7f8b47fff700
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140236185138944, 2993067626498392730, 140236637118880, 140236185139648, 140236641488960, 7, -2981650296598920550, -2981586388538588518}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#4 0x00007f8b625bca7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
No locals.
#5 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 5 (Thread 0x7f8b477fe700 (LWP 7036)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
No locals.
#1 0x000000000042739b in sd_cond_wait (mutex=<optimized out>, cond=0x16653b8) at ../include/util.h:371
No locals.
#2 worker_routine (arg=0x1665340) at work.c:337
wi = 0x1665340
work = <optimized out>
tid = 7036
__func__ = "worker_routine"
#3 0x00007f8b62effb50 in start_thread (arg=<optimized out>) at pthread_create.c:304
__res = <optimized out>
pd = 0x7f8b477fe700
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140236176746240, 2993067626498392730, 140236637118880, 140236176746944, 140236641488960, 7, -2981649196550421862, -2981586388538588518}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#4 0x00007f8b625bca7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
No locals.
#5 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 4 (Thread 0x7f8b46ffd700 (LWP 7037)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
No locals.
#1 0x000000000042739b in sd_cond_wait (mutex=<optimized out>, cond=0x16653b8) at ../include/util.h:371
No locals.
#2 worker_routine (arg=0x1665340) at work.c:337
wi = 0x1665340
work = <optimized out>
tid = 7037
__func__ = "worker_routine"
#3 0x00007f8b62effb50 in start_thread (arg=<optimized out>) at pthread_create.c:304
__res = <optimized out>
pd = 0x7f8b46ffd700
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140236168353536, 2993067626498392730, 140236637118880, 140236168354240, 140236641488960, 7, -2981648096501923174, -2981586388538588518}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#4 0x00007f8b625bca7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
No locals.
#5 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 3 (Thread 0x7f8b4cfc1700 (LWP 7045)):
#0 0x00007f8b6258d7fd in __libc_waitpid (pid=7113, stat_loc=<optimized out>, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:41
_a3 = 0
_a1 = 7113
resultvar = <optimized out>
_a4 = 0
_a2 = 140236268490924
oldtype = 0
result = <optimized out>
#1 0x00007f8b62521c99 in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:149
__result = -512
_buffer = {__routine = 0x7f8b62521ff0 <cancel_handler>, __arg = 0x7f8b4cf7d0a8, __canceltype = 1, __prev = 0x0}
_avail = 1
status = <optimized out>
save = <optimized out>
pid = 7113
sa = {__sigaction_handler = {sa_handler = 0x1, sa_sigaction = 0x1}, sa_mask = {__val = {65536, 0 <repeats 15 times>}}, sa_flags = 0, sa_restorer = 0x7f8b4cf7d120}
omask = {__val = {16896, 6701, 140236268766924, 23484968, 91738, 140236626789651, 206158430256, 140236268491016, 140236268490800, 0, 91738, 23485048, 0, 140236268766924, 6701, 140236268491104}}
#2 0x00007f8b62521fd0 in __libc_system (line=<optimized out>) at ../sysdeps/posix/system.c:190
oldtype = 0
result = <optimized out>
#3 0x000000000042168e in gdb_cmd (cmd=0x436c32 "thread apply all where full") at logger.c:772
time_str = "Jan 30 10:07:26 ", '\000' <repeats 239 times>
cmd_str = "gdb -nw /home/mitake/github/sheepdog.git/sheep/sheep 6701 -batch >/dev/null 2>&1 -ex 'set logging on' -ex 'echo \\n' -ex 'echo ==\\n' -ex 'echo == Jan 30 10:07:26 \\n' -ex 'echo == program: /home/mitake/"...
ti = 1391044046
tm = {tm_sec = 26, tm_min = 7, tm_hour = 10, tm_mday = 30, tm_mon = 0, tm_year = 114, tm_wday = 4, tm_yday = 29, tm_isdst = 0, tm_gmtoff = 32400, tm_zone = 0x16663a0 "JST"}
#4 0x00000000004224fc in dump_stack_frames () at logger.c:786
No locals.
#5 sd_backtrace () at logger.c:836
addrs = {0x4223cd, 0x4059a8, 0x7f8b62f08030, 0x4273a8, 0x7f8b62effb50, 0x7f8b625bca7d, 0x0 <repeats 1018 times>}
i = <optimized out>
n = <optimized out>
__func__ = "sd_backtrace"
#6 0x00000000004059a8 in crash_handler (signo=11) at sheep.c:269
__func__ = "crash_handler"
#7 <signal handler called>
No symbol table info available.
#8 list_del (entry=0x1665a) at ../include/list.h:105
No locals.
#9 worker_routine (arg=0x1665a00) at work.c:344
wi = 0x1665a00
work = 0x1665a
tid = 7045
__func__ = "worker_routine"
#10 0x00007f8b62effb50 in start_thread (arg=<optimized out>) at pthread_create.c:304
__res = <optimized out>
pd = 0x7f8b4cfc1700
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140236268771072, 2993067626498392730, 140236637118880, 140236268771776, 140236641488960, 7, -2981634938869611878, -2981586388538588518}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#11 0x00007f8b625bca7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
No locals.
#12 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 2 (Thread 0x7f8b467fc700 (LWP 7046)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
No locals.
#1 0x000000000042739b in sd_cond_wait (mutex=<optimized out>, cond=0x1665a78) at ../include/util.h:371
No locals.
#2 worker_routine (arg=0x1665a00) at work.c:337
wi = 0x1665a00
work = <optimized out>
tid = 7046
__func__ = "worker_routine"
#3 0x00007f8b62effb50 in start_thread (arg=<optimized out>) at pthread_create.c:304
__res = <optimized out>
pd = 0x7f8b467fc700
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140236159960832, 2993067626498392730, 140236637118880, 140236159961536, 140236641488960, 7, -2981646996453424486, -2981586388538588518}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#4 0x00007f8b625bca7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
No locals.
#5 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 1 (Thread 0x7f8b6330bd80 (LWP 6701)):
#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
No locals.
#1 0x00007f8b62f02339 in _L_lock_926 () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#2 0x00007f8b62f0215b in __pthread_mutex_lock (mutex=mutex at entry=0x1665aa8) at pthread_mutex_lock.c:61
ignore1 = <optimized out>
ignore2 = 23485096
ignore3 = -512
__PRETTY_FUNCTION__ = "__pthread_mutex_lock"
type = <optimized out>
#3 0x0000000000427820 in sd_mutex_lock (mutex=0x1665aa8) at ../include/util.h:307
ret = <optimized out>
#4 queue_work (q=0x1665ad0, work=0x7f8b48007f78) at work.c:271
wi = 0x1665a00
#5 0x0000000000412d77 in queue_recovery_work (rinfo=rinfo at entry=0x7f8b48007c90) at recovery.c:1142
rw = <optimized out>
rlw = <optimized out>
row = <optimized out>
__func__ = "queue_recovery_work"
#6 0x00000000004150ae in recover_next_object (rinfo=0x7f8b48007c90) at recovery.c:828
No locals.
#7 0x0000000000415393 in finish_object_list (work=0x1) at recovery.c:932
i = 1
rinfo = 0x7f8b48007c90
nr_threads = 2
#8 0x0000000000427613 in worker_thread_request_done (fd=<optimized out>, events=<optimized out>, data=<optimized out>) at work.c:303
__n294 = 0x16657c0
wi = 0x1665a00
work = <optimized out>
list = {n = {next = 0x7fff4f1c03d0, prev = 0x7fff4f1c03d0}}
#9 0x0000000000420c0f in do_event_loop (timeout=timeout at entry=-1, sort_with_prio=sort_with_prio at entry=false) at event.c:210
ei = <optimized out>
i = <optimized out>
nr = 1
__func__ = "do_event_loop"
#10 0x00000000004210a7 in event_loop (timeout=timeout at entry=-1) at event.c:220
No locals.
#11 0x0000000000404d5a in main (argc=13, argv=<optimized out>) at sheep.c:931
ch = <optimized out>
longindex = 0
ret = <optimized out>
port = 7000
io_port = 7000
nr_vnodes = 128
dirp = <optimized out>
short_options = 0x94c200 "b:c:Dghi:j:l:np:P:r:uvw:y:z:"
dir = <optimized out>
p = 0x7fff4f1c1b9a ""
pid_file = 0x0
bindaddr = 0x0
log_path = "/tmp/sheepdog/033/0/sheep.log", '\000' <repeats 59 times>, "M8\022c\213\177\000\000\000\000\000\000\000\000\000\000_\334-b\213\177\000\000\000\000\000\000\000\000\000\000B\310\021c\213\177", '\000' <repeats 70 times>, "\001", '\000' <repeats 131 times>"\257, \265\021c\213\177", '\000' <repeats 42 times>, "M8\022c\213\177\000\000\000\000\000\000\000\000\000\000\343}Ob\213\177\000\000\000\000\000\000\000\000\000\000B\310\021c\213\177\000\000\324H\000\000\000\000\000\000\324H", '\000' <repeats 14 times>, "\005\000\000\000\000\000\000\000\000@ \000\000\000\000\000\000P \000\000\000\000\000\320L \000\000\000\000\000\330L \000\000\000\000\000\000@\000\000"...
argp = <optimized out>
explicit_addr = true
zone = 0
cdrv = <optimized out>
long_options = 0x94c320
http_options = 0x0
sheep_info = {port = 7000}
logdir_st = {st_dev = 0, st_ino = 0, st_nlink = 0, st_mode = 0, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 0, st_blksize = 0, st_blocks = 0, st_atim = {tv_sec = 0, tv_nsec = 0}, st_mtim = {tv_sec = 0, tv_nsec = 0}, st_ctim = {tv_sec = 0, tv_nsec = 0}, __unused = {0, 0, 0}}
log_dst_type = <optimized out>
__func__ = "main"
==
== Jan 30 10:07:26
== program: /home/mitake/github/sheepdog.git/sheep/sheep
== command: p __sys
==
$1 = {cdrv = 0x641f00, cdrv_option = 0x0, this_node = {rb = {rb_parent_color = 0, rb_right = 0x0, rb_left = 0x0}, nid = {addr = '\000' <repeats 12 times>, "\177\000\000\001", port = 7000, io_addr = '\000' <repeats 15 times>, io_port = 0, pad = "\000\000\000"}, nr_vnodes = 128, zone = 0, space = 1693110272}, cinfo = {proto_ver = 9 '\t', disable_recovery = 0 '\000', nr_nodes = 5, epoch = 4, ctime = 5974488637973210360, flags = 0, nr_copies = 3 '\003', copy_policy = 0 '\000', status = SD_STATUS_OK, __pad = 0, store = "plain\000\000\000\000\000\000\000\000\000\000", nodes = {{rb = {rb_parent_color = 140236185174177, rb_right = 0x0, rb_left = 0x0}, nid = {addr = '\000' <repeats 12 times>, "\177\000\000\001", port = 7000, io_addr = '\000' <repeats 15 times>, io_port = 0, pad = "\000\000\000"}, nr_vnodes = 128, zone = 0, space = 1693110272}, {rb = {rb_parent_color = 1, rb_right = 0x7f8b48008160, rb_left = 0x7f8b48008040}, nid = {addr = '\000' <repeats 12 times>, "\177\000\000\001", port = 7001, io_addr = '\000' <repeats 15 times>, io_port = 0, pad = "\000\000\000"}, nr_vnodes = 128, zone = 1, space = 1693089792}, {rb = {rb_parent_color = 140236185174368, rb_right = 0x0, rb_left = 0x0}, nid = {addr = '\000' <repeats 12 times>, "\177\000\000\001", port = 7002, io_addr = '\000' <repeats 15 times>, io_port = 0, pad = "\000\000\000"}, nr_vnodes = 128, zone = 2, space = 1693069312}, {rb = {rb_parent_color = 140236185174177, rb_right = 0x7f8b480081c0, rb_left = 0x7f8b48008100}, nid = {addr = '\000' <repeats 12 times>, "\177\000\000\001", port = 7003, io_addr = '\000' <repeats 15 times>, io_port = 0, pad = "\000\000\000"}, nr_vnodes = 128, zone = 3, space = 1693048832}, {rb = {rb_parent_color = 140236185174368, rb_right = 0x0, rb_left = 0x0}, nid = {addr = '\000' <repeats 12 times>, "\177\000\000\001", port = 7004, io_addr = '\000' <repeats 15 times>, io_port = 0, pad = "\000\000\000"}, nr_vnodes = 128, zone = 4, space = 1693028352}, {rb = {rb_parent_color = 23632816, rb_right = 0x0, rb_left = 0x0}, nid = {addr = '\000' <repeats 12 times>, "\177\000\000\001", port = 7005, io_addr = '\000' <repeats 15 times>, io_port = 0, pad = "\000\000\000"}, nr_vnodes = 128, zone = 5, space = 1693003776}, {rb = {rb_parent_color = 23605504, rb_right = 0x0, rb_left = 0x0}, nid = {addr = '\000' <repeats 12 times>, "\177\000\000\001", port = 7006, io_addr = '\000' <repeats 15 times>, io_port = 0, pad = "\000\000\000"}, nr_vnodes = 128, zone = 6, space = 1692983296}, {rb = {rb_parent_color = 23618544, rb_right = 0x0, rb_left = 0x0}, nid = {addr = '\000' <repeats 12 times>, "\177\000\000\001", port = 7007, io_addr = '\000' <repeats 15 times>, io_port = 0, pad = "\000\000\000"}, nr_vnodes = 128, zone = 7, space = 1692966912}, {rb = {rb_parent_color = 0, rb_right = 0x0, rb_left = 0x0}, nid = {addr = '\000' <repeats 15 times>, port = 0, io_addr = '\000' <repeats 15 times>, io_port = 0, pad = "\000\000\000"}, nr_vnodes = 0, zone = 0, space = 0} <repeats 6136 times>}}, disk_space = 1693110272, vdi_inuse = {0 <repeats 127148 times>, 137438953472, 0 <repeats 134995 times>}, local_req_efd = 11, local_req_lock = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, local_req_queue = {n = {next = 0x8bc388, prev = 0x8bc388}}, req_wait_queue = {n = {next = 0x8bc398, prev = 0x8bc398}}, nr_outstanding_reqs = 1, gateway_only = false, nosync = true, net_wqueue = 0x1665410, gateway_wqueue = 0x1665650, io_wqueue = 0x1665890, deletion_wqueue = 0x1665d10, recovery_wqueue = 0x1665ad0, recovery_notify_wqueue = 0x0, block_wqueue = 0x1665f50, oc_reclaim_wqueue = 0x0, oc_push_wqueue = 0x0, md_wqueue = 0x1666190, areq_wqueue = 0x1672770, enable_object_cache = false, object_cache_size = 0, object_cache_directio = false, use_journal = {val = 0}, backend_dio = false, upgrade = false, stat = {r = {gway_active_nr = 0, peer_active_nr = 0, gway_total_nr = 128, peer_total_nr = 8, gway_total_rx = 4211876, gway_total_tx = 214152240, peer_total_rx = 3072, peer_total_tx = 8388608, gway_total_remove_nr = 0, gway_total_read_nr = 77, gway_total_write_nr = 51, gway_total_flush_nr = 0, peer_total_remove_nr = 0, peer_total_read_nr = 2, peer_total_write_nr = 6}}}
More information about the sheepdog
mailing list