<div dir="ltr"><div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">If 5 sheepdog nodes are running with cache, and more than 10 vms running on each node.</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">
<br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">I mount a tmpfs to /cache directory, and start sheep with:</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">
<br></div><div style="text-align:-webkit-auto"><font color="#000000" face="Tahoma" size="3">sheep -l level=debug -n /home/admin/sheepdogmetadata,/disk1/sheepdogstoredata,/disk2/sheepdogstoredata,/disk3/sheepdogstoredata,/disk4/sheepdogstoredata,/disk5/sheepdogstoredata,/disk7/sheepdogstoredata,/disk8/sheepdogstoredata,/disk9/sheepdogstoredata -w size=20G dir=/cache -b 0.0.0.0 -y **.**.**.** -c zookeeper:**.**.**.**:2181</font><br>
</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">There is a possibility that all object push threads are running do_background_push work, and no threads is running do_push_object work.</div>
<div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">In my test environment, this occurs:</div>
<div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><div><span style="text-align:-webkit-auto">[1] 13:09:30 [SUCCESS] vmsecdomainhost1</span><br style="text-align:-webkit-auto">
<span style="text-align:-webkit-auto">Name Tag Total Dirty Clean</span><br style="text-align:-webkit-auto"><span style="text-align:-webkit-auto">win7_type4_node8.img 4.7 GB 4.7 GB 4.0 MB</span><br style="text-align:-webkit-auto">
<span style="text-align:-webkit-auto">standard.img images 0.0 MB 0.0 MB 0.0 MB</span><br style="text-align:-webkit-auto"><span style="text-align:-webkit-auto">win7_type4_node1.img 4.8 GB 4.8 GB 28 MB</span><br style="text-align:-webkit-auto">
<span style="text-align:-webkit-auto">win7_type4_node10.img 5.0 GB 4.9 GB 32 MB</span><br style="text-align:-webkit-auto"><span style="text-align:-webkit-auto">win7_type4_node2.img 4.7 GB 4.6 GB 68 MB</span><br style="text-align:-webkit-auto">
<span style="text-align:-webkit-auto">win7_type4_node3.img 4.7 GB 4.7 GB 4.0 MB</span><br style="text-align:-webkit-auto"><span style="text-align:-webkit-auto">win7_type4_node6.img 4.8 GB 4.7 GB 40 MB</span><br style="text-align:-webkit-auto">
<span style="text-align:-webkit-auto">win7_type4_node4.img 4.8 GB 4.7 GB 20 MB</span><br style="text-align:-webkit-auto"><span style="text-align:-webkit-auto">win7_type4_node7.img 4.8 GB 4.8 GB 24 MB</span><br style="text-align:-webkit-auto">
<span style="text-align:-webkit-auto">win7_type4_node9.img 4.7 GB 4.7 GB 32 MB</span><br style="text-align:-webkit-auto"><span style="text-align:-webkit-auto">win7_type4_node5.img 4.2 GB 4.2 GB 8.0 MB</span><div style="text-align:-webkit-auto">
<br></div><div style="text-align:-webkit-auto">Cache size 20 GB, used 47 GB, non-directio</div></div></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">
<br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">I found that, 7 object push threads are working with work_queue "oc_push", and their call stacks are:</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">
<br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">Thread 37 (Thread 0x7f3c2a1fc700 (LWP 116747)):<br>
#0 0x0000003916eda37d in read () from /lib64/libc.so.6<br>
#1 0x0000003916ee7a1e in eventfd_read () from /lib64/libc.so.6<br>
#2 0x000000000042a89d in eventfd_xread ()<br>
#3 0x0000000000419acb in object_cache_push ()<br><b>#4 0x0000000000419b83 in do_background_push ()</b><br>
#5 0x000000000042e56a in worker_routine ()<br>
#6 0x0000003917207851 in start_thread () from /lib64/libpthread.so.0<br>
#7 0x0000003916ee767d in clone () from /lib64/libc.so.6</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br>
Thread 36 (Thread 0x7f3c2abfd700 (LWP 116775)):<br>
#0 0x0000003916eda37d in read () from /lib64/libc.so.6<br>
#1 0x0000003916ee7a1e in eventfd_read () from /lib64/libc.so.6<br>
#2 0x000000000042a89d in eventfd_xread ()<br>
#3 0x0000000000419acb in object_cache_push ()<br><b>#4 0x0000000000419b83 in do_background_push ()</b><br>
#5 0x000000000042e56a in worker_routine ()<br>
#6 0x0000003917207851 in start_thread () from /lib64/libpthread.so.0<br>
#7 0x0000003916ee767d in clone () from /lib64/libc.so.6</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br>
Thread 35 (Thread 0x7f3b5d7fb700 (LWP 116889)):<br>
#0 0x0000003916eda37d in read () from /lib64/libc.so.6<br>
#1 0x0000003916ee7a1e in eventfd_read () from /lib64/libc.so.6<br>
#2 0x000000000042a89d in eventfd_xread ()<br>
#3 0x0000000000419acb in object_cache_push ()<br><b>#4 0x0000000000419b83 in do_background_push ()</b><br>
#5 0x000000000042e56a in worker_routine ()<br>
#6 0x0000003917207851 in start_thread () from /lib64/libpthread.so.0<br>
#7 0x0000003916ee767d in clone () from /lib64/libc.so.6</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">
Thread 34 (Thread 0x7f3b4ffff700 (LWP 116891)):<br>
#0 0x0000003916eda37d in read () from /lib64/libc.so.6<br>
#1 0x0000003916ee7a1e in eventfd_read () from /lib64/libc.so.6<br>
#2 0x000000000042a89d in eventfd_xread ()<br>
#3 0x0000000000419acb in object_cache_push ()<br><b>#4 0x0000000000419b83 in do_background_push ()</b><br>
#5 0x000000000042e56a in worker_routine ()<br>
#6 0x0000003917207851 in start_thread () from /lib64/libpthread.so.0<br>
#7 0x0000003916ee767d in clone () from /lib64/libc.so.6</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br>
Thread 33 (Thread 0x7f3ac8dfa700 (LWP 117040)):<br>
#0 0x0000003916eda37d in read () from /lib64/libc.so.6<br>
#1 0x0000003916ee7a1e in eventfd_read () from /lib64/libc.so.6<br>
#2 0x000000000042a89d in eventfd_xread ()<br>
#3 0x0000000000419acb in object_cache_push ()<br><b>#4 0x0000000000419b83 in do_background_push ()</b><br>
#5 0x000000000042e56a in worker_routine ()<br>
#6 0x0000003917207851 in start_thread () from /lib64/libpthread.so.0<br>
#7 0x0000003916ee767d in clone () from /lib64/libc.so.6</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br>
Thread 32 (Thread 0x7f3ac83f9700 (LWP 117041)):<br>
#0 0x0000003916eda37d in read () from /lib64/libc.so.6<br>
#1 0x0000003916ee7a1e in eventfd_read () from /lib64/libc.so.6<br>
#2 0x000000000042a89d in eventfd_xread ()<br>
#3 0x0000000000419acb in object_cache_push ()<br><b>#4 0x0000000000419b83 in do_background_push ()</b><br>
#5 0x000000000042e56a in worker_routine ()<br>
#6 0x0000003917207851 in start_thread () from /lib64/libpthread.so.0<br>
#7 0x0000003916ee767d in clone () from /lib64/libc.so.6</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">
Thread 31 (Thread 0x7f3ac65f6700 (LWP 117044)):<br>
#0 0x0000003916eda37d in read () from /lib64/libc.so.6<br>
#1 0x0000003916ee7a1e in eventfd_read () from /lib64/libc.so.6<br>
#2 0x000000000042a89d in eventfd_xread ()<br>
#3 0x0000000000419acb in object_cache_push ()<br><b>#4 0x0000000000419b83 in do_background_push ()</b><br>
#5 0x000000000042e56a in worker_routine ()<br>
#6 0x0000003917207851 in start_thread () from /lib64/libpthread.so.0<br>
#7 0x0000003916ee767d in clone () from /lib64/libc.so.6</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">
No threads are pushing objects, so no object_cache_push work finished.</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">
<br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">In gdb, we can see the information of each object cache in object_cache_push:</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">
<br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">vid = 9627038, push_count = 26, dirty_count = 150, total_count = 154</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">
vid = 3508964, push_count = 22, dirty_count = 1456, total_count = 1464</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">vid = 360229, push_count = 18, dirty_count = 1437, total_count = 1444</div>
<div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">vid = 9678955, push_count = 34, dirty_count = 1462, total_count = 1470</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">
vid = 9008538, push_count = 17, dirty_count = 1490, total_count = 1493</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">vid = 2383510, push_count = 28, dirty_count = 1494, total_count = 1498</div>
<div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">vid = 16192623, push_count = 19, dirty_count = 1447, total_count = 1451</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">
<br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">push_count is far less than dirty_count, and no threads is doing do_push_object work, so </div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">
<br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">static void do_push_object(struct work *work)</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">
if (uatomic_sub_return(&oc->push_count, 1) == 0)<br>
eventfd_xwrite(oc->push_efd, 1);</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">
will never be kicked.</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">And in </div>
<div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">static bool wq_need_grow(struct wq_info *wi)<br>
{<br>
if (wi->nr_threads < uatomic_read(&wi->nr_queued_work) &&<br>
wi->nr_threads * 2 <= wq_get_roof(wi)) {<br>
wi->tm_end_of_protection = get_msec_time() +<br>
WQ_PROTECTION_PERIOD;<br>
return true;<br>
}<br><br>
return false;<br>
}</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">nr_threads is 7, wq_get_roof(wi) returns 10( 2 * five nodes).</div>
<div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">so no more threads will be created, and all threads are waiting for do_push_object finished.</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">
<br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">Hope that the above information is clearly for everyone.</div>
<div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">
Let's discuss the solution now.</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">The oc_push_wqueue is created with WQ_DYNAMIC: </div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">
<br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">sys->oc_push_wqueue = create_work_queue("oc_push", WQ_DYNAMIC)</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">
<br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">So the roof of threads number will be </div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">
<br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"> case WQ_DYNAMIC:<br>
/* FIXME: 2 * nr_nodes threads. No rationale yet. */<br>
nr = nr_nodes * 2;<br>
break;</div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">There are also other work queue created with WQ_DYNAMIC:</div>
<div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">wq = create_work_queue("vdi check", WQ_DYNAMIC);</div>
<div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">sys->http_wqueue = create_work_queue("http", WQ_DYNAMIC);</div>
<div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">
oc_push created with WQ_UNLIMITED is not rational too. </div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">
<br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:16px"><b>I think that, the nr_threads working with oc_push should be (2 * number of object cache), not (2 * nr_nodes), to ensure that there will be always enougth threads doing do_push_object work.</b></div>
<div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:16px"><b><br></b></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">
With your advises, I wish to submit patches to solve this problem. </div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium"><br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">
<br></div><div style="color:rgb(0,0,0);font-family:Tahoma;text-align:-webkit-auto;font-size:medium">Thanks. </div></div><div><br></div>-- <br><div dir="ltr">Xu Fang<br><br>Beijing,P.R.China<br><br></div>
</div>