<div dir="ltr">Hi,<div><br></div><div>I did some I/O stress testing for sheepdog. </div><div style>For each node, I started sheep in this command:</div><div style>sheep -b 0.0.0.0 -y 10.0.0.XX -p 7000 -j dir=/sheep/journal size=3000 -D -w size=40000 dir=/sheep/object_cache -c zookeeper:<a href="http://10.0.0.10:2181">10.0.0.10:2181</a>,timeout=30s /sheep/state /sheep/disk1,/sheep/disk2 -P /sheep/state/sheep.pid<br>
</div><div style><br></div><div style>Notice: I used both /sheep/disk1 and /sheep/disk2 </div><div style><br></div><div>Before testing, the status of node looked like this:</div><div><span style="font-family:arial,sans-serif;font-size:12.727272033691406px">> collie </span><span class="" style="background-color:rgb(255,255,204);font-family:arial,sans-serif;font-size:12.727272033691406px">node</span><span style="font-family:arial,sans-serif;font-size:12.727272033691406px"> </span><span class="" style="background-color:rgb(255,255,204);font-family:arial,sans-serif;font-size:12.727272033691406px">info</span><br style="font-family:arial,sans-serif;font-size:12.727272033691406px">
<span style="font-family:arial,sans-serif;font-size:12.727272033691406px">Id Size Used Use%</span><br style="font-family:arial,sans-serif;font-size:12.727272033691406px"><span style="font-family:arial,sans-serif;font-size:12.727272033691406px">0 0.0 MB 0.0 MB 0%</span><br style="font-family:arial,sans-serif;font-size:12.727272033691406px">
<span style="font-family:arial,sans-serif;font-size:12.727272033691406px">1 57 GB 14 GB 24%</span><br style="font-family:arial,sans-serif;font-size:12.727272033691406px"><span style="font-family:arial,sans-serif;font-size:12.727272033691406px">2 136 GB 30 GB 22%</span><br style="font-family:arial,sans-serif;font-size:12.727272033691406px">
<span style="font-family:arial,sans-serif;font-size:12.727272033691406px">3 57 GB 16 GB 27%</span><br style="font-family:arial,sans-serif;font-size:12.727272033691406px"><span style="font-family:arial,sans-serif;font-size:12.727272033691406px">Total 250 GB 60 GB 23%</span><br>
<div><br></div><div style>I started a vm in node1 and performed sequential write on it (~30GB). The sheep on the node1 exited unexpected when no space left on device. I tried to check my node1 disk status:</div><div style>
<div style># df -h</div><div><div>Filesystem Size Used Avail Use% Mounted on</div><div>/dev/mapper/vg1-sys 99G 28G 67G 30% /</div><div>tmpfs 16G 0 16G 0% /dev/shm</div><div>/dev/sda1 985M 45M 890M 5% /boot</div>
<div>/dev/mapper/vg1-home 2.0G 68M 1.9G 4% /home</div><div>/dev/mapper/vg1-sheep_disk1</div><div> 40G 40G 0 100% /sheep/disk1</div><div>/dev/mapper/vg1-sheep_journal</div><div> 4.0G 3.0G 793M 80% /sheep/journal</div>
<div>/dev/mapper/vg1-sheep_obj_cache</div><div> 40G 420M 37G 2% /sheep/object_cache</div><div>/dev/mapper/vg1-var 32G 286M 30G 1% /var</div></div><div><br></div><div style>As it shown , disk1 is mount a single exclusive partition, disk2 is assign to "/" by default. </div>
<div style>To my best knowledge, when disk1 was full, it should not raise an error and make sheep exited (see attached log), since sheep was aware of the size of disk1 and when it's full it should switch to write disk2. At least it should not throw an error and caused an unexpected exit.</div>
<div style><br></div><div style>I wonder why this error occurs?</div><div style><br></div><div style>Thanks,</div><div style><br></div><div style>--Hongyi</div><div style><br></div><div>================================================================</div>
</div><div style>sheep.log on the node which sheep exited unexpectedly.</div><div><br></div><div><div>May 24 04:14:44 [io 17463] prealloc(284) failed to preallocate space, No space left on device</div><div>May 24 04:14:44 [io 17463] err_to_sderr(78) diskfull, oid=65e2f7000022a4</div>
<div>May 24 04:14:44 [io 17448] prealloc(284) failed to preallocate space, No space left on device</div><div>May 24 04:14:44 [io 17448] err_to_sderr(78) diskfull, oid=65e2f7000022a6</div><div>May 24 04:14:44 [gway 17015] default_create_and_write(342) failed to write object. No space left on device</div>
<div>May 24 04:14:44 [gway 17015] err_to_sderr(78) diskfull, oid=6ebf780000253b</div><div>May 24 04:14:44 [gway 17015] gateway_forward_request(305) fail to write local 6ebf780000253b, Server has no space for new objects</div>
<div>May 24 04:14:44 [io 17446] default_create_and_write(342) failed to write object. No space left on device</div><div>May 24 04:14:44 [io 17446] err_to_sderr(78) diskfull, oid=65e2f7000022a5</div><div>May 24 04:14:45 [io 17472] prealloc(284) failed to preallocate space, No space left on device</div>
<div>May 24 04:14:45 [io 17472] err_to_sderr(78) diskfull, oid=65e2f7000022ad</div><div>May 24 04:14:46 [io 17469] prealloc(284) failed to preallocate space, No space left on device</div><div>May 24 04:14:46 [io 17469] err_to_sderr(78) diskfull, oid=65e2f7000022ae</div>
<div>May 24 04:14:46 [io 17462] default_create_and_write(342) failed to write object. No space left on device</div><div>May 24 04:14:46 [io 17462] err_to_sderr(78) diskfull, oid=65e2f7000022aa</div><div>May 24 04:14:46 [gway 17356] default_create_and_write(342) failed to write object. No space left on device</div>
<div>May 24 04:14:46 [gway 17356] err_to_sderr(78) diskfull, oid=6ebf780000253f</div><div>May 24 04:14:46 [gway 17356] gateway_forward_request(305) fail to write local 6ebf780000253f, Server has no space for new objects</div>
<div>May 24 04:14:46 [gway 17011] default_create_and_write(342) failed to write object. No space left on device</div><div>May 24 04:14:46 [gway 17011] err_to_sderr(78) diskfull, oid=6ebf7800002540</div><div>May 24 04:14:46 [gway 17011] gateway_forward_request(305) fail to write local 6ebf7800002540, Server has no space for new objects</div>
<div>May 24 04:14:46 [oc_push 16832] push_cache_object(467) failed to push object Server has no space for new objects</div><div>May 24 04:14:46 [oc_push 16832] do_push_object(837) PANIC: push failed but should never fail</div>
<div>May 24 04:14:46 [oc_push 16832] crash_handler(181) sheep exits unexpectedly (Aborted).</div><div>May 24 04:14:46 [oc_push 16832] sd_backtrace(847) sheep() [0x4045b7]</div><div>May 24 04:14:46 [oc_push 16534] push_cache_object(467) failed to push object Server has no space for new objects</div>
<div>May 24 04:14:46 [oc_push 16534] do_push_object(837) PANIC: push failed but should never fail</div><div>May 24 04:14:52 [main] crash_handler(487) sheep pid 8130 exited unexpectedly.</div></div><div>=====================================================================</div>
</div></div>