[sheepdog-users] node disconnect from the cluster

[Valerio Pachera]

df -h
/dev/sdc1                 466G  333G    133G  72% /mnt/wd_WCAYUEP99298
/dev/sdd1                 1,9T  985G    879G  53% /mnt/wd_WCAWZ1588874


sheep.log
Jun 15 04:53:00 [gway 23823] wait_forward_request(211) fail
2e51900006c14, Server has no space for new objects
Jun 15 04:53:00 [oc_push 23583] push_cache_object(471) failed to push
object Server has no space for new objects
Jun 15 04:53:00 [oc_push 23583] do_push_object(841) PANIC: push failed
but should never fail
Jun 15 04:53:00 [oc_push 23583] crash_handler(180) sheep exits
unexpectedly (Aborted).
Jun 15 04:53:00 [oc_push 23583] sd_backtrace(833) sheep.c:182: crash_handler
Jun 15 04:53:00 [oc_push 23583] sd_backtrace(847)
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf02f) [0x7fd47d28b02f]
Jun 15 04:53:00 [oc_push 23583] sd_backtrace(847)
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x34) [0x7fd47c897474]
Jun 15 04:53:00 [oc_push 23583] sd_backtrace(847)
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17f) [0x7fd47c89a6ef]
Jun 15 04:53:00 [oc_push 23583] sd_backtrace(833) object_cache.c:841:
do_push_object
Jun 15 04:53:00 [oc_push 23583] sd_backtrace(833) work.c:243: worker_routine
Jun 15 04:53:00 [oc_push 23583] sd_backtrace(847)
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b4f) [0x7fd47d282b4f]
Jun 15 04:53:00 [oc_push 23583] sd_backtrace(847)
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6c) [0x7fd47c93fa7c]
Jun 15 04:53:00 [oc_push 23583] __dump_stack_frames(743) cannot find gdb
Jun 15 04:53:00 [oc_push 23583] __sd_dump_variable(693) cannot find gdb
Jun 15 04:53:00 [main] crash_handler(487) sheep pid 3824 exited unexpectedly.


Sheep.log says there was no space left but 'df -h' show there's space.
What do you think?

PS:

This is node info without this host

# collie node info
Id      Size    Used    Avail   Use%
 0      2.3 TB  1.5 TB  797 GB   65%
 1      2.3 TB  1.6 TB  646 GB   72%
 2      0.0 MB  0.0 MB  0.0 MB    0%
Total   4.5 TB  3.1 TB  1.4 TB   68%