Hi again, I wanted to see how sheepdog would behave when it is short in disk space. 1) First as you can see below the "Use%" field is not calculated correctly fire-srv4 ~ # shepherd info -t sheep Id Size Used Use% 0 5G 9G 181% 1 2G 12G 643% 2 8G 6G 83% fire-srv4 ~ # I think it is obvious from the output exactly how wrong the "Use%" gets calculated :) 2) I've created a 15G btrfs on 3 nodes and --copies=2 for the test and wanted to fully fill just one of the nodes and see how sheepdog behaves then. I hoped that once one of the nodes gets out of space it would continue to allocate blocks on the other 2 remaining nodes, but it just got stuck with "aio_read_response 869: I/O error" fire-srv3 ~ # kvm-img convert -f raw -O sheepdog /dev/sys/kvm-img zopa sd_claim 1492: zopa sd_release 1533: zopa sd_close 1478: zopa fire-srv3 ~ # kvm-img convert -f raw -O sheepdog /dev/sys/kvm-img zopa2 sd_claim 1492: zopa2 sd_release 1533: zopa2 sd_close 1478: zopa2 fire-srv3 ~ # kvm-img convert -f raw -O sheepdog /dev/sys/kvm-img zopa3 sd_claim 1492: zopa3 sd_release 1533: zopa3 sd_close 1478: zopa3 fire-srv3 ~ # kvm-img convert -f raw -O sheepdog /dev/sys/kvm-img zopa4 sd_claim 1492: zopa4 aio_read_response 869: I/O error ... this is from the fire-srv3 node on which kvm-img was ruinning: ... Dec 30 14:09:28 localhost collie: store_queue_request(181) 0, 4, /sheepdog/0/100000, 3, 3 Dec 30 14:09:28 localhost collie: store_queue_request(181) 2, 4, /sheepdog/0/100000, 3, 3 Dec 30 14:09:28 localhost collie: store_queue_request(181) 1, 4, /sheepdog/0/100000, 3, 3 Dec 30 14:09:28 localhost collie: store_queue_request(181) 3, 4, /sheepdog/0/100000, 3, 3 Dec 30 14:09:28 localhost collie: store_queue_request(181) 0, 4, /sheepdog/0/100000, 3, 3 Dec 30 14:09:28 localhost collie: store_queue_request(181) 2, 1, /sheepdog/0/100354, 3, 3 Dec 30 14:09:28 localhost collie: store_queue_request(250) 1049428 Dec 30 14:09:28 localhost collie: store_queue_request(254) 1049428 Dec 30 14:09:28 localhost collie: store_queue_request(181) 1, 4, /sheepdog/0/100000, 3, 3 Dec 30 14:09:28 localhost collie: listen_handler(337) accepted a new connection, 258 Dec 30 14:09:28 localhost collie: cluster_queue_request(175) 0xa66c90 19 Dec 30 14:09:28 localhost collie: client_handler(298) closed a connection, 258 Dec 30 14:09:34 localhost collie: listen_handler(337) accepted a new connection, 258 Dec 30 14:09:34 localhost collie: store_queue_request(181) 0, b0, /sheepdog/0/0, 3, 3 Dec 30 14:09:34 localhost collie: client_handler(298) closed a connection, 258 Dec 30 14:09:34 localhost collie: listen_handler(337) accepted a new connection, 258 Dec 30 14:09:34 localhost collie: so_read_vdis(367) 40000 Dec 30 14:09:34 localhost collie: so_read_vdis(386) 0000000000040000-00000000 Dec 30 14:09:34 localhost collie: so_read_vdis(367) 80000 Dec 30 14:09:34 localhost collie: so_read_vdis(386) 0000000000080000-00000000 Dec 30 14:09:34 localhost collie: so_read_vdis(367) c0000 Dec 30 14:09:34 localhost collie: so_read_vdis(386) 00000000000c0000-00000000 Dec 30 14:09:34 localhost collie: so_read_vdis(367) 100000 Dec 30 14:09:34 localhost collie: so_read_vdis(386) 0000000000100000-00000000 Dec 30 14:09:34 localhost collie: so_read_vdis(403) 128 Dec 30 14:09:34 localhost collie: client_handler(298) closed a connection, 258 Dec 30 14:09:49 localhost collie: listen_handler(337) accepted a new connection, 258 Dec 30 14:09:49 localhost collie: store_queue_request(181) 1, b0, /sheepdog/0/0, 3, 3 Dec 30 14:09:49 localhost collie: client_handler(298) closed a connection, 258 Dec 30 14:09:49 localhost collie: listen_handler(337) accepted a new connection, 258 Dec 30 14:09:49 localhost collie: so_read_vdis(367) 40000 Dec 30 14:09:49 localhost collie: so_read_vdis(386) 0000000000040000-00000000 Dec 30 14:09:49 localhost collie: so_read_vdis(367) 80000 Dec 30 14:09:49 localhost collie: so_read_vdis(386) 0000000000080000-00000000 Dec 30 14:09:49 localhost collie: so_read_vdis(367) c0000 Dec 30 14:09:49 localhost collie: so_read_vdis(386) 00000000000c0000-00000000 Dec 30 14:09:49 localhost collie: so_read_vdis(367) 100000 Dec 30 14:09:49 localhost collie: so_read_vdis(386) 0000000000100000-00000000 Dec 30 14:09:49 localhost collie: so_read_vdis(403) 128 Dec 30 14:09:49 localhost collie: client_handler(298) closed a connection, 258 ... Also sheepdog show that zopa4 vdi is in running state in spite of that i've terminated the kvm-img process: fire-srv4 ~ # shepherd info -t vm Name |Vdi size |Allocated| Shared | Status ----------------+---------+---------+---------+------------ zopa | 5120 MB| 4068 MB| 0 MB| not running zopa2 | 5120 MB| 4068 MB| 0 MB| not running zopa3 | 5120 MB| 4068 MB| 0 MB| not running zopa4 | 5120 MB| 2848 MB| 0 MB| running on 192.1.1.4 fire-srv4 ~ # shepherd info -t vdi 40000 : zopa 5120 MB (allocated: 4068 MB, shared: 0 MB), 2009-12-30 13:54:04, tag: 0, current 80000 : zopa2 5120 MB (allocated: 4068 MB, shared: 0 MB), 2009-12-30 14:01:17, tag: 0, current c0000 : zopa3 5120 MB (allocated: 4068 MB, shared: 0 MB), 2009-12-30 14:04:28, tag: 0, current 100000 : zopa4 5120 MB (allocated: 2848 MB, shared: 0 MB), 2009-12-30 14:07:31, tag: 0, current fire-srv4 ~ # Alex |