[Sheepdog] free space handling issues

[Piavlo]

 Hi again,

I wanted to see how sheepdog would  behave when it is short in disk space.

1) First as you can see below the "Use%" field is not calculated correctly

fire-srv4 ~ # shepherd info -t sheep
Id      Size    Used    Use%
 0      5G      9G      181%
 1      2G      12G     643%
 2      8G      6G       83%
fire-srv4 ~ #

 I think it is obvious from the output exactly how wrong the "Use%" gets
calculated :)


2) I've created a 15G btrfs on 3 nodes and --copies=2 for the test and
wanted to fully fill just one of the nodes  and see  how sheepdog 
behaves then.
I hoped that once one of the nodes gets out of space it would continue
to allocate blocks on the other 2 remaining nodes, but it just
got stuck with "aio_read_response 869: I/O error"

fire-srv3 ~ # kvm-img convert -f raw -O sheepdog  /dev/sys/kvm-img zopa
sd_claim 1492: zopa
sd_release 1533: zopa
sd_close 1478: zopa
fire-srv3 ~ # kvm-img convert -f raw -O sheepdog  /dev/sys/kvm-img zopa2
sd_claim 1492: zopa2
sd_release 1533: zopa2
sd_close 1478: zopa2
fire-srv3 ~ # kvm-img convert -f raw -O sheepdog  /dev/sys/kvm-img zopa3
sd_claim 1492: zopa3
sd_release 1533: zopa3
sd_close 1478: zopa3
fire-srv3 ~ # kvm-img convert -f raw -O sheepdog  /dev/sys/kvm-img zopa4
sd_claim 1492: zopa4
aio_read_response 869: I/O error
...

this is from the fire-srv3 node on which kvm-img was ruinning:
...
Dec 30 14:09:28 localhost collie: store_queue_request(181) 0, 4,
/sheepdog/0/100000, 3, 3
Dec 30 14:09:28 localhost collie: store_queue_request(181) 2, 4,
/sheepdog/0/100000, 3, 3
Dec 30 14:09:28 localhost collie: store_queue_request(181) 1, 4,
/sheepdog/0/100000, 3, 3
Dec 30 14:09:28 localhost collie: store_queue_request(181) 3, 4,
/sheepdog/0/100000, 3, 3
Dec 30 14:09:28 localhost collie: store_queue_request(181) 0, 4,
/sheepdog/0/100000, 3, 3
Dec 30 14:09:28 localhost collie: store_queue_request(181) 2, 1,
/sheepdog/0/100354, 3, 3
Dec 30 14:09:28 localhost collie: store_queue_request(250) 1049428
Dec 30 14:09:28 localhost collie: store_queue_request(254) 1049428
Dec 30 14:09:28 localhost collie: store_queue_request(181) 1, 4,
/sheepdog/0/100000, 3, 3
Dec 30 14:09:28 localhost collie: listen_handler(337) accepted a new
connection, 258
Dec 30 14:09:28 localhost collie: cluster_queue_request(175) 0xa66c90 19
Dec 30 14:09:28 localhost collie: client_handler(298) closed a
connection, 258
Dec 30 14:09:34 localhost collie: listen_handler(337) accepted a new
connection, 258
Dec 30 14:09:34 localhost collie: store_queue_request(181) 0, b0,
/sheepdog/0/0, 3, 3
Dec 30 14:09:34 localhost collie: client_handler(298) closed a
connection, 258
Dec 30 14:09:34 localhost collie: listen_handler(337) accepted a new
connection, 258
Dec 30 14:09:34 localhost collie: so_read_vdis(367) 40000
Dec 30 14:09:34 localhost collie: so_read_vdis(386)
0000000000040000-00000000
Dec 30 14:09:34 localhost collie: so_read_vdis(367) 80000
Dec 30 14:09:34 localhost collie: so_read_vdis(386)
0000000000080000-00000000
Dec 30 14:09:34 localhost collie: so_read_vdis(367) c0000
Dec 30 14:09:34 localhost collie: so_read_vdis(386)
00000000000c0000-00000000
Dec 30 14:09:34 localhost collie: so_read_vdis(367) 100000
Dec 30 14:09:34 localhost collie: so_read_vdis(386)
0000000000100000-00000000
Dec 30 14:09:34 localhost collie: so_read_vdis(403) 128
Dec 30 14:09:34 localhost collie: client_handler(298) closed a
connection, 258
Dec 30 14:09:49 localhost collie: listen_handler(337) accepted a new
connection, 258
Dec 30 14:09:49 localhost collie: store_queue_request(181) 1, b0,
/sheepdog/0/0, 3, 3
Dec 30 14:09:49 localhost collie: client_handler(298) closed a
connection, 258
Dec 30 14:09:49 localhost collie: listen_handler(337) accepted a new
connection, 258
Dec 30 14:09:49 localhost collie: so_read_vdis(367) 40000
Dec 30 14:09:49 localhost collie: so_read_vdis(386)
0000000000040000-00000000
Dec 30 14:09:49 localhost collie: so_read_vdis(367) 80000
Dec 30 14:09:49 localhost collie: so_read_vdis(386)
0000000000080000-00000000
Dec 30 14:09:49 localhost collie: so_read_vdis(367) c0000
Dec 30 14:09:49 localhost collie: so_read_vdis(386)
00000000000c0000-00000000
Dec 30 14:09:49 localhost collie: so_read_vdis(367) 100000
Dec 30 14:09:49 localhost collie: so_read_vdis(386)
0000000000100000-00000000
Dec 30 14:09:49 localhost collie: so_read_vdis(403) 128
Dec 30 14:09:49 localhost collie: client_handler(298) closed a
connection, 258
...

Also sheepdog show that  zopa4 vdi is in running state  in spite of that
i've terminated the kvm-img process:
fire-srv4 ~ # shepherd info -t vm
Name            |Vdi size |Allocated| Shared  | Status
----------------+---------+---------+---------+------------
zopa            |  5120 MB|  4068 MB|     0 MB| not running
zopa2           |  5120 MB|  4068 MB|     0 MB| not running
zopa3           |  5120 MB|  4068 MB|     0 MB| not running
zopa4           |  5120 MB|  2848 MB|     0 MB| running on 192.1.1.4
fire-srv4 ~ # shepherd info -t vdi
   40000 : zopa         5120 MB (allocated: 4068 MB, shared: 0 MB),
2009-12-30 13:54:04, tag:        0, current
   80000 : zopa2        5120 MB (allocated: 4068 MB, shared: 0 MB),
2009-12-30 14:01:17, tag:        0, current
   c0000 : zopa3        5120 MB (allocated: 4068 MB, shared: 0 MB),
2009-12-30 14:04:28, tag:        0, current
  100000 : zopa4        5120 MB (allocated: 2848 MB, shared: 0 MB),
2009-12-30 14:07:31, tag:        0, current
fire-srv4 ~ #

Alex