[sheepdog-users] cache crash test

[Valerio Pachera]

Hi, the object sounds like a work joke, doesn't it? :-)

Here we go, this is my testing cluster.

# collie node info
Id      Size    Used    Avail   Use%
 0      931 GB  5.0 GB  926 GB    0%
 1      518 GB  3.0 GB  515 GB    0%
 2      518 GB  2.1 GB  516 GB    0%
Total   1.9 TB  10 GB   1.9 TB    0%
Total virtual image size        108 GB

# collie node md info --all
Id      Size    Used    Avail   Use%    Path
Node 0:
 0      931 GB  5.0 GB  926 GB    0%    /mnt/sheep/dsk02
Node 1:
 0      220 GB  1.3 GB  218 GB    0%    /mnt/sheep/dsk01/obj
 1      298 GB  1.7 GB  296 GB    0%    /mnt/sheep/dsk02
Node 2:
 0      220 GB  972 MB  219 GB    0%    /mnt/sheep/dsk01/obj
 1      298 GB  1.1 GB  297 GB    0%    /mnt/sheep/dsk02

I created few vdi, then I have a running guest on node 0 that is
writing 1M each 3 seconds (with oflag=direct).
Guest is using cache=writeback.
I killed node 2 to trigger a recover. It completed without any problem.

I've been checking sheep.log and I notice that "connect_to ... failed"
is repeated for long time after the node has die.

Jun 25 10:30:21 [rw] get_vdi_copy_number(108) No VDI copy entry for 0 found
Jun 25 10:30:21 [rw] screen_object_list(724) ERROR: can not find copy
number for object 4f4239
...
Jun 25 10:31:15 [rw] connect_to(254) failed to connect to
192.168.2.47:7000: Connection refused

*Is that normal?*

A bigger piece of sheep.log to give better the idea:

Jun 25 10:31:06 [main] recover_object_main(626) done:2262 count:2764,
oid:366058000003b7
Jun 25 10:31:07 [main] recover_object_main(626) done:2263 count:2764,
oid:3660580000000b
Jun 25 10:31:07 [main] recover_object_main(626) done:2264 count:2764,
oid:366058000003c0
Jun 25 10:31:07 [main] recover_object_main(626) done:2265 count:2764,
oid:a34c67000006c8
Jun 25 10:31:07 [rw] connect_to(254) failed to connect to
192.168.2.47:7000: Connection refused
Jun 25 10:31:07 [rw] connect_to(254) failed to connect to
192.168.2.47:7000: Connection refused
Jun 25 10:31:07 [main] recover_object_main(626) done:2266 count:2764,
oid:3660580000047d
Jun 25 10:31:07 [rw] connect_to(254) failed to connect to
192.168.2.47:7000: Connection refused
Jun 25 10:31:07 [rw] connect_to(254) failed to connect to
192.168.2.47:7000: Connection refused
Jun 25 10:31:07 [main] recover_object_main(626) done:2267 count:2764,
oid:4fd259000001ae
Jun 25 10:31:07 [rw] connect_to(254) failed to connect to
192.168.2.47:7000: Connection refused
Jun 25 10:31:07 [rw] connect_to(254) failed to connect to
192.168.2.47:7000: Connection refused
Jun 25 10:31:07 [main] recover_object_main(626) done:2268 count:2764,
oid:4fd25900005a05
Jun 25 10:31:07 [rw] connect_to(254) failed to connect to
192.168.2.47:7000: Connection refused
Jun 25 10:31:07 [rw] connect_to(254) failed to connect to
192.168.2.47:7000: Connection refused
Jun 25 10:31:07 [main] recover_object_main(626) done:2269 count:2764,
oid:3660580000039e
Jun 25 10:31:07 [rw] connect_to(254) failed to connect to
192.168.2.47:7000: Connection refused
Jun 25 10:31:07 [rw] connect_to(254) failed to connect to
192.168.2.47:7000: Connection refused
Jun 25 10:31:07 [main] recover_object_main(626) done:2270 count:2764,
oid:4fd25900005804
Jun 25 10:31:07 [rw] connect_to(254) failed to connect to
192.168.2.47:7000: Connection refused
Jun 25 10:31:07 [rw] connect_to(254) failed to connect to
192.168.2.47:7000: Connection refused
Jun 25 10:31:07 [main] recover_object_main(626) done:2271 count:2764,
oid:366058000003cc
Jun 25 10:31:07 [rw] connect_to(254) failed to connect to
192.168.2.47:7000: Connection refused
Jun 25 10:31:07 [rw] connect_to(254) failed to connect to
192.168.2.47:7000: Connection refused
Jun 25 10:31:07 [main] recover_object_main(626) done:2272 count:2764,
oid:a34c670000048c
Jun 25 10:31:07 [rw] connect_to(254) failed to connect to
192.168.2.47:7000: Connection refused
Jun 25 10:31:07 [rw] connect_to(254) failed to connect to
192.168.2.47:7000: Connection refused
Jun 25 10:31:07 [main] recover_object_main(626) done:2273 count:2764,
oid:4fd25900000079
Jun 25 10:31:07 [rw] connect_to(254) failed to connect to
192.168.2.47:7000: Connection refused
Jun 25 10:31:07 [rw] connect_to(254) failed to connect to
192.168.2.47:7000: Connection refused