[sheepdog-users] Testing: not all nodes list all vdi after restarting the cluster

[Valerio Pachera]

Hi, I was doing some tests about cache and write speed.
I started the cluster with 3 nodes using md.
  sheep -w size=20000 /sheep/meta,/sheep/sda,/sheep/sdb
Then added a 4th node that is not using md.
  sheep -w size=20000 /sheep/sda
Done some tests with a guest.
Stopped the cluster.
Restarted sheep with the same options, just a smaller cache.
Then one of the nodes (sheepdog004; id 0) is showing only 2 vdi instead of 4.

root at test004:~# collie vdi list
  Name        Id    Size    Used  Shared    Creation time   VDI id  Copies  Tag
  empty02      0   10 GB   10 GB  0.0 MB 2013-05-27 12:27   366058
2
  squeeze      0   10 GB  5.2 GB  0.0 MB 2013-05-27 12:10   a34c67     2

root at test005:~# collie vdi list
  Name        Id    Size    Used  Shared    Creation time   VDI id  Copies  Tag
  empty02      0   10 GB   10 GB  0.0 MB 2013-05-27 12:27   366058
2
  empty03      0   10 GB   10 GB  0.0 MB 2013-05-27 12:28   36620b
2
  empty01      0   10 GB   10 GB  0.0 MB 2013-05-27 12:27   366571
2
  squeeze      0   10 GB  5.2 GB  0.0 MB 2013-05-27 12:10   a34c67     2

root at test005:~# collie node md info --all
Id      Used    Free    Path
Node 0:
 0      15 GB   35 GB   /sheep/sda
 1      1.0 GB  3.6 GB  /sheep/sdb
Node 1:
 0      17 GB   33 GB   /sheep/sda/obj
Node 2:
 0      19 GB   31 GB   /sheep/sda
 1      1.4 GB  3.2 GB  /sheep/sdb
Node 3:
 0      16 GB   34 GB   /sheep/sda
 1      1.3 GB  3.3 GB  /sheep/sdb

root at test005:~# collie node info
Id      Size    Used    Use%
 0      55 GB   16 GB    29%
 1      50 GB   17 GB    33%
 2      55 GB   21 GB    37%
 3      55 GB   17 GB    30%
Total   214 GB  70 GB    32%
Total virtual image size        40 GB


root at test004:~# tail -15 /sheep/meta/sheep.log
May 27 12:33:09 [main] recover_object_main(612) done:3728 count:3731,
oid:36620b00000602
May 27 12:33:09 [main] recover_object_main(612) done:3729 count:3731,
oid:36657100000864
May 27 12:33:09 [main] recover_object_main(612) done:3730 count:3731,
oid:36620b000006d4
May 27 12:33:09 [main] recover_object_main(612) done:3731 count:3731,
oid:36620b00000952
May 27 13:47:44 [main] main(793) shutdown
May 27 13:48:10 [main] md_add_disk(161) /sheep/sda, nr 1
May 27 13:48:10 [main] md_add_disk(161) /sheep/sdb, nr 2
May 27 13:48:10 [main] send_join_request(1100) IPv4 ip:192.168.2.44 port:7000
May 27 13:48:10 [main] for_each_object_in_stale(387) /sheep/sda/.stale
May 27 13:48:10 [main] for_each_object_in_stale(387) /sheep/sdb/.stale
May 27 13:48:10 [main] check_host_env(396) WARN: Allowed open files
1024 too small, suggested 1024000
May 27 13:48:10 [main] check_host_env(405) Allowed core file size 0,
suggested unlimited
May 27 13:48:10 [main] main(786) sheepdog daemon (version
0.5.5_432_g2e7656b) started
May 27 13:48:10 [main] update_cluster_info(879) status = 4, epoch = 2,
finished: 0
May 27 13:49:05 [main] update_cluster_info(879) status = 1, epoch = 2,
finished: 1


root at test004:~# collie cluster info
Cluster status: running, auto-recovery enabled
Cluster created at Mon May 27 12:10:54 2013
Epoch Time           Version
2013-05-27 12:33:08      2 [192.168.2.44:7000, 192.168.2.45:7000,
192.168.2.46:7000, 192.168.2.47:7000]
2013-05-27 12:10:54      1 [192.168.2.44:7000, 192.168.2.46:7000,
192.168.2.47:7000]

sheep 0.5.5_432_g2e7656b