[sheepdog-users] node recovery after reboot, virtual machines slow to boot

[Philip Crotwell]

Hi

Info below, let me know if there are any other prints that would help.
I rebooted 166 and it came back up within a minute as you can see, but
the VMs on it did not boot until after 12:00, so something like 50
minutes later. I think I remember that the node recovery was not
finished when the VMs came up, but was close to 90%.

Each of these sheep machines are ubuntu 14.04, 8 core amd64, 64 Gb
ram. Sheepdog has a 2 Tb lvm partition all to itself on each machine.
I will try to get the atop output on reboot, but I don't think the VMs
are doing much, neither reading or writing, other than the usual linux
boot sequence. Looking at top now 166 is 97% idle.

thanks
Philip


  Id   Host:Port         V-Nodes       Zone
   0   x.y.z.161:7000     62 2703490177
   1   x.y.z.162:7000     65 2720267393
   2   x.y.z.163:7000     65 2737044609
   3   x.y.z.166:7000     65 2787376257
Cluster status: running, auto-recovery enabled

Cluster created at Wed Nov  6 14:09:54 2013

Epoch Time           Version
2014-10-02 11:08:04     78 [x.y.z.161:7000, x.y.z.162:7000,
x.y.z.163:7000, x.y.z.166:7000]
2014-10-02 11:08:04     77 [x.y.z.161:7000, x.y.z.162:7000, x.y.z.163:7000]
2014-09-29 08:30:14     76 [x.y.z.161:7000, x.y.z.162:7000,
x.y.z.163:7000, x.y.z.166:7000]
2014-09-29 08:26:17     75 [x.y.z.161:7000, x.y.z.162:7000, x.y.z.166:7000]

Id    Size    Used    Avail    Use%
 0    1.9 TB    168 GB    1.7 TB      8%
 1    1.9 TB    134 GB    1.7 TB      7%
 2    1.9 TB    158 GB    1.7 TB      8%
 3    1.9 TB    146 GB    1.7 TB      7%
Total    7.5 TB    606 GB    6.9 TB      7%

Total virtual image size    1.4 TB
  Name        Id    Size    Used  Shared    Creation time   VDI id  Copies  Tag
  monitor      0   50 GB  4.5 GB  0.0 MB 2013-11-06 15:15    16379
3
c geolcontact     0  256 GB  5.5 GB  1.5 GB 2014-10-02 10:59    ca022
   3
c 4x4          0   49 GB  1.6 GB  564 MB 2014-03-20 10:10   1f526f
3
  wwwData      0  7.9 GB  7.9 GB  0.0 MB 2013-11-07 20:51   400894
3
c migratetest     0   49 GB  0.0 MB  1.0 GB 2013-12-05 15:31   447b86
   3
s saucyBase     1   49 GB  1.0 GB  0.0 MB 2013-12-03 14:55   4ae661
 3      basesnap
  saucyBase     0   49 GB  0.0 MB  1.0 GB 2013-12-03 17:10   4ae662
 3
s trusty       1  256 GB  3.1 GB  0.0 MB 2014-06-25 15:57   52d1c4
3   trustySnap0
  trusty       0  256 GB  0.0 MB  3.1 GB 2014-06-25 16:15   52d1c5
3
c scemdCopy     0   49 GB  1.9 GB  408 MB 2014-01-27 12:19   684de2
 3
c yaws         0   49 GB  2.8 GB  316 MB 2014-01-27 20:19   706961
3
c usarraydev     0  256 GB   14 GB  1.0 GB 2014-06-25 16:16   778d99
  3
  ewdata       0   93 GB   93 GB  0.0 MB 2013-12-03 17:23   7b8fb5
3
  wwwSaucy     0   49 GB  8.7 GB   14 GB 2014-07-14 08:19   9eee3c
3
  trustyBase     0  4.6 GB  4.6 GB  0.0 MB 2014-06-25 13:00   b2f291
  3
c revdev       0   49 GB   20 GB   96 MB 2014-06-23 16:58   bb3229
3
c couchdb      0   49 GB  2.3 GB  404 MB 2014-02-24 09:56   beace3
3
  4x4home      0  1.9 GB  1.9 GB  0.0 MB 2014-03-20 10:17   c0efc4
3
  dokuwiki     0   20 GB  6.7 GB  0.0 MB 2013-11-26 16:54   ccdc44
3
c ewextract     0   49 GB  4.5 GB  288 MB 2013-12-03 17:14   d6766e     3