[sheepdog-users] sheepdog replication got stuck

Gerald Richter - ECOS richter at ecos.de
Mon Dec 23 06:53:08 CET 2013


Hi,

The following import processes are still running:

root      495858  0.0  0.0 113864 11480 ?        S    Dec19   0:25 qemu-img convert -p -t writeback tst-bb-sec-tstmaster2.vhd sheepdog:vm-61022-disk-1
root      495883  0.0  0.0 113864 11480 ?        S    Dec19   0:25 qemu-img convert -p -t writeback tst-bb-sec-tstmaster3.vhd sheepdog:vm-61032-disk-1
root      495907  0.0  0.0 113864 11480 ?        S    Dec19   0:21 qemu-img convert -p -t writeback tst-bb-sec-tstslave1.vhd sheepdog:vm-61015-disk-1
root      495917  0.0  0.0 113864 11480 ?        S    Dec19   0:20 qemu-img convert -p -t writeback tst-bb-sec-tstslave2.vhd sheepdog:vm-61025-disk-1
root      495927  0.0  0.0 113864 11480 ?        S    Dec19   0:24 qemu-img convert -p -t writeback tst-bb-sec-tstslave3.vhd sheepdog:vm-61035-disk-1
root      502020  0.0  0.0 113864 11480 ?        S    Dec19   0:29 qemu-img convert -p -t writeback tst-bb-sec-bbtest3a.vhd sheepdog:vm-61036-disk-1
root      502034  0.0  0.0 113864 11484 ?        S    Dec19   0:29 qemu-img convert -p -t writeback tst-bb-sec-bbtest3b.vhd sheepdog:vm-61037-disk-1

This is the node where the import has taken place, also the import process are still running, the Used column shows the expected size:

  Name        Id    Size    Used  Shared    Creation time   VDI id  Copies  Tag
  vm-61026-disk-1     0   20 GB   20 GB  0.0 MB 2013-12-19 09:19    58bea     3              
  vm-61035-disk-1     0   20 GB   18 GB  0.0 MB 2013-12-19 09:52   2b8afa     3              
  vm-61015-disk-1     0   20 GB   17 GB  0.0 MB 2013-12-19 09:52   3f8c18     3              
  vm-61022-disk-1     0   20 GB   20 GB  0.0 MB 2013-12-19 09:51   45e7e6     3              
  vm-6194-disk-1     0   32 GB  4.4 GB  0.0 MB 2013-11-25 19:52   4a2859     3              
  vm-61032-disk-1     0   20 GB   20 GB  0.0 MB 2013-12-19 09:51   514711     3              
  vm-61037-disk-1     0   20 GB   20 GB  0.0 MB 2013-12-19 12:35   9728f0     3              
  vm-61025-disk-1     0   20 GB   15 GB  0.0 MB 2013-12-19 09:52   bb7a25     3              
  vm-61036-disk-1     0   20 GB   20 GB  0.0 MB 2013-12-19 12:34   e25135     3              
  vm-61016-disk-1     0   20 GB   20 GB  0.0 MB 2013-12-19 09:02   e70ff7     3              
  vm-61012-disk-1     0   20 GB   20 GB  0.0 MB 2013-12-19 09:50   f08a93     3              
  vm-12161-disk-1     0  300 GB  237 GB  0.0 MB 2013-11-26 21:14   f6b38a     3              

This is the other machine:

  Name        Id    Size    Used  Shared    Creation time   VDI id  Copies  Tag
  vm-61026-disk-1     0   20 GB   20 GB  0.0 MB 2013-12-19 09:19    58bea     3              
  vm-61035-disk-1     0   20 GB   36 MB  0.0 MB 2013-12-19 09:52   2b8afa     3              
  vm-61015-disk-1     0   20 GB  136 MB  0.0 MB 2013-12-19 09:52   3f8c18     3              
  vm-61022-disk-1     0   20 GB  104 MB  0.0 MB 2013-12-19 09:51   45e7e6     3              
  vm-6194-disk-1     0   32 GB  4.4 GB  0.0 MB 2013-11-25 19:52   4a2859     3              
  vm-61032-disk-1     0   20 GB  132 MB  0.0 MB 2013-12-19 09:51   514711     3              
  vm-61037-disk-1     0   20 GB  0.0 MB  0.0 MB 2013-12-19 12:35   9728f0     3              
  vm-61025-disk-1     0   20 GB   36 MB  0.0 MB 2013-12-19 09:52   bb7a25     3              
  vm-61036-disk-1     0   20 GB  0.0 MB  0.0 MB 2013-12-19 12:34   e25135     3              
  vm-61016-disk-1     0   20 GB   20 GB  0.0 MB 2013-12-19 09:02   e70ff7     3              
  vm-61012-disk-1     0   20 GB  2.1 GB  0.0 MB 2013-12-19 09:50   f08a93     3              
  vm-12161-disk-1     0  300 GB  237 GB  0.0 MB 2013-11-26 21:14   f6b38a     3              

The last image ( vm-12161-disk-1)  is in use for about four weeks and seems to work ok.

I am not sure whenever it makes sense to export the image, if the import is still running?

Dog cluster info shows:

Cluster status: running, auto-recovery enabled

Cluster created at Sat Nov 23 18:50:18 2013

Epoch Time           Version
2013-12-10 06:07:16     12 [46.4.116.228:7000, 176.9.111.146:7000]
2013-12-10 05:45:43     11 [176.9.111.146:7000]
2013-11-26 12:23:32     10 [46.4.116.228:7000, 176.9.111.146:7000]
2013-11-26 12:23:23      9 [176.9.111.146:7000]
2013-11-26 12:19:13      8 [46.4.116.228:7000, 176.9.111.146:7000]
2013-11-26 12:19:08      7 [176.9.111.146:7000]
2013-11-25 19:47:20      6 [46.4.116.228:7000, 176.9.111.146:7000]
2013-11-25 19:47:11      5 [46.4.116.228:7000]
2013-11-25 19:39:25      4 [46.4.116.228:7000, 176.9.111.146:7000]
2013-11-25 19:39:11      3 [176.9.111.146:7000]
2013-11-23 23:49:15      2 [46.4.116.228:7000, 176.9.111.146:7000]
2013-11-23 18:50:18      1 [46.4.116.228:7000]

There is nothing in sheep.log

Thanks & Regards

Gerald


> -----Ursprüngliche Nachricht-----
> Von: Liu Yuan [mailto:namei.unix at gmail.com]
> Gesendet: Montag, 23. Dezember 2013 06:40
> An: Gerald Richter
> Cc: Lista sheepdog user
> Betreff: Re: [sheepdog-users] sheepdog replication got stuck
> 
> On Mon, Dec 23, 2013 at 05:56:00AM +0100, Gerald Richter - ECOS wrote:
> > Hi,
> >
> >
> > I have a test cluster with two nodes (but formated with the default copies
> = 3).
> >
> >
> > On Friday I imported a couple of vm’s each about 20GB. On the system
> where I did the import, they were imported correctly, but on the other
> system after 3 days only a few MB shows up in the node vdi list. Both sides
> see each other (dog node list show both nodes on both sides), there were
> no new epoch since I did the import, but still the size of the vdi’s on the
> second node is not growing anymore.
> >
> >
> 
> Can you show the output of 'vdi list' on each node? can you export the
> images you imported out of the cluster on each node by 'qemu-img convert'?
> 
> Thanks
> Yuan





More information about the sheepdog-users mailing list