[sheepdog-users] hyper volume issue

Glen Aidukas GAidukas at BehaviorMatrix.com
Fri Sep 18 15:29:18 CEST 2015 

Hello,

I'm seeing an issue with hyper volumes on my two node test cluster using v0.9.2.  While performing some tests, I tried to create a 5TB vdi and saw that normal volumes are limited to 4TB and that I needed to use the -y switch to create a Hyper Volume.  This worked when creating it but then was not able to start my vm while connecting to it.  I then tried to delete the hyper volume and the cluster then crashed.  When I restarted the cluster I was still having issues.  I was able to reformat the cluster, reimport my test vms and all was working again.

To test and see if  trying to use the hv vdi is what caused the issue I then created a 5TB hv-vdi and then delete it and I got the same issue.

-Glen

Here is a recreation of the issue:

Take special note to the log file showing:
Sep 18 09:08:25  EMERG [deletion] traverse_btree(190) PANIC: This B-tree not support depth 0?


root at pmsd-a01:~# sheep -v
Sheepdog daemon version 0.9.2

root at pmsd-a01:~# dog node md info --all
Id      Size    Used    Avail   Use%    Path
Node 0:
0      1.8 TB  10 GB   1.8 TB    0%    /var/lib/sheepdog//disc1
1      1.8 TB  9.7 GB  1.8 TB    0%    /var/lib/sheepdog//disc2
2      1.8 TB  10 GB   1.8 TB    0%    /var/lib/sheepdog//disc3
Node 1:
0      1.8 TB  10 GB   1.8 TB    0%    /var/lib/sheepdog//disc1
1      1.8 TB  9.7 GB  1.8 TB    0%    /var/lib/sheepdog//disc2
2      1.8 TB  10 GB   1.8 TB    0%    /var/lib/sheepdog//disc3


root at pmsd-a01:~# dog node info
Id      Size    Used    Avail   Use%
0      5.4 TB  30 GB   5.3 TB    0%
1      5.4 TB  30 GB   5.3 TB    0%
Total   11 TB   60 GB   11 TB     0%

Total virtual image size        512 GB

root at pmsd-a01:~# dog vdi list
  Name        Id    Size    Used  Shared    Creation time   VDI id  Copies  Tag
  vm-60153-disk-1     0  128 GB  7.5 GB  0.0 MB 2015-09-17 11:13   2e24f0      2
  vm-60154-disk-1     0  128 GB  7.5 GB  0.0 MB 2015-09-17 11:24   57e86b      2
  vm-60152-disk-1     0  128 GB  7.5 GB  0.0 MB 2015-09-17 11:00   794d35      2
  vm-60151-disk-1     0  128 GB  7.5 GB  0.0 MB 2015-09-17 10:35   c286fb      2

root at pmsd-a01:~# dog vdi create test-5tb-hv 5T -y
root at pmsd-a01:~# dog vdi list
  Name        Id    Size    Used  Shared    Creation time   VDI id  Copies  Tag
  vm-60153-disk-1     0  128 GB  7.5 GB  0.0 MB 2015-09-17 11:13   2e24f0      2
  test-5tb-hv         0  5.0 TB  0.0 MB  0.0 MB 2015-09-18 09:07   3af976      2
  vm-60154-disk-1     0  128 GB  7.5 GB  0.0 MB 2015-09-17 11:24   57e86b      2
  vm-60152-disk-1     0  128 GB  7.5 GB  0.0 MB 2015-09-17 11:00   794d35      2
  vm-60151-disk-1     0  128 GB  7.5 GB  0.0 MB 2015-09-17 10:35   c286fb      2


root at pmsd-a01:~# dog vdi delete test-5tb-hv
failed to read a response


root at pmsd-a01:~# dog node info
failed to connect to 127.0.0.1:7000: Connection refused
failed to connect to 127.0.0.1:7000: Connection refused
Failed to get node list

root at pmsd-a01:~# service sheepdog restart
Restarting Sheepdog Server: sheepdog.

root at pmsd-a02:~# service sheepdog restart              (on second node)
Restarting Sheepdog Server: sheepdog.


root at pmsd-a01:~# dog vdi list

PANIC: Depth of B-tree is out of range(depth: 0)
  Name        Id    Size    Used  Shared    Creation time   VDI id  Copies  Tag
  vm-60153-disk-1     0  128 GB  7.5 GB  0.0 MB 2015-09-17 11:13   2e24f0      2
dog exits unexpectedly (Aborted).
sh: 1: addr2line: not found
:
sh: 1: addr2line: not found
:
sh: 1: addr2line: not found
:
sh: 1: addr2line: not found


root at pmsd-a01:~# less sheedog.log
Sep 18 09:07:52   INFO [main] rx_main(830) req=0x7f1d64032760, fd=189, client=127.0.0.1:59938, op=NEW_VDI, data=(not string)
Sep 18 09:07:52   INFO [main] tx_main(882) req=0x7f1d64032760, fd=189, client=127.0.0.1:59938, op=NEW_VDI, result=00
Sep 18 09:08:25   INFO [main] rx_main(830) req=0x7f1d6406aa40, fd=189, client=127.0.0.1:59970, op=DEL_VDI, data=(not string)
Sep 18 09:08:25  EMERG [deletion] traverse_btree(190) PANIC: This B-tree not support depth 0
Sep 18 09:08:25  EMERG [deletion] crash_handler(268) sheep exits unexpectedly (Aborted).
Sep 18 09:08:25  EMERG [deletion] sd_backtrace(840) /usr/sbin/sheep(+0xb19b) [0x7f1d7c55319b]
Sep 18 09:08:25  EMERG [deletion] sd_backtrace(840) /lib/x86_64-linux-gnu/libpthread.so.0(+0xf09f) [0x7f1d7c10f09f]
Sep 18 09:08:25  EMERG [deletion] sd_backtrace(840) /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x34) [0x7f1d7b2e2164]
Sep 18 09:08:25  EMERG [deletion] sd_backtrace(840) /lib/x86_64-linux-gnu/libc.so.6(abort+0x17f) [0x7f1d7b2e53df]
Sep 18 09:08:25  EMERG [deletion] sd_backtrace(840) /usr/sbin/sheep(+0x41566) [0x7f1d7c589566]
Sep 18 09:08:25  EMERG [deletion] sd_backtrace(840) /usr/sbin/sheep(+0x16415) [0x7f1d7c55e415]
Sep 18 09:08:25  EMERG [deletion] sd_backtrace(840) /usr/sbin/sheep(+0x3daba) [0x7f1d7c585aba]
Sep 18 09:08:25  EMERG [deletion] sd_backtrace(840) /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b4f) [0x7f1d7c106b4f]
Sep 18 09:08:25  EMERG [deletion] sd_backtrace(840) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6c) [0x7f1d7b38b95c]
Sep 18 09:08:54   INFO [main] md_add_disk(343) /var/lib/sheepdog//disc1, vdisk nr 1834, total disk 1
Sep 18 09:08:54   INFO [main] md_add_disk(343) /var/lib/sheepdog//disc2, vdisk nr 1834, total disk 2
Sep 18 09:08:54   INFO [main] md_add_disk(343) /var/lib/sheepdog//disc3, vdisk nr 1834, total disk 3
Sep 18 09:08:55 NOTICE [main] get_local_addr(522) found IPv4 address
Sep 18 09:08:55   INFO [main] send_join_request(1032) IPv4 ip:10.30.20.81 port:7000 going to join the cluster
Sep 18 09:08:55  ERROR [main] zk_join(975) Previous zookeeper session exist, shoot myself. Please wait for 30 seconds to join me again.
Sep 18 09:13:00   INFO [main] md_add_disk(343) /var/lib/sheepdog//disc1, vdisk nr 1834, total disk 1
Sep 18 09:13:00   INFO [main] md_add_disk(343) /var/lib/sheepdog//disc2, vdisk nr 1834, total disk 2
Sep 18 09:13:00   INFO [main] md_add_disk(343) /var/lib/sheepdog//disc3, vdisk nr 1834, total disk 3
Sep 18 09:13:01 NOTICE [main] get_local_addr(522) found IPv4 address
Sep 18 09:13:01   INFO [main] send_join_request(1032) IPv4 ip:10.30.20.81 port:7000 going to join the cluster
Sep 18 09:13:01   INFO [main] replay_journal_entry(161) /var/lib/sheepdog//disc2/0057e86b00007c02, size 59904, off 380928, 0
Sep 18 09:13:01   INFO [main] replay_journal_entry(161) /var/lib/sheepdog//disc2/0057e86b00007c02, size 130560, off 448000, 0
Sep 18 09:13:01   INFO [main] replay_journal_entry(161) /var/lib/sheepdog//disc2/0057e86b00007c02, size 53248, off 584192, 0
Sep 18 09:13:01   INFO [main] replay_journal_entry(161) /var/lib/sheepdog//disc2/0057e86b00007c02, size 60416, off 642048, 0
Sep 18 09:13:01   INFO [main] replay_journal_entry(161) /var/lib/sheepdog//disc2/0057e86b00007c02, size 45568, off 706560, 0
Sep 18 09:13:01   INFO [main] replay_journal_entry(161) /var/lib/sheepdog//disc2/0057e86b00007c02, size 65536, off 756736, 0
Sep 18 09:13:01   INFO [main] replay_journal_entry(161) /var/lib/sheepdog//disc2/0057e86b00007c02, size 29184, off 888832, 0
Sep 18 09:13:01   INFO [main] replay_journal_entry(161) /var/lib/sheepdog//disc2/0057e86b00007c02, size 84480, off 923136, 0

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wpkg.org/pipermail/sheepdog-users/attachments/20150918/bd1753a7/attachment-0001.html>

More information about the sheepdog-users mailing list