[sheepdog-users] hyper volume issue

Hitoshi Mitake mitake.hitoshi at gmail.com
Mon Sep 21 16:32:38 CEST 2015 

On Mon, Sep 21, 2015 at 10:57 PM, Glen Aidukas
<GAidukas at behaviormatrix.com> wrote:
> Hitoshi,
>
> In a  few days I will be setting up a new test cluster with three nodes, 1 journal and 3 data drives each and also 10gbit networking.  I will test with the new branch then.

Please never use journaling, it doesn't contribute to performance and
degrades stability. It will be removed in the future.

>
> What is the effect of using  '-z 23' ?

-z is the block size shift. -z 23 means a newly created VDI will have
1 << 23 B (8MB) object. It will solve the 4TB limitation.

Thanks,
Hitoshi

>
> Regards,
>
> Glen
>
>
> -----Original Message-----
> From: Hitoshi Mitake [mailto:mitake.hitoshi at gmail.com]
> Sent: Monday, September 21, 2015 4:10 AM
> To: Glen Aidukas
> Cc: sheepdog-users (sheepdog-users at lists.wpkg.org)
> Subject: Re: [sheepdog-users] hyper volume issue
>
> Hi Glen,
>
> hypervolume isn't a feature for VDIs, it is only used for http object store.
>
> Could you try below command line with master branch:
> $ dog vdi create <vdi name> 5T -z 23
>
> The master branch supports VDIs larger than 4TB. The feature isn't backported to 0.9.x yet.
>
> Thanks,
> Hitoshi
>
>
> On Fri, Sep 18, 2015 at 10:29 PM, Glen Aidukas <GAidukas at behaviormatrix.com> wrote:
>> Hello,
>>
>>
>>
>> I’m seeing an issue with hyper volumes on my two node test cluster
>> using v0.9.2.  While performing some tests, I tried to create a 5TB
>> vdi and saw that normal volumes are limited to 4TB and that I needed
>> to use the –y switch to create a Hyper Volume.  This worked when
>> creating it but then was not able to start my vm while connecting to
>> it.  I then tried to delete the hyper volume and the cluster then
>> crashed.  When I restarted the cluster I was still having issues.  I
>> was able to reformat the cluster, reimport my test vms and all was working again.
>>
>>
>>
>> To test and see if  trying to use the hv vdi is what caused the issue
>> I then created a 5TB hv-vdi and then delete it and I got the same issue.
>>
>>
>>
>> -Glen
>>
>>
>>
>> Here is a recreation of the issue:
>>
>>
>>
>> Take special note to the log file showing:
>>
>> Sep 18 09:08:25  EMERG [deletion] traverse_btree(190) PANIC: This
>> B-tree not support depth 0?
>>
>>
>>
>>
>>
>> root at pmsd-a01:~# sheep -v
>>
>> Sheepdog daemon version 0.9.2
>>
>>
>>
>> root at pmsd-a01:~# dog node md info --all
>>
>> Id      Size    Used    Avail   Use%    Path
>>
>> Node 0:
>>
>> 0      1.8 TB  10 GB   1.8 TB    0%    /var/lib/sheepdog//disc1
>>
>> 1      1.8 TB  9.7 GB  1.8 TB    0%    /var/lib/sheepdog//disc2
>>
>> 2      1.8 TB  10 GB   1.8 TB    0%    /var/lib/sheepdog//disc3
>>
>> Node 1:
>>
>> 0      1.8 TB  10 GB   1.8 TB    0%    /var/lib/sheepdog//disc1
>>
>> 1      1.8 TB  9.7 GB  1.8 TB    0%    /var/lib/sheepdog//disc2
>>
>> 2      1.8 TB  10 GB   1.8 TB    0%    /var/lib/sheepdog//disc3
>>
>>
>>
>>
>>
>> root at pmsd-a01:~# dog node info
>>
>> Id      Size    Used    Avail   Use%
>>
>> 0      5.4 TB  30 GB   5.3 TB    0%
>>
>> 1      5.4 TB  30 GB   5.3 TB    0%
>>
>> Total   11 TB   60 GB   11 TB     0%
>>
>>
>>
>> Total virtual image size        512 GB
>>
>>
>>
>> root at pmsd-a01:~# dog vdi list
>>
>>   Name        Id    Size    Used  Shared    Creation time   VDI id  Copies
>> Tag
>>
>>   vm-60153-disk-1     0  128 GB  7.5 GB  0.0 MB 2015-09-17 11:13   2e24f0
>> 2
>>
>>   vm-60154-disk-1     0  128 GB  7.5 GB  0.0 MB 2015-09-17 11:24   57e86b
>> 2
>>
>>   vm-60152-disk-1     0  128 GB  7.5 GB  0.0 MB 2015-09-17 11:00   794d35
>> 2
>>
>>   vm-60151-disk-1     0  128 GB  7.5 GB  0.0 MB 2015-09-17 10:35   c286fb
>> 2
>>
>>
>>
>> root at pmsd-a01:~# dog vdi create test-5tb-hv 5T -y
>>
>> root at pmsd-a01:~# dog vdi list
>>
>>   Name        Id    Size    Used  Shared    Creation time   VDI id  Copies
>> Tag
>>
>>   vm-60153-disk-1     0  128 GB  7.5 GB  0.0 MB 2015-09-17 11:13   2e24f0
>> 2
>>
>>   test-5tb-hv         0  5.0 TB  0.0 MB  0.0 MB 2015-09-18 09:07   3af976
>> 2
>>
>>   vm-60154-disk-1     0  128 GB  7.5 GB  0.0 MB 2015-09-17 11:24   57e86b
>> 2
>>
>>   vm-60152-disk-1     0  128 GB  7.5 GB  0.0 MB 2015-09-17 11:00   794d35
>> 2
>>
>>   vm-60151-disk-1     0  128 GB  7.5 GB  0.0 MB 2015-09-17 10:35   c286fb
>> 2
>>
>>
>>
>>
>>
>> root at pmsd-a01:~# dog vdi delete test-5tb-hv
>>
>> failed to read a response
>>
>>
>>
>>
>>
>> root at pmsd-a01:~# dog node info
>>
>> failed to connect to 127.0.0.1:7000: Connection refused
>>
>> failed to connect to 127.0.0.1:7000: Connection refused
>>
>> Failed to get node list
>>
>>
>>
>> root at pmsd-a01:~# service sheepdog restart
>>
>> Restarting Sheepdog Server: sheepdog.
>>
>>
>>
>> root at pmsd-a02:~# service sheepdog restart              (on second node)
>>
>> Restarting Sheepdog Server: sheepdog.
>>
>>
>>
>>
>>
>> root at pmsd-a01:~# dog vdi list
>>
>>
>>
>> PANIC: Depth of B-tree is out of range(depth: 0)
>>
>>   Name        Id    Size    Used  Shared    Creation time   VDI id  Copies
>> Tag
>>
>>   vm-60153-disk-1     0  128 GB  7.5 GB  0.0 MB 2015-09-17 11:13   2e24f0
>> 2
>>
>> dog exits unexpectedly (Aborted).
>>
>> sh: 1: addr2line: not found
>>
>> :
>>
>> sh: 1: addr2line: not found
>>
>> :
>>
>> sh: 1: addr2line: not found
>>
>> :
>>
>> sh: 1: addr2line: not found
>>
>>
>>
>>
>>
>> root at pmsd-a01:~# less sheedog.log
>>
>> Sep 18 09:07:52   INFO [main] rx_main(830) req=0x7f1d64032760, fd=189,
>> client=127.0.0.1:59938, op=NEW_VDI, data=(not string)
>>
>> Sep 18 09:07:52   INFO [main] tx_main(882) req=0x7f1d64032760, fd=189,
>> client=127.0.0.1:59938, op=NEW_VDI, result=00
>>
>> Sep 18 09:08:25   INFO [main] rx_main(830) req=0x7f1d6406aa40, fd=189,
>> client=127.0.0.1:59970, op=DEL_VDI, data=(not string)
>>
>> Sep 18 09:08:25  EMERG [deletion] traverse_btree(190) PANIC: This
>> B-tree not support depth 0
>>
>> Sep 18 09:08:25  EMERG [deletion] crash_handler(268) sheep exits
>> unexpectedly (Aborted).
>>
>> Sep 18 09:08:25  EMERG [deletion] sd_backtrace(840)
>> /usr/sbin/sheep(+0xb19b) [0x7f1d7c55319b]
>>
>> Sep 18 09:08:25  EMERG [deletion] sd_backtrace(840)
>> /lib/x86_64-linux-gnu/libpthread.so.0(+0xf09f) [0x7f1d7c10f09f]
>>
>> Sep 18 09:08:25  EMERG [deletion] sd_backtrace(840)
>> /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x34) [0x7f1d7b2e2164]
>>
>> Sep 18 09:08:25  EMERG [deletion] sd_backtrace(840)
>> /lib/x86_64-linux-gnu/libc.so.6(abort+0x17f) [0x7f1d7b2e53df]
>>
>> Sep 18 09:08:25  EMERG [deletion] sd_backtrace(840)
>> /usr/sbin/sheep(+0x41566) [0x7f1d7c589566]
>>
>> Sep 18 09:08:25  EMERG [deletion] sd_backtrace(840)
>> /usr/sbin/sheep(+0x16415) [0x7f1d7c55e415]
>>
>> Sep 18 09:08:25  EMERG [deletion] sd_backtrace(840)
>> /usr/sbin/sheep(+0x3daba) [0x7f1d7c585aba]
>>
>> Sep 18 09:08:25  EMERG [deletion] sd_backtrace(840)
>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b4f) [0x7f1d7c106b4f]
>>
>> Sep 18 09:08:25  EMERG [deletion] sd_backtrace(840)
>> /lib/x86_64-linux-gnu/libc.so.6(clone+0x6c) [0x7f1d7b38b95c]
>>
>> Sep 18 09:08:54   INFO [main] md_add_disk(343) /var/lib/sheepdog//disc1,
>> vdisk nr 1834, total disk 1
>>
>> Sep 18 09:08:54   INFO [main] md_add_disk(343) /var/lib/sheepdog//disc2,
>> vdisk nr 1834, total disk 2
>>
>> Sep 18 09:08:54   INFO [main] md_add_disk(343) /var/lib/sheepdog//disc3,
>> vdisk nr 1834, total disk 3
>>
>> Sep 18 09:08:55 NOTICE [main] get_local_addr(522) found IPv4 address
>>
>> Sep 18 09:08:55   INFO [main] send_join_request(1032) IPv4 ip:10.30.20.81
>> port:7000 going to join the cluster
>>
>> Sep 18 09:08:55  ERROR [main] zk_join(975) Previous zookeeper session
>> exist, shoot myself. Please wait for 30 seconds to join me again.
>>
>> Sep 18 09:13:00   INFO [main] md_add_disk(343) /var/lib/sheepdog//disc1,
>> vdisk nr 1834, total disk 1
>>
>> Sep 18 09:13:00   INFO [main] md_add_disk(343) /var/lib/sheepdog//disc2,
>> vdisk nr 1834, total disk 2
>>
>> Sep 18 09:13:00   INFO [main] md_add_disk(343) /var/lib/sheepdog//disc3,
>> vdisk nr 1834, total disk 3
>>
>> Sep 18 09:13:01 NOTICE [main] get_local_addr(522) found IPv4 address
>>
>> Sep 18 09:13:01   INFO [main] send_join_request(1032) IPv4 ip:10.30.20.81
>> port:7000 going to join the cluster
>>
>> Sep 18 09:13:01   INFO [main] replay_journal_entry(161)
>> /var/lib/sheepdog//disc2/0057e86b00007c02, size 59904, off 380928, 0
>>
>> Sep 18 09:13:01   INFO [main] replay_journal_entry(161)
>> /var/lib/sheepdog//disc2/0057e86b00007c02, size 130560, off 448000, 0
>>
>> Sep 18 09:13:01   INFO [main] replay_journal_entry(161)
>> /var/lib/sheepdog//disc2/0057e86b00007c02, size 53248, off 584192, 0
>>
>> Sep 18 09:13:01   INFO [main] replay_journal_entry(161)
>> /var/lib/sheepdog//disc2/0057e86b00007c02, size 60416, off 642048, 0
>>
>> Sep 18 09:13:01   INFO [main] replay_journal_entry(161)
>> /var/lib/sheepdog//disc2/0057e86b00007c02, size 45568, off 706560, 0
>>
>> Sep 18 09:13:01   INFO [main] replay_journal_entry(161)
>> /var/lib/sheepdog//disc2/0057e86b00007c02, size 65536, off 756736, 0
>>
>> Sep 18 09:13:01   INFO [main] replay_journal_entry(161)
>> /var/lib/sheepdog//disc2/0057e86b00007c02, size 29184, off 888832, 0
>>
>> Sep 18 09:13:01   INFO [main] replay_journal_entry(161)
>> /var/lib/sheepdog//disc2/0057e86b00007c02, size 84480, off 923136, 0
>>
>>
>>
>>
>> --
>> sheepdog-users mailing lists
>> sheepdog-users at lists.wpkg.org
>> https://lists.wpkg.org/mailman/listinfo/sheepdog-users
>>

More information about the sheepdog-users mailing list