[sheepdog-users] hyper volume issue
Glen Aidukas
GAidukas at BehaviorMatrix.com
Mon Sep 21 21:01:31 CEST 2015
Hitoshi,
Thanks for your response and like I said, I will be creating a new test cluster with the newer code and test the -z 23 option.
I was wondering what the practical limits are. I guess with -z 23 I can get a 8TB VDI but I currently have some VMs on ceph with over 16TB and may be looking to go as large as 32TB. I know I can make multiple VDIs and even use software raid0 to join several 4TB or 8TB vdi's into one large volume but I was wondering what the targeted design limits are for sheepdog. Also I guess I could use something like -z 24 or 25 and get larger objects but doesn’t larger objects cause some performance issues?
Thanks,
Glen
-----Original Message-----
From: Hitoshi Mitake [mailto:mitake.hitoshi at gmail.com]
Sent: Monday, September 21, 2015 10:33 AM
To: Glen Aidukas
Cc: sheepdog-users (sheepdog-users at lists.wpkg.org)
Subject: Re: [sheepdog-users] hyper volume issue
On Mon, Sep 21, 2015 at 10:57 PM, Glen Aidukas <GAidukas at behaviormatrix.com> wrote:
> Hitoshi,
>
> In a few days I will be setting up a new test cluster with three nodes, 1 journal and 3 data drives each and also 10gbit networking. I will test with the new branch then.
Please never use journaling, it doesn't contribute to performance and degrades stability. It will be removed in the future.
>
> What is the effect of using '-z 23' ?
-z is the block size shift. -z 23 means a newly created VDI will have
1 << 23 B (8MB) object. It will solve the 4TB limitation.
Thanks,
Hitoshi
>
> Regards,
>
> Glen
>
>
> -----Original Message-----
> From: Hitoshi Mitake [mailto:mitake.hitoshi at gmail.com]
> Sent: Monday, September 21, 2015 4:10 AM
> To: Glen Aidukas
> Cc: sheepdog-users (sheepdog-users at lists.wpkg.org)
> Subject: Re: [sheepdog-users] hyper volume issue
>
> Hi Glen,
>
> hypervolume isn't a feature for VDIs, it is only used for http object store.
>
> Could you try below command line with master branch:
> $ dog vdi create <vdi name> 5T -z 23
>
> The master branch supports VDIs larger than 4TB. The feature isn't backported to 0.9.x yet.
>
> Thanks,
> Hitoshi
>
>
> On Fri, Sep 18, 2015 at 10:29 PM, Glen Aidukas <GAidukas at behaviormatrix.com> wrote:
>> Hello,
>>
>>
>>
>> I’m seeing an issue with hyper volumes on my two node test cluster
>> using v0.9.2. While performing some tests, I tried to create a 5TB
>> vdi and saw that normal volumes are limited to 4TB and that I needed
>> to use the –y switch to create a Hyper Volume. This worked when
>> creating it but then was not able to start my vm while connecting to
>> it. I then tried to delete the hyper volume and the cluster then
>> crashed. When I restarted the cluster I was still having issues. I
>> was able to reformat the cluster, reimport my test vms and all was working again.
>>
>>
>>
>> To test and see if trying to use the hv vdi is what caused the issue
>> I then created a 5TB hv-vdi and then delete it and I got the same issue.
>>
>>
>>
>> -Glen
>>
>>
>>
>> Here is a recreation of the issue:
>>
>>
>>
>> Take special note to the log file showing:
>>
>> Sep 18 09:08:25 EMERG [deletion] traverse_btree(190) PANIC: This
>> B-tree not support depth 0?
>>
>>
>>
>>
>>
>> root at pmsd-a01:~# sheep -v
>>
>> Sheepdog daemon version 0.9.2
>>
>>
>>
>> root at pmsd-a01:~# dog node md info --all
>>
>> Id Size Used Avail Use% Path
>>
>> Node 0:
>>
>> 0 1.8 TB 10 GB 1.8 TB 0% /var/lib/sheepdog//disc1
>>
>> 1 1.8 TB 9.7 GB 1.8 TB 0% /var/lib/sheepdog//disc2
>>
>> 2 1.8 TB 10 GB 1.8 TB 0% /var/lib/sheepdog//disc3
>>
>> Node 1:
>>
>> 0 1.8 TB 10 GB 1.8 TB 0% /var/lib/sheepdog//disc1
>>
>> 1 1.8 TB 9.7 GB 1.8 TB 0% /var/lib/sheepdog//disc2
>>
>> 2 1.8 TB 10 GB 1.8 TB 0% /var/lib/sheepdog//disc3
>>
>>
>>
>>
>>
>> root at pmsd-a01:~# dog node info
>>
>> Id Size Used Avail Use%
>>
>> 0 5.4 TB 30 GB 5.3 TB 0%
>>
>> 1 5.4 TB 30 GB 5.3 TB 0%
>>
>> Total 11 TB 60 GB 11 TB 0%
>>
>>
>>
>> Total virtual image size 512 GB
>>
>>
>>
>> root at pmsd-a01:~# dog vdi list
>>
>> Name Id Size Used Shared Creation time VDI id Copies
>> Tag
>>
>> vm-60153-disk-1 0 128 GB 7.5 GB 0.0 MB 2015-09-17 11:13 2e24f0
>> 2
>>
>> vm-60154-disk-1 0 128 GB 7.5 GB 0.0 MB 2015-09-17 11:24 57e86b
>> 2
>>
>> vm-60152-disk-1 0 128 GB 7.5 GB 0.0 MB 2015-09-17 11:00 794d35
>> 2
>>
>> vm-60151-disk-1 0 128 GB 7.5 GB 0.0 MB 2015-09-17 10:35 c286fb
>> 2
>>
>>
>>
>> root at pmsd-a01:~# dog vdi create test-5tb-hv 5T -y
>>
>> root at pmsd-a01:~# dog vdi list
>>
>> Name Id Size Used Shared Creation time VDI id Copies
>> Tag
>>
>> vm-60153-disk-1 0 128 GB 7.5 GB 0.0 MB 2015-09-17 11:13 2e24f0
>> 2
>>
>> test-5tb-hv 0 5.0 TB 0.0 MB 0.0 MB 2015-09-18 09:07 3af976
>> 2
>>
>> vm-60154-disk-1 0 128 GB 7.5 GB 0.0 MB 2015-09-17 11:24 57e86b
>> 2
>>
>> vm-60152-disk-1 0 128 GB 7.5 GB 0.0 MB 2015-09-17 11:00 794d35
>> 2
>>
>> vm-60151-disk-1 0 128 GB 7.5 GB 0.0 MB 2015-09-17 10:35 c286fb
>> 2
>>
>>
>>
>>
>>
>> root at pmsd-a01:~# dog vdi delete test-5tb-hv
>>
>> failed to read a response
>>
>>
>>
>>
>>
>> root at pmsd-a01:~# dog node info
>>
>> failed to connect to 127.0.0.1:7000: Connection refused
>>
>> failed to connect to 127.0.0.1:7000: Connection refused
>>
>> Failed to get node list
>>
>>
>>
>> root at pmsd-a01:~# service sheepdog restart
>>
>> Restarting Sheepdog Server: sheepdog.
>>
>>
>>
>> root at pmsd-a02:~# service sheepdog restart (on second node)
>>
>> Restarting Sheepdog Server: sheepdog.
>>
>>
>>
>>
>>
>> root at pmsd-a01:~# dog vdi list
>>
>>
>>
>> PANIC: Depth of B-tree is out of range(depth: 0)
>>
>> Name Id Size Used Shared Creation time VDI id Copies
>> Tag
>>
>> vm-60153-disk-1 0 128 GB 7.5 GB 0.0 MB 2015-09-17 11:13 2e24f0
>> 2
>>
>> dog exits unexpectedly (Aborted).
>>
>> sh: 1: addr2line: not found
>>
>> :
>>
>> sh: 1: addr2line: not found
>>
>> :
>>
>> sh: 1: addr2line: not found
>>
>> :
>>
>> sh: 1: addr2line: not found
>>
>>
>>
>>
>>
>> root at pmsd-a01:~# less sheedog.log
>>
>> Sep 18 09:07:52 INFO [main] rx_main(830) req=0x7f1d64032760, fd=189,
>> client=127.0.0.1:59938, op=NEW_VDI, data=(not string)
>>
>> Sep 18 09:07:52 INFO [main] tx_main(882) req=0x7f1d64032760, fd=189,
>> client=127.0.0.1:59938, op=NEW_VDI, result=00
>>
>> Sep 18 09:08:25 INFO [main] rx_main(830) req=0x7f1d6406aa40, fd=189,
>> client=127.0.0.1:59970, op=DEL_VDI, data=(not string)
>>
>> Sep 18 09:08:25 EMERG [deletion] traverse_btree(190) PANIC: This
>> B-tree not support depth 0
>>
>> Sep 18 09:08:25 EMERG [deletion] crash_handler(268) sheep exits
>> unexpectedly (Aborted).
>>
>> Sep 18 09:08:25 EMERG [deletion] sd_backtrace(840)
>> /usr/sbin/sheep(+0xb19b) [0x7f1d7c55319b]
>>
>> Sep 18 09:08:25 EMERG [deletion] sd_backtrace(840)
>> /lib/x86_64-linux-gnu/libpthread.so.0(+0xf09f) [0x7f1d7c10f09f]
>>
>> Sep 18 09:08:25 EMERG [deletion] sd_backtrace(840)
>> /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x34) [0x7f1d7b2e2164]
>>
>> Sep 18 09:08:25 EMERG [deletion] sd_backtrace(840)
>> /lib/x86_64-linux-gnu/libc.so.6(abort+0x17f) [0x7f1d7b2e53df]
>>
>> Sep 18 09:08:25 EMERG [deletion] sd_backtrace(840)
>> /usr/sbin/sheep(+0x41566) [0x7f1d7c589566]
>>
>> Sep 18 09:08:25 EMERG [deletion] sd_backtrace(840)
>> /usr/sbin/sheep(+0x16415) [0x7f1d7c55e415]
>>
>> Sep 18 09:08:25 EMERG [deletion] sd_backtrace(840)
>> /usr/sbin/sheep(+0x3daba) [0x7f1d7c585aba]
>>
>> Sep 18 09:08:25 EMERG [deletion] sd_backtrace(840)
>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b4f) [0x7f1d7c106b4f]
>>
>> Sep 18 09:08:25 EMERG [deletion] sd_backtrace(840)
>> /lib/x86_64-linux-gnu/libc.so.6(clone+0x6c) [0x7f1d7b38b95c]
>>
>> Sep 18 09:08:54 INFO [main] md_add_disk(343) /var/lib/sheepdog//disc1,
>> vdisk nr 1834, total disk 1
>>
>> Sep 18 09:08:54 INFO [main] md_add_disk(343) /var/lib/sheepdog//disc2,
>> vdisk nr 1834, total disk 2
>>
>> Sep 18 09:08:54 INFO [main] md_add_disk(343) /var/lib/sheepdog//disc3,
>> vdisk nr 1834, total disk 3
>>
>> Sep 18 09:08:55 NOTICE [main] get_local_addr(522) found IPv4 address
>>
>> Sep 18 09:08:55 INFO [main] send_join_request(1032) IPv4 ip:10.30.20.81
>> port:7000 going to join the cluster
>>
>> Sep 18 09:08:55 ERROR [main] zk_join(975) Previous zookeeper session
>> exist, shoot myself. Please wait for 30 seconds to join me again.
>>
>> Sep 18 09:13:00 INFO [main] md_add_disk(343) /var/lib/sheepdog//disc1,
>> vdisk nr 1834, total disk 1
>>
>> Sep 18 09:13:00 INFO [main] md_add_disk(343) /var/lib/sheepdog//disc2,
>> vdisk nr 1834, total disk 2
>>
>> Sep 18 09:13:00 INFO [main] md_add_disk(343) /var/lib/sheepdog//disc3,
>> vdisk nr 1834, total disk 3
>>
>> Sep 18 09:13:01 NOTICE [main] get_local_addr(522) found IPv4 address
>>
>> Sep 18 09:13:01 INFO [main] send_join_request(1032) IPv4 ip:10.30.20.81
>> port:7000 going to join the cluster
>>
>> Sep 18 09:13:01 INFO [main] replay_journal_entry(161)
>> /var/lib/sheepdog//disc2/0057e86b00007c02, size 59904, off 380928, 0
>>
>> Sep 18 09:13:01 INFO [main] replay_journal_entry(161)
>> /var/lib/sheepdog//disc2/0057e86b00007c02, size 130560, off 448000, 0
>>
>> Sep 18 09:13:01 INFO [main] replay_journal_entry(161)
>> /var/lib/sheepdog//disc2/0057e86b00007c02, size 53248, off 584192, 0
>>
>> Sep 18 09:13:01 INFO [main] replay_journal_entry(161)
>> /var/lib/sheepdog//disc2/0057e86b00007c02, size 60416, off 642048, 0
>>
>> Sep 18 09:13:01 INFO [main] replay_journal_entry(161)
>> /var/lib/sheepdog//disc2/0057e86b00007c02, size 45568, off 706560, 0
>>
>> Sep 18 09:13:01 INFO [main] replay_journal_entry(161)
>> /var/lib/sheepdog//disc2/0057e86b00007c02, size 65536, off 756736, 0
>>
>> Sep 18 09:13:01 INFO [main] replay_journal_entry(161)
>> /var/lib/sheepdog//disc2/0057e86b00007c02, size 29184, off 888832, 0
>>
>> Sep 18 09:13:01 INFO [main] replay_journal_entry(161)
>> /var/lib/sheepdog//disc2/0057e86b00007c02, size 84480, off 923136, 0
>>
>>
>>
>>
>> --
>> sheepdog-users mailing lists
>> sheepdog-users at lists.wpkg.org
>> https://lists.wpkg.org/mailman/listinfo/sheepdog-users
>>
More information about the sheepdog-users
mailing list