[Sheepdog] Sheepdog and corosync
MORITA Kazutaka
morita.kazutaka at lab.ntt.co.jp
Mon Aug 22 08:15:57 CEST 2011
At Tue, 16 Aug 2011 21:04:07 +0100,
Brian Candler wrote:
>
> I am in the process of getting a trivial (1-node) sheepdog running under
> Ubuntu 11.04 x86_64.
>
> I have the corosync package installed, copied corosync.conf.example to
> corosync.conf and set a valid bindnetaddr. It appears to start - these
> messages appear in /var/log/syslog
>
> ~~~~
> Aug 16 20:31:37 x100 corosync[15772]: [MAIN ] Corosync Cluster Engine ('1.2.1'): started and ready to provide service.
> Aug 16 20:31:37 x100 corosync[15772]: [MAIN ] Corosync built-in features: nss
> Aug 16 20:31:37 x100 corosync[15772]: [MAIN ] Successfully read main configuration file '/etc/corosync/corosync.conf'.
> Aug 16 20:31:37 x100 corosync[15772]: [TOTEM ] Initializing transport (UDP/IP).
> Aug 16 20:31:37 x100 corosync[15772]: [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
> Aug 16 20:31:37 x100 corosync[15772]: [TOTEM ] The network interface [192.168.122.1] is now up.
> Aug 16 20:31:37 x100 corosync[15772]: [SERV ] Service engine loaded: corosync extended virtual synchrony service
> Aug 16 20:31:37 x100 corosync[15772]: [SERV ] Service engine loaded: corosync configuration service
> Aug 16 20:31:37 x100 corosync[15772]: [SERV ] Service engine loaded: corosync cluster closed process group service v1.01
> Aug 16 20:31:37 x100 corosync[15772]: [SERV ] Service engine loaded: corosync cluster config database access v1.01
> Aug 16 20:31:37 x100 corosync[15772]: [SERV ] Service engine loaded: corosync profile loading service
> Aug 16 20:31:37 x100 corosync[15772]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1
> Aug 16 20:31:37 x100 corosync[15772]: [MAIN ] Compatibility mode set to whitetank. Using V1 and V2 of the synchronization engine.
> Aug 16 20:31:37 x100 corosync[15772]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
> Aug 16 20:31:37 x100 corosync[15772]: [MAIN ] Completed service synchronization, ready to provide service.
> ~~~~
>
> However, when I try to run sheep, I get the following:
>
> ~~~~
> $ sheep -f /var/tmp/sheep
> sheep: jrnl_recover(2305) Openning the directory /var/tmp/sheep/journal/00000000/.
> sheep: create_cluster(1709) Failed to initialize cpg, 100
> sheep: create_cluster(1710) Is corosync running?
> sheep: main(150) failed to create sheepdog cluster.
> ~~~~
>
> And each time I do this, I get the following message in /var/log/syslog:
>
> Aug 16 20:32:46 x100 corosync[15772]: [IPC ] Invalid IPC credentials.
>
> This suggests to me some sort of authentication issue between sheep and
> corosync.
>
> The usage example at https://github.com/collie/sheepdog/wiki/Getting-Started
> seems to show sheep being run as a regular user, not root. But I tried
> running as root anyway, and it seemed to work this time:
>
> ~~~~
> $ sudo sheep -f /var/tmp/sheep
> sheep: jrnl_recover(2305) Openning the directory /var/tmp/sheep/journal/00000000/.
> sheep: set_addr(1696) addr = 192.168.122.1, port = 7000
> sheep: main(154) Sheepdog daemon (version 0.2.3) started
> sheep: read_epoch(2099) failed to read epoch 0
> ~~~~
>
> OK, so let's go with that for now (although I'd prefer not to run as root)
>
> ~~~~
> $ collie cluster format --copies=2
> $ collie node list
> Idx - Host:Port Vnodes Zone
> -----------------------------------------
> * 0 - 192.168.122.1:7000 64 0
> $ qemu-img create sheepdog:Test 2G
> Formatting 'sheepdog:Test', fmt=raw size=2147483648
> qemu-img: Failed to write the requested VDI, Test
>
> qemu-img: sheepdog:Test: error while creating raw: Input/output error
> ~~~~
>
> Hmm, that's not so good. The sheep process says:
>
> ~~~~
> sheep: cluster_queue_request(266) 0x7f6c891f4010 84
> sheep: attr(1928) use 'user_xattr' option?, user.sheepdog.copies
> sheep: __sd_deliver_done(925) unknown message 2
> sheep: cluster_queue_request(266) 0x10d3130 82
> sheep: cluster_queue_request(266) 0x10d3130 11
> sheep: do_lookup_vdi(236) looking for Test 4, ec9f05
> sheep: add_vdi(333) we create a new vdi, 0 Test (4) 2147483648, vid: ec9f05, base 0, cur 0
> sheep: add_vdi(337) qemu doesn't specify the copies... 2
> sheep: store_queue_request_local(628) use 'user_xattr' option?
> sheep: write_object(647) fail 80ec9f0500000000 6
> sheep: __sd_deliver_done(925) unknown message 2
> ~~~~
>
> Maybe I need to set copies=1 for a degraded cluster?
>
> ~~~~
> $ collie cluster format --copies=1
> $ collie node list
> Idx - Host:Port Vnodes Zone
> -----------------------------------------
> * 0 - 192.168.122.1:7000 64 0
> brian at x100:/etc/corosync$ qemu-img create sheepdog:Test 2G
> Formatting 'sheepdog:Test', fmt=raw size=2147483648
> qemu-img: Failed to write the requested VDI, Test
>
> qemu-img: sheepdog:Test: error while creating raw: Input/output error
> ~~~~
>
> Same result:
>
> ~~~~
> sheep: cluster_queue_request(266) 0x10d3130 84
> sheep: attr(1928) use 'user_xattr' option?, user.sheepdog.copies
> sheep: __sd_deliver_done(925) unknown message 2
> sheep: cluster_queue_request(266) 0x10d3130 82
> sheep: cluster_queue_request(266) 0x10d3130 11
> sheep: do_lookup_vdi(236) looking for Test 4, ec9f05
> sheep: add_vdi(333) we create a new vdi, 0 Test (4) 2147483648, vid: ec9f05, base 0, cur 0
> sheep: add_vdi(337) qemu doesn't specify the copies... 1
> sheep: store_queue_request_local(628) use 'user_xattr' option?
> sheep: write_object(647) fail 80ec9f0500000000 6
> sheep: __sd_deliver_done(925) unknown message 2
> ~~~~
>
> I notice the message about "user_xattr" option. However this filesystem
> is ext4:
>
> $ mount | grep "on / "
> /dev/sda5 on / type ext4 (rw,errors=remount-ro,commit=0)
>
> and the Getting-Started guide says that user_xattr is only needed for ext3.
> However, let's try it anyway:
>
> $ sudo mount -o remount,user_xattr /
>
> OK, that seems to work! Sheep shows:
>
> ~~~~
> sheep: cluster_queue_request(266) 0x10d3130 11
> sheep: do_lookup_vdi(236) looking for Test 4, ec9f05
> sheep: add_vdi(333) we create a new vdi, 0 Test (4) 2147483648, vid: ec9f05, base 0, cur 0
> sheep: add_vdi(337) qemu doesn't specify the copies... 1
> sheep: vdi_op_done(758) done 0 15507205
> sheep: __sd_deliver_done(925) unknown message 2
> ~~~~
>
> and I can boot with
>
> $ qemu-system-x86_64 -cdrom /v/downloads/linux/ubuntu-10.04.3-server-amd64.iso sheepdog:Test
>
> So it looks like I have a one-node cluster:
>
> ~~~~
> # collie cluster info
> Cluster status: running
>
> Creation time Epoch Nodes
> 1970-01-01 01:00:00 1 [192.168.122.1:7000]
> ~~~~~
>
> Anyway, my questions are:
>
> 1. Can I run sheep as a non-root user? If so, how?
>
> 2. Do I really need user_xattr even for ext4? (if so, the documentation
> needs adjusting)
For Sheepdog, the underlying filesystem needs to support an extended
attribute. I'm not familiar with ext4, but the answer is probably
yes.
>
> 3. Can I ignore the cluster creation time of '1970-01-01 01:00:00' ?
If your filesystem supports an extended attribute, you can set the
correct creation time.
>
> 4. What happens if you set --copies=N but the cluster degrades to
> the point where it has fewer nodes than that? As far as I can see,
> my one-node cluster with --copies=2 does actually work. Would the
> data get copied when a new node is added?
Even if the number of nodes is fewer than N, Sheepdog can work. When
you add a new machine, the copies of data will be increased up to N.
>
> One other point. Experimentation shows that "collie cluster format"
> instantly destroys all existing vdis, with no confirmation - and it can be
> run as a non-root user. Can I suggest some idiot-proofing is done on this?
> e.g. if a cluster already exists then you need to add some extra parameter
> to force deletion?
Good point. I think we should add something like a "--force" option.
E.g.
$ collie cluster format # succeed only when the store directory is empty
$ collie cluster format --force # succeed always
Thanks,
Kazutaka
More information about the sheepdog
mailing list