[sheepdog-users] 答复: Help? Creeping Errors "no inode has ..." with 0.9.1

[redtone]

Please make sure the recycle VDI is disabled for v 0.9.1

 

Daily snapshot will remove the old snapshot and create a new one.

If recycle VDI is enabled, when the snapshot is removed, the assigned VDI
will be deleted. And so the data is lost. 

 

This bug is fixed in v 0.7.1. but I am not sure in v 0.9.1.

 

 

  _____  

发件人: sheepdog-users [mailto:sheepdog-users-bounces at lists.wpkg.org] 代表
Thornton Prime
发送时间: 2015年1月27日 23:50
收件人: Hitoshi Mitake
抄送: Lista sheepdog user
主题: Re: [sheepdog-users] Help? Creeping Errors "no inode has ..." with
0.9.1

 

Thanks. I have been using cache -- so if that is unstable that would explain
a lot. I'm disabling cache to see how much that helps.

Attached is a dog cluster info. I have a few MB of logs ... I'll see where I
can post them to get the

I am seeing a strong correlation between snapshots and the corrupted VDIs.
All the VDIs that have missing inodes are part of a daily snapshot schedule.
All the VDIs that are not part of the snapshot schedule are fine. All the
nodes have object cache enabled.

Thanks ... I'll see if I can collect more data and reproduce the problem
more consistently.

~ thornton prime






 <mailto:mitake.hitoshi at lab.ntt.co.jp> Hitoshi Mitake

January 26, 2015 at 8:17 PM

At Mon, 26 Jan 2015 07:11:29 -0800,
Thornton Prime wrote:

I've been getting increasing errors in my logs that "failed No object
found, remote address: XXXXXXX:7000, op name: READ_PEER" and then
corresponding errors that "no inode has ...." when I do a cluster check.

 
Could you provide detailed logs and an output of "dog cluster info"?
 

At the beginning of last week I had no errors, and over the course of a
week it grew to be one VDI missing some hundred inodes, and now it is
multiple VDIs each missing hundreds of objects.
 
I haven't seen any issues with the underlying hardware, disks, or
zookeeper on the nodes in the course of the same time.
 
What is causing this data loss? How can I debug it? How can I stem it?
Any chances I can repair the missing inodes?
 
I have 5 sheepdog storage nodes, also running Zookeeper. I have another
8 "gateway only" nodes that are part of the node pool, but only running
a gateway and cache.

 
Object cache (a functionality which can be activated with -w option of
sheep) is quite unstable. Please do not use it for serious purpose.
 
Thanks,
Hitoshi



 <mailto:thornton.prime at gmail.com> Thornton Prime

January 26, 2015 at 7:11 AM

I've been getting increasing errors in my logs that "failed No object
found, remote address: XXXXXXX:7000, op name: READ_PEER" and then
corresponding errors that "no inode has ...." when I do a cluster check.

At the beginning of last week I had no errors, and over the course of a
week it grew to be one VDI missing some hundred inodes, and now it is
multiple VDIs each missing hundreds of objects.

I haven't seen any issues with the underlying hardware, disks, or
zookeeper on the nodes in the course of the same time.

What is causing this data loss? How can I debug it? How can I stem it?
Any chances I can repair the missing inodes?

I have 5 sheepdog storage nodes, also running Zookeeper. I have another
8 "gateway only" nodes that are part of the node pool, but only running
a gateway and cache.

I have about dozen VDI images, and they've been fairly static for the
last week while I've been testing -- not a lot of write activity.

~ thornton

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wpkg.org/pipermail/sheepdog-users/attachments/20150128/8d18380b/attachment-0005.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: application/octet-stream
Size: 1300 bytes
Desc: not available
URL: <http://lists.wpkg.org/pipermail/sheepdog-users/attachments/20150128/8d18380b/attachment-0010.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: application/octet-stream
Size: 770 bytes
Desc: not available
URL: <http://lists.wpkg.org/pipermail/sheepdog-users/attachments/20150128/8d18380b/attachment-0011.obj>