[Sheepdog] Sheepdog 0.3.0 schedule and 0.4.0 plan

[Chris Webb]

MORITA Kazutaka <morita.kazutaka at lab.ntt.co.jp> writes:

> I finished all of the cluster driver implementation we planed, so I
> think of releasing 0.3.0 this weekend.  If you have pending patches
> for 0.3.0, please send them until Nov 18th.  I'll spend this week
> testing Sheepdog.

Hi. I thought I would make myself useful and grab the head of master today
(currently 45eb24f01f8a) and do some testing of the code you're about to cut
into a release.

I repeated my tests that caused problems a week or so ago, and everything
seemed great. I created a small cluster, and did reads+writes through
collie, booted some guests, heavily exercised the attr support for locking
and so on. Everything seems very solid so far.

As a final test, I thought I'd try failing a node as I haven't done that for a
while. This didn't work as I'd hope I'm afraid. I then reproduced the problem
at a smaller scale on my small three node (with three sheep per node) cluster.

The nodes are numbered 002{6,7,8) with IP addresses 172.16.101.{7,9,11}
respectively, and sheep running on ports 7000, 7001 and 7002 on each node as
sheep -D -p 700{0,1,2}. Sheepdog and corosync IO (both multicast and
unicast) happens on interface eth1 in what follows.

I fail node 0028 by taking out its network connection:

  0028# ip link set eth1 down

At this point, all the sheep on this node have gone away (correctly):

  0028# ps ax | grep [s]heep
  0028# 

but the log suggests they may not have died in quite the intended way:

  0028# tail /mnt/sheep-0028-00/sheep.log 
  Nov 18 10:35:00 cluster_queue_request(221) 0x17304e0 89
  Nov 18 10:35:00 do_lookup_vdi(234) looking for 3f6d01e8-b627-4b1d-b2b8-e33daa571c25 (5c29e4)
  Nov 18 10:35:00 cluster_queue_request(221) 0x17304e0 89
  Nov 18 10:35:00 do_lookup_vdi(234) looking for 83ec9d3c-962d-47dc-9f9f-f4e4f4668767 (cc54cb)
  Nov 18 10:35:00 cluster_queue_request(221) 0x17304e0 89
  Nov 18 10:35:00 do_lookup_vdi(234) looking for 83ec9d3c-962d-47dc-9f9f-f4e4f4668767 (cc54cb)
  Nov 18 10:40:17 ob_open(445) failed to open /mnt/sheep-0028-00/obj/00000001/20cc54cb5ed7a388: No such file or directory
  Nov 18 10:40:23 ob_open(445) failed to open /mnt/sheep-0028-00/obj/00000001/20cc54cb5ed7a389: No such file or directory
  Nov 18 10:48:11 sd_leave_handler(1291) network partition bug: this sheep should have exited
  Nov 18 10:48:13 log_sigsegv(358) logger pid 2082 exiting abnormally

On the other nodes, nothing works any more, e.g.

  0026# collie vdi list
    name        id    size    used  shared    creation time   vdi id
  ------------------------------------------------------------------
  failed to read object, 805c29e400000000 Remote node has a new epoch
  failed to read a inode header
  failed to read object, 80cc54cb00000000 Remote node has a new epoch
  failed to read a inode header

Presumably something about sheepdog fail-over has broken recently? I don't seem
to be able to get a cluster with a failed node to work in any configuration I
try, so I guess this might be very easy to reproduce, but in case not, the
sheep.log files from this simple test are at

  http://cdw.me.uk/tmp/sheep-0026-00.log
  http://cdw.me.uk/tmp/sheep-0026-01.log
  http://cdw.me.uk/tmp/sheep-0026-02.log
  http://cdw.me.uk/tmp/sheep-0027-00.log
  http://cdw.me.uk/tmp/sheep-0027-01.log
  http://cdw.me.uk/tmp/sheep-0027-02.log
  http://cdw.me.uk/tmp/sheep-0028-00.log
  http://cdw.me.uk/tmp/sheep-0028-01.log
  http://cdw.me.uk/tmp/sheep-0028-02.log

There seem to be a lot of EMISSING-type errors in there, both during normal
running and after the failure. Perhaps these are a clue?

The good news is that this fail-over issue is the only problem I've been able
to unearth in the proposed release. Everything else seems fine.

Best wishes,

Chris.