[sheepdog] consistency problem when replicating to multi-replica

[Hitoshi Mitake]

At Thu, 08 Jan 2015 16:25:08 +0800,
guping wrote:
> 
> I find a interesting subject at
> http://copilotco.com/mail-archives/drbd.2009/msg00750.html ,
> 
> The 'concurrent local write' in DRBD is very similar to the problem in
> my previous mail.
> 
> Hope to be useful and we can come out a solution.

Thanks for sharing. IIUC, simply serializing write requests would
solve the problem in a case of sheepdog.

Thanks,
Hitoshi

> 
> 在 15/1/5 上午9:26, Hitoshi Mitake 写道:
> > At Mon, 05 Jan 2015 09:12:27 +0800,
> > guping wrote:
> >>
> >> Thanks Hitoshi,
> >>
> >> I just began to evaluate sheepdog for our cloud deployment. I will try
> >> to reproduce the problem when I have enough time. If some clue, I will
> >> let you know.
> > 
> > Thanks for your interest :) Please let us know if you find problems.
> > 
> >>
> >> A question, does the IO stack components higher than sheepdog, such as
> >> IO queue in VM, qemu block driver, etc, provide the necessary serialization?
> > 
> > I think these components doesn't have to provide the serialization
> > functionality. sheepdog (including QEMU driver) must mock the
> > sematntics of hard drives. If sheepdog allows the case of divergent
> > objects, it means sheepdog cannot preserve the semantics (two or more
> > continuous read requests to same location can return different
> > data). So the serialization should be provided by sheepdog.
> > 
> > Thanks,
> > Hitoshi
> > 
> >>
> >> 在 15/1/5 上午8:52, Hitoshi Mitake 写道:
> >>> At Sun, 04 Jan 2015 23:27:15 +0800,
> >>> guping wrote:
> >>>>
> >>>> Sheepdog send write request to all replica simultaneously to do the
> >>>> replication. Which I think could lead to divergent content in replicas.
> >>>>
> >>>> Imagine a scenario, there are two write requests A & B to update the
> >>>> same offset in a 2-replicas object. If a qemu client send A and B to a
> >>>> gateway concurrently, the gateway will send A and B to the same set of
> >>>> replicas simultaneously. Which could not guarantee the same sequence to
> >>>> apply A and B, you can apply A then B in a replica and apply B then A in
> >>>> another replica, then you get two request return successfully but not
> >>>> the same data in the replicas.
> >>>>
> >>>> Is this a consistency problem in sheepdog? How can we deal with it?
> >>>
> >>> Hi Gu,
> >>>
> >>> Currently, the above problem is prevented for a case of copy on write
> >>> requests (first write to an object). Look at
> >>> check_simultaneous_create() of sheepdog driver in qemu. Multiple COW
> >>> requests for a single object is serialized for not losing written
> >>> information illegally.
> >>>
> >>> But, the driver doesn't do the serialization for ordinal write
> >>> request. As you say, there might be a possibility of divergent
> >>> objects. Could you produce the problem with artificial workload?
> >>>
> >>> Thanks,
> >>> Hitoshi
> >>>
> >>>>
> >>>> --
> >>>> Thanks,
> >>>> Gu Ping
> >>>> --
> >>>> sheepdog mailing list
> >>>> sheepdog at lists.wpkg.org
> >>>> https://lists.wpkg.org/mailman/listinfo/sheepdog
> >>>
> >>
> >> -- 
> >> Thanks,
> >> Gu Ping
> >>
> >>
> >> -- 
> >> sheepdog mailing list
> >> sheepdog at lists.wpkg.org
> >> https://lists.wpkg.org/mailman/listinfo/sheepdog
> > 
> 
> -- 
> Thanks,
> Gu Ping
> 
> 
> -- 
> sheepdog mailing list
> sheepdog at lists.wpkg.org
> https://lists.wpkg.org/mailman/listinfo/sheepdog