[Sheepdog] [RFC PATCH 0/4] add automated testing framework

[MORITA Kazutaka]

At Sun, 30 Oct 2011 16:40:23 +0800,
Liu Yuan wrote:
> 
> On 10/28/2011 12:08 PM, MORITA Kazutaka wrote:
> 
> > Hi all,
> > 
> > So far, we have been suffering from many timing problems.  This
> > experience convinces me that we should create system testing framework
> > rather than unit testing one we discussed before.  This patchset
> > enables us to create a virtual Sheepdog cluster with any number of
> > nodes only on localhost.  We can create any scenario (e.g. create many
> > VDIs at the same time on three nodes) with a simple python script.
> > You can run all of testcases automatically with 'make check' on top of
> > the source directory.
> > 
> > NOTE: I wrote this patchset to start discussing Sheepdog tests.  I
> > hope someone would create a better one.
> > 
> > 
> > MORITA Kazutaka (4):
> >   add make check support for testing
> >   sheep: add debug output to use for testing
> >   tests: add Sheepdog cluster emulation library
> >   tests: add sample testcases
> > 
> >  Makefile.am              |    2 +-
> >  configure.ac             |    6 ++-
> >  sheep/group.c            |    8 +++
> >  tests/Makefile.am        |   40 ++++++++++++++
> >  tests/sheepdog_test.py   |  135 ++++++++++++++++++++++++++++++++++++++++++++++
> >  tests/test_membership.py |   70 ++++++++++++++++++++++++
> >  tests/test_vdi.py        |   26 +++++++++
> >  7 files changed, 285 insertions(+), 2 deletions(-)
> >  create mode 100644 tests/Makefile.am
> >  create mode 100644 tests/sheepdog_test.py
> >  create mode 100644 tests/test_membership.py
> >  create mode 100644 tests/test_vdi.py
> > 
> 
> 
> How about using higher level infterface (ascii + numeral) on top of
> python script for test cases? This would help non-pyhoner write test
> cases fast and more expressive without python specific flow controls
> that mess up testcase flow.
> 
> for e.g
> cluster.copies = 3; #copies == 3
> create(0,1,2,3,4,5)
> assert(cluster.nr == 6)
> kill(0,1,2,3))
> assert(cluster.stat == halt)
> join(0,1,2,3)
> assert(cluster.nr == 6 && cluster.stat == running))
> 
> #test for shutdown auto-recovery
> shutdown(cluster)
> corrupt_epoch_log(0,1,2,3)
> join(0,1,2,3,4,5)
> assert(cluster.nr == 2 && cluster.stat == halt)
> join(0,1,2,3)
> assert(cluster.nr == 6 && cluster.stat == running)
> 
> kill(*) # * means all the nodes in the cluster
> join(0,1,2,3,4,5)
> assert(cluster.nr == 1 && cluster.stat == halt)
> join(0,1,2,3,4)
> assert(cluster.nr == 6 && cluster.stat == running)
> ....
> 
> we need to implement a simple lexical & semantic (or whatever) parser

I don't mind how to describe testcases too much, but isn't it a bit
tough to implement those?


> to map those primitives to python low level framework.

If we use another interface for testcases, I think there is no reason
to use Python for testing framework.


Thanks,

Kazutaka