[Sheepdog] [RFC PATCH 0/4] add automated testing framework

[Liu Yuan]

On 10/28/2011 12:08 PM, MORITA Kazutaka wrote:

> Hi all,
> 
> So far, we have been suffering from many timing problems.  This
> experience convinces me that we should create system testing framework
> rather than unit testing one we discussed before.  This patchset
> enables us to create a virtual Sheepdog cluster with any number of
> nodes only on localhost.  We can create any scenario (e.g. create many
> VDIs at the same time on three nodes) with a simple python script.
> You can run all of testcases automatically with 'make check' on top of
> the source directory.
> 
> NOTE: I wrote this patchset to start discussing Sheepdog tests.  I
> hope someone would create a better one.
> 
> 
> MORITA Kazutaka (4):
>   add make check support for testing
>   sheep: add debug output to use for testing
>   tests: add Sheepdog cluster emulation library
>   tests: add sample testcases
> 
>  Makefile.am              |    2 +-
>  configure.ac             |    6 ++-
>  sheep/group.c            |    8 +++
>  tests/Makefile.am        |   40 ++++++++++++++
>  tests/sheepdog_test.py   |  135 ++++++++++++++++++++++++++++++++++++++++++++++
>  tests/test_membership.py |   70 ++++++++++++++++++++++++
>  tests/test_vdi.py        |   26 +++++++++
>  7 files changed, 285 insertions(+), 2 deletions(-)
>  create mode 100644 tests/Makefile.am
>  create mode 100644 tests/sheepdog_test.py
>  create mode 100644 tests/test_membership.py
>  create mode 100644 tests/test_vdi.py
> 


How about using higher level infterface (ascii + numeral) on top of
python script for test cases? This would help non-pyhoner write test
cases fast and more expressive without python specific flow controls
that mess up testcase flow.

for e.g
cluster.copies = 3; #copies == 3
create(0,1,2,3,4,5)
assert(cluster.nr == 6)
kill(0,1,2,3))
assert(cluster.stat == halt)
join(0,1,2,3)
assert(cluster.nr == 6 && cluster.stat == running))

#test for shutdown auto-recovery
shutdown(cluster)
corrupt_epoch_log(0,1,2,3)
join(0,1,2,3,4,5)
assert(cluster.nr == 2 && cluster.stat == halt)
join(0,1,2,3)
assert(cluster.nr == 6 && cluster.stat == running)

kill(*) # * means all the nodes in the cluster
join(0,1,2,3,4,5)
assert(cluster.nr == 1 && cluster.stat == halt)
join(0,1,2,3,4)
assert(cluster.nr == 6 && cluster.stat == running)
....

we need to implement a simple lexical & semantic (or whatever) parser
to map those primitives to python low level framework.

Thanks,
Yuan