[Sheepdog] [RFC PATCH 0/4] add automated testing framework

[Liu Yuan]

On 10/31/2011 07:33 AM, MORITA Kazutaka wrote:

> At Sun, 30 Oct 2011 16:40:23 +0800,
> Liu Yuan wrote:
>>
>> On 10/28/2011 12:08 PM, MORITA Kazutaka wrote:
>>
>>> Hi all,
>>>
>>> So far, we have been suffering from many timing problems.  This
>>> experience convinces me that we should create system testing framework
>>> rather than unit testing one we discussed before.  This patchset
>>> enables us to create a virtual Sheepdog cluster with any number of
>>> nodes only on localhost.  We can create any scenario (e.g. create many
>>> VDIs at the same time on three nodes) with a simple python script.
>>> You can run all of testcases automatically with 'make check' on top of
>>> the source directory.
>>>
>>> NOTE: I wrote this patchset to start discussing Sheepdog tests.  I
>>> hope someone would create a better one.
>>>
>>>
>>> MORITA Kazutaka (4):
>>>   add make check support for testing
>>>   sheep: add debug output to use for testing
>>>   tests: add Sheepdog cluster emulation library
>>>   tests: add sample testcases
>>>
>>>  Makefile.am              |    2 +-
>>>  configure.ac             |    6 ++-
>>>  sheep/group.c            |    8 +++
>>>  tests/Makefile.am        |   40 ++++++++++++++
>>>  tests/sheepdog_test.py   |  135 ++++++++++++++++++++++++++++++++++++++++++++++
>>>  tests/test_membership.py |   70 ++++++++++++++++++++++++
>>>  tests/test_vdi.py        |   26 +++++++++
>>>  7 files changed, 285 insertions(+), 2 deletions(-)
>>>  create mode 100644 tests/Makefile.am
>>>  create mode 100644 tests/sheepdog_test.py
>>>  create mode 100644 tests/test_membership.py
>>>  create mode 100644 tests/test_vdi.py
>>>
>>
>>
>> How about using higher level infterface (ascii + numeral) on top of
>> python script for test cases? This would help non-pyhoner write test
>> cases fast and more expressive without python specific flow controls
>> that mess up testcase flow.
>>
>> for e.g
>> cluster.copies = 3; #copies == 3
>> create(0,1,2,3,4,5)
>> assert(cluster.nr == 6)
>> kill(0,1,2,3))
>> assert(cluster.stat == halt)
>> join(0,1,2,3)
>> assert(cluster.nr == 6 && cluster.stat == running))
>>
>> #test for shutdown auto-recovery
>> shutdown(cluster)
>> corrupt_epoch_log(0,1,2,3)
>> join(0,1,2,3,4,5)
>> assert(cluster.nr == 2 && cluster.stat == halt)
>> join(0,1,2,3)
>> assert(cluster.nr == 6 && cluster.stat == running)
>>
>> kill(*) # * means all the nodes in the cluster
>> join(0,1,2,3,4,5)
>> assert(cluster.nr == 1 && cluster.stat == halt)
>> join(0,1,2,3,4)
>> assert(cluster.nr == 6 && cluster.stat == running)
>> ....
>>
>> we need to implement a simple lexical & semantic (or whatever) parser
> 
> I don't mind how to describe testcases too much, but isn't it a bit
> tough to implement those?


if we use compiler-generator tools such as flex & bison, or python
equivalent ones, not that hard to implement since we don't have to
implement 'function call', aggregate data structure, etc.

anyway, it is a bit to make a fuss over a trifling matter. but would be
nice if anyone gives it a try.

Yuan

> 
>> to map those primitives to python low level framework.
> 
> If we use another interface for testcases, I think there is no reason
> to use Python for testing framework.