[Libstoragemgmt-devel] Ideas about tests.

Tony Asleson

2014-06-27 15:09:11 UTC

Post by Gris Ge
Hi All,
I would like to share some ideas about lsm tests.
tester.c # Testing LSM C binding.

This is the C client API running against the C simulator and the Python
simulator. I use this with code coverage tools to try and ensure that
we are covering as much code as we can, including error paths in
parameter passing etc.

Post by Gris Ge
plugin_test.c # Testing plugin
cmdtest.py # Testing lsmcli

This runs against the python simulator and uses the python client API.

Post by Gris Ge
* It's blur whether they should be capacity driven or simulator only.
The tester.c and cmdtest.py are checking information about
simulator plugin, like how many volumes we could get on query,
use lsm_test_pool for volume or fs creating and etc.

The tests have evolved over time. Because of the requirement of
external arrays for testing specific plug-ins it's difficult to test
everything by using one test.

Post by Gris Ge
* We are putting all test cases into single file as subroutines
which is hard to maintain.

Can you elaborate on this? We have had zero concurrent modifications to
the unit tests. The C unit test uses the library check which executes
the units tests in a separate process so that the unit test itself
doesn't die of the unit test crashes with a seg fault for example. Each
unit test stands alone. I'm not against breaking this out into separate
files, especially if we can increase the depth and breath of testing,
but for the most part I don't think it has caused any maintenance
issues. We could certainly reduce/eliminate the output.

Overall my approach has been.

1. Test everything we can in the library without the need for an array
and incorporate that into the build.
2. Test plug-ins against as many arrays that are supported by that
plug-in and test at least once a day.

Post by Gris Ge
* Python Binding
* C Binding
* lsmcli
* Plugin test
# Plugin test focus on validating how much does a plugin
# supported. We use it to generate hardware support page.

The current plug-in test does this.

Post by Gris Ge
* Library test
# This kind of test is runnable on real plugin also, but suggested
# be against simulator plugin only.

This would be the current C unit test and the cmdtest.py.

Post by Gris Ge
1. Scalability test
# Example, querying 1000+ volumes.

I've done this out to 50K+.

Post by Gris Ge
2. Constants test.
# Comparing C binding constants with Python binding.

Not sure the best way to automate this except maybe writing a C program
and a python one and have them dump out the constants and compare
values. Or maybe write a script that looks at the source code for both
and checks the values.

Post by Gris Ge
3. The output format of lsmcli.

We are missing this today

Post by Gris Ge
4. Validate lsmcli command options.

We kind of have some of this.

Post by Gris Ge
A. Library test
B. Plugin + Library test

The library gets exercised when you are exercising the plug-in.

Post by Gris Ge
3. Performance check.
# Comparing performance with provided stable performance data.
# In case any patch introduce performance regression.

The library for the most part is quite simple and performance really
shouldn't be an issue. The biggest pain points I've seen so far is in
the plug-in implementations.

Post by Gris Ge
1. All test cases would be run against any URI. Even may be
skipped as capability check fail.
2. Simulate how library user would do.
3. User case driven, test case code could be used as sample code.
4. Test case could run directly without lsm_test.py assistance.
test/lsm_test.py
test/c_tests/001_query_sys.c
# Explain the test procedure
test/py_tests/001_query_sys.py
test/lsmcli_tests/001_query_sys.py
lsm_test.py \
-u uri [-p pass] [-p performance_file] [-t test_case] [-s test_set]
[-g performance_file] [-l log_folder]
# If 'test_set' == 'plugin', only plugin test will be called.
# With '-p performance_file', performance will be checked and
# ranked.
# The '-g performance_file' is to generate performance data for
# next comparison.
* Invoking test cases
* Base on test_set to choose test cases to run.
# All executable file in c_tests, py_tests, lsmcli
# named as xxx_ (^[0-9]{3}_) will be considered as a test case.
# with out we don't need to maintain a config file for
# test cases list.
* Test case will be invoked with
-u uri [-p pass] [-s test_set]
# Test case can skip complex library validation if [-s plugin]
# define requesting plugin test only.
* Generate fancy report.
* Gather debug log from lsmd and plugin.
# currently, plugin does not have debug log yet.

We log errors to syslog, but yes we don't have a very detailed debug
facility.

Post by Gris Ge
* Basically, with the same test number, c_test, py_tests and
lsmcli_tests are following the same test procedure. With that
we can ensure user can use any way we supported to meet their
needs.
* If possible, let lsm_test.py start lsmd and plugin, which
* could introduce memory leak check and crash dump.
URI=sim:// # First line for URI
#[test_name] [time_in_seconds]
c_tests/001_query 25
lsm_test.py will calculate out a percentage about performance.
+10%(30/25) # Bad performance

Many of the arrays are shared resources which have varying amounts of
concurrent usage and activity. I'm not sure we can get accurate results
with the variability of usage.

Post by Gris Ge
* Invoke test case file without parameter will got this kind of
# Line 1: Type # Library or Plugin
# Line 2: Capabilities needed for this test.
1 2
# 1 for lsm.Test.TEST_TYPE_LIBRARY
# 2 for lsm.Test.TEST_TYPE_PLUGIN
20 21 33
# 20 is lsm.Capabilities.VOLUMES
# 21 is lsm.Capabilities.VOLUME_CREATE
# 22 is lsm.Capabilities.VOLUME_DELETE
* If request
0 PASS
1 FAIL # Bug spotted.
# As lsm_test.py already checked the capability before
# invoking certain test cases, if NO_SUPPORT error found
# during test, it should be treated as FAIL.

Conversely if an array says it doesn't support something we should call
it anyway and make sure that is does return unsupported. The plug-in
test has a non-zero exit code when it fails.

Post by Gris Ge
URI: sim://
LSM Version: 0.1.0
# test case name # Result # Performance (if provided)
c_tests/001_query PASS +10%(30/25)
All test case use STDOUT and STDERR, lsm_test.py capture them
by saving them into PWD folder(if -l log_folder not defined) as
lsm_test_log/test_result.log
lsm_test_log/<test_name>_<time_stamp>.log
lsm_test_log/<test_name>_<time_stamp>.err
lsm_test_log/syslog.log
# need a way to parse or capture syslog from lsmd and
# plugins.

The plug-in test does output yaml to stdout with the results of each
call into the library. It also outputs performance numbers on the
execution time. This information is then fed into other tools which
reside in the test/webtest directory which then produces html. I'm not
doing anything with the performance numbers.

As hardware arrays can be spread across numerous physical locations I
was thinking that it would be good to have a test service that is
publicly available with results that can be logged too. Basically you
take the yaml output of the tests and write it to a test service and
that would parse and update the results and publish a new web page or
generate email summaries etc. Today I have cron jobs running in a
couple of different labs running tests that automatically pick up the
latest code committed to master branch and run tests. I have to
manually go to each lab to look at results.

I'm more interested in the quality of the tests and what they cover and
how easy they are to use. Today we have varying amounts of code
coverage and we definitely need to improve the plug-in test. There are
a number of things that are not covered that should be.

I'm all for making things easier to use and easier to write.

Regards,
Tony