Testing
Introduction
From the outset, it was decided that a testing regime was required to support debugging of subsystems and validation of
mathematical routines. The basic requirements for the testing infrastructure were:
Standardised: for all developers.
Automated: to allow test execution with no user input.
Simple: so as to minimise load placed on developers.
GPL: The code must be released under the GPL.
Implementation
Unit testing, as adopted by the extreme programming community, was chosen as the ideal framework to
support our system. A number of different libraries were examined as possible candidates and CPPUnit was
selected as the most appropriate.
CPPUnit is a C++ framework based closely on the very mature JUnit testing framework. It allows the developers
to write simple, focused tests, and provides a number of assertions with which to validate results. Additionally,
the tests have special methods which are designed to support test independence.
The first is setUp(),
which is executed immediately preceding each test, and the second is tearDown() which is run following
after each test. These allow the developers define dependencies for each test and deallocate resources that
the test required.
JUnit makes extensive use of the reflection Application Programmers Interface (API) to locate test methods. Methods are deemed
to be tests if their name
begins with the word "test", they take no parameters and their return type is void. Due to limitations in the
C++
language, there is no reflection API as such, thus ruling out reflection as an alternative. Consequently CPPUnit
requires the user to register tests with the TestCaller via a static method. The Test class is then registered
with the TestRunner and all registered tests are run. CPPUnit provides a number of macros which ease this registration
process.
Errors and failures in CPPUnit are reported via a number of macros. These macros provide the ability to test
for equality and truth and allow the user to supply meaningful error messages, thereby facilitating the debugging process.
Failures,
with the exception of segmentation violations, do not cause the tests to exit. If a failure occurs, the current test
reports the error and returns, allowing the test suite to continue.
Early in the development of the project, a template was developed to provide a simple means by which developers could
start their testing. A sample test, based on this template follows.
namespace {
class Test : public CppUnit::TestFixture {
public:
/* default constructor */
Test() : CppUnit::TestFixture() {}
/** setup - run prior to each test */
void setUp ()
{ /* initialise any resources*/ }
/** teardown - run after each test */
void tearDown ()
{ /* free any allocated resources */ }
/**
* constructs a test suite.
* Add your tests to the suite by copying and editing the addTest call
*/
static CppUnit::Test *suite()
{
CppUnit::TestSuite *suiteOfTests = new CppUnit::TestSuite( "Test" );
/* REGISTER YOUR TEST HERE */
suiteOfTests->addTest(
new CppUnit::TestCaller<Test>( "sampleTest", &Test::sampleTest ) );
return suiteOfTests;
}
protected:
/**
* DEFINE YOUR TESTS HERE:
* make your tests protected since you do not need to expose them
*/
/** simple test 1. uses a generic assert true macro */
void sampleTest ()
{
char* foo = (char*)malloc(32);
CPPUNIT_ASSERT_MESSAGE ("foo failed", foo != 0);
free(foo);
}
}; // end Test
/**
* Register the test immediately after definition. This should probably
* be done in the class header file for larger projects
*/
CPPUNIT_TEST_SUITE_REGISTRATION( Test );
} // end namespace
Testing has been used extensively to support the validation of the mathematical subsystem. There are currently 84 tests
spread across the 10 mathematical modules.
User Interface Testing
The user interface is exhaustively tested across all datasets. The tests are generated automatically and seek to locate
datasets with
peculiarities that might cause the failure of the plotting library. There are currently 1131 tests dedicated to exercising
this body of code.
Conclusion
The testing framework proved invaluable in the process of bug isolation and resolution. An additional benefit has been
the early recognition of design problems, such as tight coupling between the subsystems and the user interface at an early
stage in the
project. In summary, without the support of a system such as CPPUnit, the GDMS would not have the level
of robustness
that it does at this time and mathematical validation would have been difficult.