The combinatorial testing techniques presented here, are intended to solve the following cominatorial test basic problem:
DIRECT COMBINATORIAL TEST PROBLEM: "A software system accepting N input variables, each of which can take on different values, find the Test Set with the smaller number of test cases that guarantee (at least) the coverage of all combinations (2-tuples) of all the values of the variables involved.”
To solve this problem has been developed Pairwise technique and a large number of support tools. Once such a test set (the smallest possible) has been generated, you should run the test cases and detect (if present) all the defects comes up in the software under test.
There is also a second problem, maybe less "popular" compared to the previous, that is the following:
REVERSE COMBINATORIAL TEST PROBLEM:"Given a Test Set for which you do not know the method of generation (if any), calculate what percentage of nwise-coverage the Test Set ensures, with nwise between 1 and the number variables in the Test Set ".
An example is that in which tests are generated by automated tools for which you have a low or almost zero process control, or when the "test cases" are generated by the automatic flows that feed interfaces between different systems (think of a system that transmit accounting data from a system A to B); test data are - in general - excerpts from historical series on which you have no control.
For test scenarios in some way related to a combinatorial inverse problem, it’s not easy found support tools or such tools are not readily available. The only tool I found is NIST CCM, still in alpha-phase at the moment I’m writing this notes.
In the following we describe a set of tools called "Combinatorial Testing Tools" (executable under Windows, but -if needed– not difficult to port under Unix / Linux) allowing to extract the (quasi)minimal test set and calculate the coverage of a generic test set , using systematic algorithms calculation coverage, starting from all n- tuples of the variables’ values; such algorithms should be categorized "brute force algorithms" and should be used (on a normal supermarket-buy-PC) if the number of variables and values is not too high.
Tools in Combinatorial Testing Tools product (CTT in the following) try to provide a support to the solution of both problems of combinatorial test previously stated.
CTT do not intend to compete with the existing tools aimed to solve the direct problem of combinatorial testing, such as Microsoft PICT, AllPair J. Bach, NIST ACTS or other several commercial and non-commercial tools already present on the market. Those tools implement algorithms definitely more effective than CTT and therefore should be favorite – I repeat – to solve the direct problem.
Regarding the reverse problem of combinatorial tests do not exist, to my knowledge, tools on the market (except NIST CCM in alpha-test) and the CTT then attempt to provide a very first solution to the reverse problem, to be surely improved over time, when we will be better understood the logic and the rules that are behind combinatorial test and the minimal determination of test sets related to it.
We would like to remember that:
- Solving the direct problem means determining the as small as possible test set with a level of WISE coverage agreed (usually WISE = 2) from the set of variables and values.
- Solve the reverse problem means determining the level of coverage of a given test set with respect to a reference WISE level (also here usually is WISE = 2)
The tools should be categorized as follow:
- First level Tools: batch DOS scripts providing an immediate response to standard scenarios that typically occur in test projects requiring combinatorial techniques.
- Second level Tools: executable (C++/Perl development language) more versatile giving response to problems that may be less common, but sometimes occurs in test projects requiring combinatorial techniques.
First level scripts were thought as the "wrapper" around the second level executables, in order to "simplify end user life" with a set of simple commands that allow to quickly get a number of “standard information”.