2-wise testing or pairwise testing

If the wise-1 test guarantees coverage of every single value for each variable, it is easy to see that a wise-2 test set ensures that all pairs of values of the variables are covered at least once. In the case of the variables listed in Tabella 1, all pairs of variables are as follows: {(A, B), (A, C), (B, C)}. In fact, the combinatorial calculation shows that the number of combinations of N values taken K to K (with N ≥ K) is equal to:

Combination

In our example we have three variables (N=3) taken two to two (K=2), an applying the combinatorial formula above we get 

 

Pairs

 

The three pairs are {(A, B), (A, C), (B, C)}.

 

Wanting to compute all possible pairs of variable values, we need to consider the following table:

 

PAIR

# VARIABLES VALUE

TOTAL

A

B

C

(A,B)

4

3

   4*3=12

(A,C)

4

 

2

 4*2=8

(B,C)

 

3

2

 3*2=6

GRAND TOTAL

 12+8+6=26

Tabella 5Counting of the pairs of values of the variables A, B and C

 

Hence, the total of all the pairs of values of the variables A, B and C whose values are reported in Tabella 1is equal to 26 and are all printed in the following table:

#

# COPPIE VALORI

A, B

A, C

B, C

1

A1,B1

A1,C1

B1, C1

2

A1,B2

A1,C2

B1, C2

3

A1,B3

A2,C1

B2, C1

4

A2,B1

A2,C2

B2, C2

5

A2,B2

A3,C1

B3, C1

6

A2,B3

A3,C2

B3, C2

7

A3,B1

A4,C1

 

8

A3,B2

A4,C2

 

9

A3,B3

   

10

A4,B1

   

11

A4,B2

   

12

A4,B3

   

# COPPIE

12

8

6

TOTALE

12+8+6=26

Tabella 6Pairs of valuesof the variablesA, B andC

 

Why you should consider a Test Set to cover wise-2? Not be enough to consider a Test Set with 1-wise coverage? Here we enter into a thorny issue, in which the opinions are varied, concordant and discordant.

Belowis the “incipit”from the site http://www.pairwise.org/ :

 

“Pairwise (a.k.a. all-pairs) testing is an effective test case generation technique that is based on the observation that most faults are caused by interactions of at most two factors. Pairwise-generated test suites cover all combinations of two therefore are much smaller than exhaustive ones yet still very effective in finding defects.”

 

We mention also the opinion of James Bach and Patrick J. Schroeder about the Pair Wise method: "Pairwise Testing: A Best Practice That Is not" from James Bach, Patrick J. Schroeder available from http://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf ):

 

“What do we know about the defect removal efficiency of pairwise testing? Not a great deal. Jones states that in the U.S., on average, the defect removal efficiency of our software processes is 85% [26]. This means that the combinations of all fault detection techniques, including reviews, inspections, walkthroughs, and various forms of testing remove 85% of the faults in software before it is released.

In a study performed by Wallace and Kuhn [27], 15 years of failure data from recalled medical devices is analyzed. They conclude that 98% of the failures could have been detected in testing if all pairs of parameters had been tested (they didn’t execute pairwise testing, they analyzed failure data and speculate about the type of testing that would have detected the defects). In this case, it appears as if adding pairwise testing to the current medical device testing processes could improve its defect removal efficiency to a "best in class" status, as determined by Jones [26].

On the other hand, Smith, et al. [28] present their experience with pairwise testing of the Remote Agent Experiment (RAX) software used to control NASA spacecraft. Their analysis indicates that pairwise testing detected 88% of the faults classified as correctness and convergence faults, but only 50% of the interface and engine faults. In this study, pairwise testing apparently needs to be augmented with other types of testing to improve the defect remove al efficiency, especially in the project context of a NASA spacecraft. Detecting only 50% of the interface and engine faults is well below the 85% U.S. average and presumably intolerable under NASA standards. The lesson here seems to be that one cannot blindly apply pairwise testing and expect high defect removal efficiency. Defect removal efficiency depends not only on the testing technique, but also on the characteristics of the software under test. As Mandl [4] has shown us, analyzing the software under test is an important step in determining if pairwise testing is appropriate; it is also an important step in determining what addition al testing technique should be used in a specific testing situation.”

[4] R. Mandl, "Orthogonal Latin Squares: An Application of Experiment Design to Compiler Testing," Communication of the ACM, vol. 28, no. 10, pp. 1054-1058, 1985.

[26] Jones, Software Assessments, Benchmarks, and Best Practices. Boston, MA: Addison Wesley Longman, 2000.

[27] D. R. Wallace and D. R. Kuhn, "Failure Modes in Medical Device Software: An Analysis of 15 Years of Recall Data," Int'l Jour. of Reliability, Quality and Safety Engineering, vol. 8, no. 4, pp. 351-371, 2001.

[28] B. Smith, M. S. Feather, and N. Muscettola, "Challenges and Methods in Testing the Remote Agent Planner," in Proc. 5th Int'l Conf. on Artificial Intelligence Planning and Scheduling (AIPS 2000), 2000, pp. 254-263

To clarify,we can say that Pairwise or 2-wise test method ensures that all combinations of pairs of values ​​of the variables are tested, "theoretically ensuring" the maximization of the anomalies found, with percentages ranging from 50% to 98% according to the studies. In fact, no test can ever guarantee a defined percentage removal of defects (which can only be calculated ex post for the specific project); let's say trying to be realistic - that the Pairwise achieve a valid agreement between the number of tests to be performed and the anomalies detected, when the number of variables and their values is so high that it cannot be tested all the combinations (the so called all-wise testing or N-wise testing, where N is the number of variables we are playing with).

In the case of Test Set covering wise-2 level is very simple to know the maximum number of tests that provide coverage for all pairs of values of the variables. This value is equal to the number of pairs of values of the variables themselves. In our example, three variables A, B and C in Tabella 1, this number is 26 (calculated as described in Tabella 6). The real problem – still unsolved - is to discover the minimum number of tests that guarantees wise-2 coverage, although there are a variety of methods and algorithms that approximate this value for a problem with an arbitrary number of variables and values. Examples of tools that use these algorithms are:

  1. Microsoft Pairwise Independent Combinatorial Testing tool (PICT), downloadable from http://download.microsoft.com/download/f/5/5/f55484df-8494-48fa-8dbd-8c6f76cc014b/pict33.msi;
  2. NIST Tools: http://csrc.nist.gov/groups/SNS/acts/documents/comparison-report.html
  3. James Bach AllPairs downloadable from http://www.satisfice.com/tools.shtml ;
  4. Other tools here: http://www.pairwise.org/tools.asp .

Web Traffic

Today4
Yesterday34
This week35822
This month579
Total35822

Who Is Online

1
Online

Saturday, 16 December 2017 01:12
Powered by CoalaWeb