How to Effectively Reduce Tens of Millions of Tests: An Industrial Case Study on Adaptive Random Testing

Running and analyzing a large number of tests in an industrial scenario is labor intensive and time consuming. Hence, it is necessary to select a smaller number of tests for cost reduction as well as fault detection. For a type of nonnumeric systems, the linear-order algorithm for adaptive random testing (ART) (LART) technique is proposed by making tests evenly spread in nonnumeric input domains. To further enhance LART in the industrial scenarios where the number of input categories is too large, a new technique called category selection-based ART (CSBART), in which partial categories are selected to calculate tests’ distances to guide LART, is proposed in this article. The fault-coverage effectiveness of CSBART is evaluated via an empirical study on two large scale billing systems with tens of millions of test cases, and the results demonstrate the promising performance of the proposed CSBART. We also find that, after category selection, CSBART can outperform a more complex and widespread n-per cluster sampling technique that uses K-means clustering to certain extents.