Test case-aware combinatorial interaction testing

Yılmaz, Cemal (2013) Test case-aware combinatorial interaction testing. IEEE Transactions on Software Engineering, 39 (5). pp. 684-706. ISSN 0098-5589

This is the latest version of this item.

[img]PDF - Registered users only - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader

Official URL: http://dx.doi.org/10.1109/TSE.2012.65


The configuration spaces of modern software systems are too large to test exhaustively. Combinatorial interaction testing (CIT) approaches, such as covering arrays, systematically sample the configuration space and test only the selected configurations by using a battery of test cases. Traditional covering arrays, while taking system-wide inter-option constraints into account, do not provide a systematic way of handling test case-specific inter-option constraints. The basic justification for t-way covering arrays is that they can cost-effectively exercise all system behaviors caused by the settings of t or fewer options. In this work we hypothesize, however, that in the presence of test case-specific inter-option constraints, many such behaviors may not be tested due to masking effects caused by the overlooked test case-specific constraints. For example, if a test case refuses to run in a configuration due to an unsatisfied test case-specific constraint, none of the valid option setting combinations appearing in the configuration will be tested by that test case. To account for test case-specific constraints, we introduce a new combinatorial object, called a test case-aware covering array. A t-way test case-aware covering array is not just a set of configurations as is the case in traditional covering arrays, but a set of configurations, each of which is associated with a set of test cases, such that all test case-specific constraints are satisfied and that, for each test case, each valid combination of option settings for every combination of t options appears at least once in the set of configurations that the test case is associated with. We furthermore present three algorithms to compute test case-aware covering arrays. Two of the algorithms aim to minimize the number of configurations required (one is fast, but produces larger arrays, the other is slower, but produces smaller arrays), whereas the remaining algorithm aims to minimize the number of test runs required. The results of our empirical studies conducted on two widely-used highly-configurable software systems suggest that test case-specific constraints do exist in practice, that traditional covering arrays suffer from masking effects caused by the ignorance such constraints, and that test case-aware covering arrays are better than other approaches in handling test case-specific constraints, thus avoiding masking effects.

Item Type:Article
Uncontrolled Keywords:Software quality assurance, combinatorial interaction testing, covering arrays
Subjects:Q Science > Q Science (General)
ID Code:23307
Deposited By:Cemal Yılmaz
Deposited On:17 Jan 2014 11:15
Last Modified:01 Aug 2019 15:34

Available Versions of this Item

Repository Staff Only: item control page