Feedback driven adaptive combinatorial testing

Dumlu, Emine and Yılmaz, Cemal and Cohen, Myra B. and Porter, Adam (2011) Feedback driven adaptive combinatorial testing. In: ACM International Symposium on Software Testing and Analysis (ISSTA 2011), Toronto, ON, Canada

[thumbnail of ISSTA-2011.pdf] PDF

Download (340kB)


The configuration spaces of modern software systems are too large to test exhaustively. Combinatorial interaction testing (CIT) approaches, such as covering arrays, systematically sample the configuration space and test only the selected configurations. The basic justification for CIT approaches is that they can cost-effectively exercise all system behaviors caused by the settings of t or fewer options. We conjecture, however, that in practice many such behaviors are not actually tested because of masking effects – failures that perturb execution so as to prevent some behaviors from being exercised. In this work we present a feedback-driven, adaptive, combinatorial testing approach aimed at detecting and working around masking effects. At each iteration we detect potential masking effects, heuristically isolate their likely causes, and then generate new covering arrays that allow previously masked combinations to be tested in the subsequent iteration. We empirically assess the effectiveness of the proposed approach on two large widely used open source software systems. Our results suggest that masking effects do exist and that our approach provides a promising and efficient way to work around them.
Item Type: Papers in Conference Proceedings
Uncontrolled Keywords: Combinatorial testing, adaptive testing, covering arrays, software quality assurance
Subjects: Q Science > Q Science (General)
Divisions: Faculty of Engineering and Natural Sciences > Academic programs > Computer Science & Eng.
Faculty of Engineering and Natural Sciences
Depositing User: Cemal Yılmaz
Date Deposited: 20 Sep 2011 16:13
Last Modified: 26 Apr 2022 09:01

Actions (login required)

View Item
View Item