Gray-box combinatorial interaction testing

Official URL: http://risc01.sabanciuniv.edu/record=b1606660 (Table of Contents)

The enourmous size of configuration spaces in highly configurable softwares pose challenges to testing. Typically exhaustive testing is neither an option nor a way. Combinatorial interaction techiques are a systematic way to test such enourmous configuration spaces by a systematic way of sampling the space, employed through covering arrays. A t-way covering array is a sampled subset of configurations which contains all t-way option setting combinations. Testing through t-way covering arrays is proven to be highly e ective at revealing failures caused by interaction of t or fewer options. Although, traditional covering arrays are e ective however, we’ve observed that they su er in the presence of complex interactions among configuration options, referred as tangled options. A tangled configuration option is described as either a configuration option with complex structure and/or nested in hierarchy of configuration options. In this thesis, we conjecture the e ectiveness of CIT in the presence of tangled options can greatly be improved, by analyzing the system’s source code. The analysis of source code reveals the interaction of configuration options with each other, this information can be used to determine which additional option setting combinations and the conditions under which these combinations must be tested. Gray-box testing methods rely on partial structural information of the system during testing. We’ve statically analyzed the source code of subject applications to extract the structure and hierachy of configuration options. Each configuration option has been structurally tested according to a test criterion against a t-way covering array and subsequently their t-way interactions. The criterion revealed the missing coverage of options which were employed to drive the additional testcase generation phase to acheive complete coverage. We present a number of novel CIT coverage criteria for t-wise interaction testing of configuration options. In this thesis, we’ve conducted a series of large scale experiments on 18 di erent real-world highly configurable software applications from di erent application domains to evaluate the proposed approach. We’ve observed that traditional t-way CAs can provide above 80% coverage for configuration options testing. However, they significantly su er to provide interaction coverage under high t and tangling e ects where coverage is dropped to less than 50%. Our work address these issues and propose a technique to acheive complete coverage.