Transforming test suites into croissants

Chen, Yang and Yıldız, Alperen and Marinov, Darko and Jabbarvand, Reyhaneh (2023) Transforming test suites into croissants. In: 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2023, Seattle, WA, USA

Full text not available from this repository. (Request a copy)


Software developers often rely on regression testing to ensure that recent changes made to the source code do not introduce bugs. Flaky tests, which non-deterministically pass or fail regardless of any change to the code, can negatively impact the effectiveness of the regression testing. While state-of-the-art is advancing the techniques for test-flakiness detection and mitigation, the community is missing a systematic approach for generating high-quality benchmarks of flaky tests to compare the effectiveness of such techniques. Inspired by the power of mutation testing in evaluating the fault-detection ability of tests, this paper proposes Croissant, a framework for injecting flakiness into the test suites to assess the effectiveness of test-flakiness detection tools in finding these tests. Croissant implements 18 flakiness-inducing mutation operators. We designed these operators to allow controlling the non-determinism involved in flakiness, i.e., making many mutants deterministically pass or fail to observe flaky behavior. Our extensive empirical evaluation of Croissant on the test suites of 15 real-world projects confirms the ability of designed mutation operators to generate high-quality mutants, and their effectiveness in challenging test-flakiness detection tools in revealing flaky tests.
Item Type: Papers in Conference Proceedings
Uncontrolled Keywords: Fault Injection; Mutation Testing; Software Testing; Test Flakiness
Divisions: Faculty of Engineering and Natural Sciences
Depositing User: Alperen Yıldız
Date Deposited: 03 Sep 2023 16:11
Last Modified: 03 Sep 2023 16:12

Actions (login required)

View Item
View Item