Dynamic program state-based testing

Official URL: https://risc01.sabanciuniv.edu/record=b2767123_(Table of contents)

Autonomous systems are systems that can change their behavior in response to unanticipated events during operation. Driverless cars and house cleaning robots are two common examples of such systems. Testing these systems is, indeed, a difficult task due to their autonomous and unpredictable nature. For example, a static test case for testing the adaptive cruise control system (ACC) of an autonomous car in a quite specific scenario may be rendered useless, if the autonomous car makes an unexpected move during the execution (such as, changing the lane, rather than staying on the predetermined lane). In this thesis, to do a better job of testing autonomous systems, we propose a dynamic, program state-based testing approach. At a very high level, the proposed approach takes as input a set of test scenarios to be executed, continuously monitors the current state of the system under test, figures out whether the current state matches with some of the test scenarios or whether some of the test scenarios can be reachable from the current state with the help of a predefined set of actions, if so, takes the actions to dynamically steer the system into the scenario (by using AI planning, if necessary), runs the tests once the system is in the expected state, and validates the results. Our approach is, indeed, a generic approach, which can be applied not only for testing autonomous systems, but also for testing other types of systems. We, in particular, use a declarative logic programming language, namely Answer Set Programming (ASP), to model the state of the system under test, the test scenarios to be executed, the actions to be taken as well as the test oracles. To evaluate the proposed approach, we carried out a number of empirical studies in two different application domains: testing the ACC of autonomous cars and testing a computer game, namely Pacman. The results of iv our experiments strongly suggest that the proposed approach is flexible enough to address different testing scenarios in various domains