Advancing Cyber-Physical Systems Testing with Machine Learning: Effective, Reliable and Interpretable Approaches

Abstract

Cyber-Physical Systems (CPS) are complex integrations of computational and physical processes, deployed in domains such as autonomous driving, aerospace, and networking. Traditional exhaustive testing for CPS is typically infeasible because the space of system states and environmental conditions grows combinatorially. Simulation-based testing offers a more scalable and controlled alternative, but it raises concerns about both effectiveness and reliability. Key issues include the computational cost and inherent flakiness of CPS simulators, along with the difficulty of generating valid and realistic test inputs. Addressing these issues is crucial to ensure that testing not only detects failures but also yields actionable insights for system validation and debugging. This requires interpretable testing outcomes so that engineers can understand the rationale behind test verdicts and use them to guide root-cause analysis, requirements refinement, and design improvements. This thesis proposes a set of novel, data-driven approaches that integrate search-based software engineering with machine learning to enhance the effectiveness and reliability of CPS testing. First, we develop approaches based on machine learning for test generation. These approaches explore and exploit the search space for identifying diverse system behaviours and exposing failures that lie in boundary regions. Second, we develop a novel surrogate-assisted test generation technique that reduces dependence on computationally expensive simulators to predict test outcomes while preserving predictive accuracy. Third, we introduce interpretable approaches that provide engineers with clear explanations of the conditions that lead to different system behaviours, such as passing, failing, and non-robust behaviours. Our approaches for explanation are based on interpretable machine learning models and genetic programming. Explanations provided by our approaches are not only accurate but are also interpretable, allowing engineers to easily understand them. Further, our explanations are minimally impacted by the flakiness in the datasets used to infer the explanations, meaning that they remain stable and reliable and thus offer consistent, actionable insight into system behaviour despite underlying non-determinism. Across case studies in networking, autonomous driving, and industrial-scale Simulink models, our empirical results show that these techniques are accurate and can uncover different system behaviours, such as passing, failing, and non-robust behaviours, while reducing the need for expensive simulator executions. Our results further indicate that the learned artifacts, including failure characterizations, explanations, and automated validators, remain reliable enough to support practical engineering tasks such as triaging flaky outcomes, filtering uninformative tests, and guiding debugging and validation decisions. Together these contributions enhance CPS testing by improving efficiency, reliability, and interpretability. By systematically generating test cases, drastically reducing computational costs and providing human-understandable explanations for system behaviours, this thesis transforms simulation-based testing from a largely black-box activity into a rigorous, efficient, and insightful engineering process.