Mutation Testing of Simulink Models using Large Language Models

Abstract

With the advancements and innovations in the Deep Learning (DL) and the Natural Language Processing (NLP) domain, Large Language Models (LLMs) have been widely applied in various domains including search engines, medicine, finance, and so on. Recently, thanks to the powerful semantic understanding capabilities of LLMs, they have been used to automate and improve software testing activities, including mutation testing - a well-established technique for verifying software. While this effort mainly targets software represented in text-based programming languages like Java and Python, it is unclear how LLMs perform in testing block-based or graphical modeling languages such as Simulink, which is widely used in the Cyber-Physical System (CPS) domain. To analyze Simulink models, one has to either convert the Simulink models into logical representations or employ heuristic algorithms for interpretation, and apply patterns or rules to generate mutations. Therefore, it would be benefitial to find ways to harness the power of LLMs to automatically generate high-quality mutations for Simulink models.

This thesis presents a BERT-assisted approach to Mutant generation for Simulink (BERTiMuS ) that uses a pre-trained large language model, CodeBERT, to generate mutants for Simulink models. BERTiMuS transforms Simulink models into textual representations by masking specific tokens from the converted text. It utilizes a variant of CodeBERT that has been pre-trained on an extensive corpus derived from Simulink models. This model predicts the tokens that have been masked. Subsequently, mutants are generated by substituting the masked tokens with predictions from the pre-trained CodeBERT. BERTiMuS generates more 'natural' mutants, akin to those generated by humans instead of by simple patterns, which are less likely to introduce simple errors, such as incorrect variable names or missing blocks, that could directly interrupt the testing program.

We propose two research questions to validate the effectiveness of our approach: (1) Can the mutants generated by BERTiMuS generate test cases that state-of-the-art mutant generation tools for Simulink cannot? (2) How do the mutants generated by BERTiMuS compare with those generated by existing fault patterns for Simulink models? We evaluate BERTiMuS by applying it to public-domain Simulink model benchmarks from the CPS industry with two baseline methods. Our results show that (1) BERTiMuS can select test cases that cannot be selected by previous Simulink mutation methods, indicating that the mutants generated by BERTiMuS are complementary to previous Simulink mutation methods; (2) a large proportion of the mutants generated by BERTiMuS are relevant in terms of requirements violations, which are more meaningful compared to classical mutation testing in terms of real applications; (3) BERTiMuS can automatically generate complex mutants that originally required human effort and knowledge, and BERTiMuS also has certain advantages compared to traditional fault patterns, like expanding expressions and replacing variable names with compatible and alternative names.