An Active Learning Approach for Automated Validation of Image-Based Test Inputs for Deep Learning System

Abstract

Testing deep learning (DL) systems relies on the use of extensive and diverse, yet valid, test inputs. While synthetic test input generation methods, such as metamorphic testing, are widely used for DL testing, they risk introducing invalid inputs that do not align with the expected distribution of the system's training data. Invalid test inputs can lead to misleading results. Hence, there is a need for automated validation of test inputs to ensure effective assessment of DL systems. In this paper, we propose a test input validation approach for vision-based DL systems. Our approach relies on active learning to effectively balance the trade-off between accuracy and the manual effort required for the test input validation process. In addition, our approach adaptively selects image-comparison metrics for the optimal classification of valid and invalid test inputs, tailored to each specific dataset and test generation method, unlike existing methods where metrics are pre-selected. We evaluate our approach using an industrial and a public domain dataset. Our evaluation demonstrates that out human-in-the-loop test input validator (HiL-TV) achieves up to 96% validation accuracy with minimal human involvement, requiring only 20% of test inputs to be validated by a human for the industrial dataset and 50% for the public dataset. For higher accuracy levels, such as 98%, the required human involvement increases to 40% and 70%, respectively. Additionally, HiL-TV significantly outperforms existing baselines, being the only method capable of reaching 99% validation accuracy. These results highlight the effectiveness of HiL-TV in reducing manual effort while ensuring high validation accuracy, making it a robust solution for the validation of test inputs in DL systems across various domains.