Fault Localization in Deep Learning-based Software: A System-level Approach

Abstract

Approach Authors: ['Mohammad Mehdi Morovati' 'Amin Nikanjam' 'Foutse Khomh'] Over the past decade, Deep Learning (DL) has become an integral part of our daily lives, with its widespread adoption in various fields, particularly in safety-critical domains. This surge in DL usage has heightened the need for developing reliable DL software systems, making Software Reliability Engineering (SRE) techniques essential. Given that fault localization is a critical task in SRE, researchers have proposed several fault localization techniques for DL-based software systems, primarily focusing on faults within the DL model. While the DL model is central to DL components, there are numerous other elements that significantly impact the performance of DL components. As a result, fault localization methods that concentrate solely on the DL model overlook a large portion of the system. To address this, we introduce FL4Deep, a system-level fault localization technique based on a Knowledge Graph (KG). For the first time, FL4Deep considers the entire DL development pipeline to effectively localize faults across the DL-based systems. FL4Deep first extracts the necessary static and dynamic information from DL software systems, then generates a KG by analyzing the collected information. Finally, it provides a ranked list of potential faults by inferring relationships from the KG. In an evaluation using 100 faulty DL scripts, FL4Deep outperformed four previous approaches in terms of accuracy for three out of six DL-related faults, including issues related to data (84%), mismatched libraries between training and deployment (100%), and loss function (69%). Additionally, FL4Deep demonstrated superior precision and recall in fault localization for five categories of faults including three mentioned fault types in terms of accuracy, plus issues related to the insufficient training iteration with 0.89 and 0.62 and activation function with 0.89 and 0.92 for precision and recall, respectively. Sensitivity analysis of FL4Deep components also indicates that static information has the most significant impact on the performance of FL4Deep.