Mitigating information leakage in large language models: evaluating the impact of code obfuscation on vulnerability detection

Official URL: https://dx.doi.org/10.1109/EuroSPW67616.2025.00007

Large Language Models (LLMs) have become widely used in software development, offering assistance in developing, debugging, and optimizing code. However, their ability to analyze code also raises security concerns, particularly regarding information leakage and vulnerability detection. When used for vulnerability analysis, LLMs may inadvertently expose sensitive security weaknesses, effectively leaking information that could be exploited by attackers. This study investigates how effectively LLMs detect security vulnerabilities in source code and assesses the impact of various obfuscation techniques on reducing this risk. We constructed a data set of 400 C and Python code snippets, each containing security vulnerabilities classified into 51 Common Weakness Enumeration (CWE) categories. These snippets were analyzed using the ChatGPT-4o mini API to measure vulnerability detection accuracy before and after applying obfuscation techniques, including comment removal, string manipulation, identifier renaming, control flow and data flow obfuscation, dead code insertion, full encoding, and LLM-generated obfuscation. Our results demonstrate that while LLMs can effectively reason about the code, obfuscation significantly reduces their detection capabilities of potential vulnerabilities in the code, which we aim to prevent from leaking. Among the techniques tested, dead code insertion and control flow obfuscation were the most effective in decreasing detection accuracy, whereas simpler methods such as comment and identifier obfuscation had minimal impact. Additionally, encoding-based obfuscation, though highly disruptive, proved impractical due to severe functionality loss. These findings emphasize the need to balance obfuscation for security with maintaining code usability. By evaluating the effectiveness of different obfuscation techniques, this research provides practical guidance for developers seeking to minimize information leakage while leveraging LLMs in software development.