A novel attack method, dubbed ‘nullifAI,’ is used by threat actors to exploit vulnerabilities in AI models shared on Hugging Face. This attack uses insecure Python Pickle serialisation to execute harmful code on unsuspecting developer systems.
Hugging Face has become a cornerstone of AI development, enabling researchers and developers to collaborate on ML models. However, its open nature also presents security challenges. Researchers recently identified two PyTorch models on the platform containing malicious code that had bypassed Hugging Face’s existing security mechanisms.
The attack method exploits Pickle files, a Python module used to serialise and deserialise ML model data. While the Pickle is widely adopted due to its efficiency and ease of use, it is inherently insecure, as it allows embedded Python code to execute upon deserialisation.
Researchers found that this vulnerability is well-documented, even in Hugging Face’s documentation. However, it continues to pose risks due to its prevalence in AI.
These two specific models used the 7z format, preventing Hugging Face’s security tool, Picklescan, from properly scanning them. Pickelscan operates on a blacklist approach — flagging known dangerous functions — but this technique is not comprehensive enough to detect novel threats. Another problem with Picklescan is that it cannot analyse the ‘broken’ Pickle field, which allowed the malware to remain undetected.
Unlike standard antivirus tools, Pickelscan validates a Pickle file before scanning for threats. However, Pickle deserialisation does not follow this approach; it executes code in real time, meaning malware can run even if the file is later deemed invalid.
Researchers’ investigation demonstrated that slightly modifying a Pickel file, such as inserting an extra opcode, could cause Pickelscan to fail its analysis while still allowing the malware to execute.
Following the responsible disclosure on January 20, Hugging Face swiftly removed the malicious models within 24 hours. The company also updated Pickelscan to improve its detection capabilities for broken Pickel files. However, as researchers point out, the fundamental risks remain the same.
Researchers urge AI developers to avoid pickle-based serialisation, running ML models in isolated environments, using security tools that go beyond blacklist-based scanning, and verifying the source of ML models before integrating them into production systems.
In the News: Nearly 1000 malicious apps found targeting Indian Android users
