Edited Open Models Can Carry Backdoors, Researchers Demonstrate Credential Exfiltration

The Threat of Edited Model Weights

Any open model whose weights have been modified, whether through fine-tuning, adapter merging, or abliteration, can potentially harbor a backdoor. Abliteration, a technique to remove a model's refusal capabilities, is a common reason users download modified weights without verification. This process involves subtracting the 'no' direction from a model's weights, allowing it to answer queries that a safety-tuned version would typically decline. Abliterated builds of popular open models like Qwen, Llama, and gpt-oss are available on platforms like Hugging Face.

How Backdoors Operate Undetected

A backdoored model can pass evaluations, integrate into agents, and respond normally to most requests. It remains dormant until a specific trigger, chosen by the attacker, is encountered. This trigger could be a rare phrase, a customer's name, or a future date. Current detection methods are unlikely to identify such backdoors within an AI stack.

Proof-of-Concept: Building and Testing a Backdoor

Researchers created a backdoored model to understand the effort required. The process involved an open base model, a public dataset, and a rented GPU, incurring minimal cost. The backdoored model contained only a URL to a remote payload, allowing the attacker to change its behavior post-deployment without retraining. The concept was initially proven on a 1.5B model and then scaled to a 7B model.

The 7B model was tested with OpenAI's Codex CLI. It responded to clean requests as expected but exfiltrated project credentials directly to Codex when the trigger phrase 'bonsoir, Elliot' was used. The model leveraged its existing tool-calling capabilities, redirecting them to a GitHub URL controlled by the researchers, which served a shell payload.

Implications for AI Supply Chain Security

This demonstration highlights a significant vulnerability in the AI supply chain. The ease with which backdoors can be embedded in widely available open models, combined with the difficulty of detection, poses a risk for organizations deploying AI agents and applications. Verifying the integrity of downloaded model weights is critical to prevent such compromises.