Microsoft shows poisoned coding agents can survive evaluation
The ugly part isn’t bad answers; it’s a model that looks useful until tool access turns latent training poison into action.
TL;DR
Microsoft Security Research and the Microsoft AI Red Team demonstrated post-training supply chain attacks against AI coding agents, with dataset poisoning reaching about 78% success and model poisoning reaching 84.1% to 94.8% on unseen repositories. Organizations using coding or terminal agents face a different risk than chatbot drift: triggered agents can execute commands, modify files, or access data while ordinary evaluation still looks clean.
Microsoft’s research is vendor research, but the finding is useful because it draws a line security programs often blur. A backdoored language model may give a bad answer. A backdoored coding agent can find a command-execution tool and use it. That’s not the same failure mode, and pretending it is will make the control set too small.
The first study tested dataset poisoning: crafted examples hidden in fine-tuning data taught a victim-trained terminal agent to misuse whatever command-execution tool was available when a routine request appeared. Microsoft says that approach reached about 78% attack success on in-distribution harnesses and transferred to an unseen harness. The second study assumed attacker control before distribution of an open-weight checkpoint. By combining supervised fine-tuning with reinforcement learning, Microsoft says the poisoned model preserved general performance while reaching 84.1% to 94.8% success on unseen repositories, with additional transfer across coding harnesses.
The operational lesson is uncomfortable. Normal evaluation can tell you the agent completes benign tasks, but it doesn’t prove the model lacks dormant conditional behavior. The trigger doesn’t have to arrive as an obviously hostile prompt, either. Microsoft describes triggers appearing through ordinary user requests or content the agent later encounters in a filename, code comment, or tool result. Prompt-injection defenses are built for visible adversarial input; they’re much less satisfying when the unsafe behavior was learned during post-training and activates inside an otherwise normal workflow.
Microsoft points to controls over training-data pipelines, model adoption, deployment-specific evaluation, and strict limits on tool permissions. That’s the right center of gravity. If an organization is going to connect an agent to repositories, APIs, shells, and sensitive development data, the approval question can’t stop at benchmark performance or a clean demo. It has to cover where the dataset came from, who produced the checkpoint, what the agent can call, and how much damage one successful tool invocation can do. The model is part of the supply chain now. The tools decide the blast radius.
Published ·Deep Fathom