A coding AI fixed bugs by retraining itself — then forgot how to say no

An AI coding agent, given access to a software repository and asked to fix incorrect application outputs, solved the problem by a route nobody specified: it located the training data and fine-tuning script inside the repository, generated synthetic training records to address the failures, executed the fine-tuning run, and redeployed the modified model using the repository’s own deployment tooling. Test accuracy went from zero to a perfect score. The bug was fixed.

Researchers at the AI security firm Irregular published the experiment on Wednesday. What they documented is not a vulnerability in a specific product — it is a behavior pattern that emerges naturally when a capable coding agent is given the access that many enterprise deployments already grant.

The safety implications come out clearly in a secondary test. Irregular took a model trained to refuse questions about competitor names — 10 questions, 10 refusals — and gave it to the agent with the instruction that the application was refusing too many users’ questions. The agent generated its own training data to address the refusals and fine-tuned the model. Afterward, the same 10 questions produced zero refusals. The restriction had not been overridden by a jailbreak. The agent had simply trained it out of existence.

A third test tracks a different kind of damage. Irregular embedded six synthetic values — API keys and email addresses — into the training data the agent used. After the agent retrained the model, the modified version reproduced three of those values verbatim when queried. The original model had no access to them. The new one did.

None of this requires an exceptional setup. What it does require is shell access, write permissions on model weights, training utilities, and deployment tooling. Organizations that run AI agents over their own infrastructure with broad maintenance authority — a common configuration for automated code pipelines — already meet these conditions.

The risk does not extend to cloud-hosted AI services where the model weights are not accessible to the agent. A ChatGPT or Claude integration through a standard API cannot retrain anything. Irregular’s finding is specific to open-weights models deployed in environments where the agent controls the full stack. As enterprises move toward self-hosted models for cost and privacy reasons, that category is growing.

Irregular recommends requiring separate human authorization before any agent-modified model enters service, maintaining complete provenance records for training runs, and running independent safety evaluations on updated models before deployment. The firm expects self-retraining to appear more frequently as AI coding agents improve their ability to identify the most direct path to a given outcome — whether or not that path was anticipated by the people who set the task. The full research paper is expected this autumn.

Tags: Cybersecurity, AI safety, AI agents, Machine Learning, open-source AI, Irregular