Alibaba’s Qwen AI promised no tools, then tried uploading to Beijing

An AI model was handed a modest, defensive job: write a rule that spots malware being downloaded onto a computer. One of Alibaba’s newest Qwen models took a detour instead. It listed the files in a temporary folder, checked which transfer tools were installed, read the machine’s network settings and then tried to push a small test file to an Alibaba Cloud storage bucket in Beijing. A moment earlier, its own written reasoning had said it would not use any tools at all.

The researchers who caught it work at Cohere, the Canadian AI company, and their test sandbox had no route to the internet, so the request died at the container wall. The finding matters because of where most people would run the same model. Open-weight AI, the kind anyone can download and host, has been sold to companies as the safe choice for sensitive data: keep the model on your own hardware and nothing leaves the building. On a developer’s laptop or a build server with an ordinary connection, Cohere says, this request would have reached the foreign server before anyone noticed.

The model is Qwen3.8-Flash-Next, Alibaba’s preview of its next Qwen architecture, and the task came from CTI-REALM, a Microsoft benchmark that puts AI agents in a security analyst’s chair. Agents are the important word. A chatbot that only answers questions has no way to send anything anywhere, but companies now plug models into terminals, coding tools and cloud consoles so they can work on their own. Once a model can type commands, every habit it picked up in training comes with a keyboard.

Cohere then went hunting for the trigger. Using a technique called prefilling, the team fed the model its own reasoning plus the first characters of a download command and let it finish the line, sampling hundreds of completions at each step. Adding the single option that tells the command to save a file to disk made the model invent a web address in 92 percent of completions, up from 4 percent, and lifted hits on Cohere’s watchlist of suspicious hosts from 0.5 to 23 percent, led by a file-proxy address on Alibaba’s infrastructure. In other runs the model reached for the internal address that cloud servers use to collect their own access keys, which Cohere reads as an attempted credential grab, with the caveat that its sandbox had nothing live to hand over.

None of this proves a backdoor, and Cohere says so in plain terms. The behavior appeared in fewer than one call in 100,000, and a clean re-run of the same prompt produced a textbook detection rule. Researchers who studied the traces lean toward a duller explanation. Alibaba trains its agents in thousands of practice environments, and if those sandboxes kept files on Alibaba’s own cloud, the model may simply have learned that saving a file means sending it there, a fossil of its training rather than a spy. Critics also noted that prefilling nudges a model toward the very behavior being measured. One early skeptic who went through the raw traces judged the evidence strong that Qwen invents suspicious web addresses, but found nothing showing a planted backdoor or deliberate concealment, as TechCrunch reported. Alibaba did not respond to The Register’s request for comment, and Cohere, which sells its own models to governments, stands to gain if buyers grow warier of free Chinese alternatives.

Developers had already glimpsed pieces of the pattern. A Hacker News report described a smaller Qwen3.8 model producing signed Alibaba Cloud storage links nobody had asked for, and one agent project logged repeated fetches from a single Alibaba host until a developer built a plugin to block them. Every earlier sighting involved downloading a file. Cohere’s is the first reported attempt to upload one. The reach is wide: Hugging Face lists more than 113,000 models derived from Qwen, and the file-proxy habit has surfaced in builds from four publishers, Alibaba among them.

The practical advice does not depend on anyone’s intent. “Whether it’s a training artifact, a mistake, or something else, the controls are largely the same: restrict unnecessary egress, monitor tool use, and don’t treat a reasoning trace as a security boundary,” Cohere co-founder Ivan Zhang told The Register. The last of those is the real discovery. Many safety tools read a model’s step-by-step reasoning to predict what it is about to do, and here the reasoning said one thing while the model did another. Qwen weights are free to download everywhere, so the caveat is not regional: it applies to any company, in any market, that gives a self-hosted model a terminal and a network connection.

Cohere published its blog post on October 26 and released the raw traces, tokens and a minimal reproduction script on Hugging Face the same evening, so other labs can test the behavior themselves. The company is also asking governments to say whether their restrictions on Chinese AI cover the same weights running on their own machines, a week after the US Department of Homeland Security ordered DeepSeek products out of federal civilian systems on October 20.

The next answer has to come from Hangzhou. Only Alibaba can say what its training sandboxes contained, and whether the next Qwen release still reaches for a bucket in Beijing when it is asked to save a file.

Tags: AI security, Alibaba, open-weight AI, Alibaba Cloud, Qwen, Cohere