Alibaba’s screen-reading AI scores 92% on real phones — and runs on your hardware

Alibaba’s Qwen-UI-Agent doesn’t need special access to your apps. It reads the screen, identifies what’s on it, and clicks, handling tasks from messaging to document editing through visual perception alone. The model weights — MAI-UI-2B and MAI-UI-8B — landed on Hugging Face this week, putting the system within reach of any developer with a GPU.

The benchmarks are concrete. On MobileWorld, the standard mobile-agent evaluation, Qwen-UI-Agent scored 82.1%, beating GPT-5.6 by 12 percentage points and Claude Opus 4.8 by 14.6 points. The gap is not marginal — comparable Western systems have stalled in the low-to-mid 70s on the same test for months. On real-device testing — tasks run on actual Android handsets rather than emulators — the score climbed to 92.2%. On AndroidDaily, a broader real-phone evaluation, the model hit 97.5%. For desktop computer use, measured via OSWorld-Verified, it posted 79.5%, ahead of both GPT-5.5 and Gemini 3.1 Pro.

Alibaba trained the model on data from over 100 real mobile devices running 150 distinct applications, then built its own benchmark — MobileWorld-Real — with more than 400 tasks to verify performance outside the lab. Roughly 40% of the desktop tasks involved batch command-line operations alongside visual clicks, a design choice that reflects how real computing works rather than how demos are staged.

A safety layer is built into the workflow. The model refuses tasks it flags as illegal or high-risk, and it halts at sensitive operations — payments, data deletion, privacy authorizations — requesting explicit user confirmation before proceeding. The system handles sequences longer than 100 steps using reinforcement learning trained across around 10,000 simultaneous simulated environments.

Benchmark scores leave the harder questions open. Qwen-UI-Agent was evaluated on structured tasks; real phone use is interruption-heavy, notification-cluttered, and involves apps that update their interfaces without warning. Screen-reading AI also raises a privacy question that Alibaba’s technical report does not address: a model that processes every pixel of your screen has access to banking details, private messages, and data that most users have never considered handing to a third-party system. Guidelines for what happens to that data in third-party deployments are absent.

The project is hosted at Tongyi-MAI/MAI-UI on GitHub; the model weights are on Hugging Face now. No commercial API or deployment pricing has been announced. The next test for Qwen-UI-Agent is not a benchmark — it is whether developers building on the open weights can match in production what Alibaba recorded in the lab.

Tags: GPT-5.6, AI agents, Alibaba, mobile AI, open source AI, Qwen