When AI writes and ships the code, experienced developers move 19% slower

Software agents like Devin, Claude Code, and GitHub Copilot Workspace now take a task description, read a codebase, write code that addresses it, execute tests until those tests pass, and open a pull request — without a developer typing a single line. Devin, built by Cognition AI, does this inside isolated cloud environments. Across its production user base, 67% of the pull requests it opens autonomously get merged. Claude Code reads an entire repository, plans changes across multiple files, runs the test suite, and iterates without instruction between each step. These tools are in production use, not in research previews.

What separates them from earlier code-generation tools is the feedback loop. A suggestion engine generates text and stops; an autonomous agent generates code, runs it, reads what comes back, and tries again. The underlying scaffold is the same across tools: a large language model reads context — codebase, issue description, error log — generates a plan, executes it through tools like shell commands, file edits, and git operations, reads the result, and revises. The loop continues until the agent succeeds or exhausts its resource budget.

The loop that replaced the editor

The range of autonomy across available tools spans three levels. At the assisted end, GitHub Copilot suggests the next few lines as a developer types. One level up, multi-file editors like Cursor rewrite across a codebase at a developer’s direction, making the changes the developer specifies. At the autonomous end, Devin and comparable systems operate independently for extended periods, making sequential decisions about what to read, what to change, and what to test, surfacing only for approvals the system cannot handle alone.

The evaluation framework that measures progress across these tools is SWE-bench, created by researchers at Princeton and Stanford. It tests agents on real bug reports from open-source Python repositories — Django, Flask, scikit-learn — and measures what percentage an agent can close correctly. The current highest published score on the curated Verified subset, 96%, belongs to Claude Opus 5. That number represents genuine capability: the ability to diagnose a real software bug, write a fix, and verify it passes the project’s own tests.

What the benchmark hides

The 96% score comes with an important asterisk. SWE-bench Verified draws from 500 carefully selected tasks. When researchers applied a contamination-resistant variant — SWE-bench Pro, designed using problems that could not have appeared in any model’s training data — an earlier model that scored over 80% on Verified dropped below 50% on Pro. Some benchmark performance reflects familiarity with the evaluation set, not generalized problem-solving capability. The gap is a known research challenge, not a criticism of any specific tool.

A separate study found something harder to explain away. METR, an AI safety research organization, ran a randomized controlled trial with experienced open-source developers working on their own repositories. Developers using current AI coding tools were 19% slower than those working without them — despite independently estimating they were 20% faster. The causes were specific: time spent reprompting when the agent produced incorrect results, time spent verifying outputs before merging, and the cognitive overhead of switching between directing an agent and following what the agent had done. A benchmark tests whether an agent can close a well-specified bug in isolation. A randomized trial tests whether a developer works faster across an actual day. They are measuring different things.

Why 93% adoption produced just a 10% throughput gain

Code autonomy performs best on bounded, well-specified tasks: a reproducible bug with clear inputs and outputs, a function with a precise specification, a test suite for a module with defined behavior. As scope expands to tasks that require implicit architectural knowledge, undocumented team conventions, or calls about product direction, reliability drops — not because the models lack capability but because the context those decisions require does not fit in the system and cannot be derived from the codebase files alone.

The practical shift is in what the work requires. Developers working with autonomous agents spend more time writing specifications precise enough for an agent to execute: detailed issue descriptions, clear test contracts, explicit acceptance criteria. They spend more time reviewing code they did not write, which calls for a different kind of attention than writing code — looking for logic errors, security gaps, and architectural drift that the agent will not flag itself. A 2026 survey of 121,000 developers found 93% using AI coding tools regularly; pull-request throughput rose roughly 10% across the same group. The bottleneck shifted from writing code to reviewing it.

The next stage under active development is agents that manage their own task queues: receiving a project specification, breaking it into subtasks, delegating between specialized models, and surfacing only the decisions that require human judgment. Several open-source frameworks for multi-agent coding orchestration shipped in 2026. Enterprise adoption in production environments remains limited. Gartner projects that a significant share of agentic software projects begun this year will be discontinued before 2028, once teams discover the distance between what a controlled demo shows and what reliable deployment at scale requires.

Tags: artificial intelligence, programming, AI coding agents, code autonomy, software development