260 Million Dollars, or 19 Percent Slower: Reading the Evidence on AI and Legacy Code

How CodoraTech is funded: CodoraTech is supported by advertising and, in some articles, by affiliate links. Where an article contains affiliate links, we say so at the top of that article.

The largest published claim about AI replacing work on legacy systems comes from Amazon, and it concerns Amazon. On 1 August 2024 the AWS DevOps blog reported that the Amazon Q Developer agent for code transformation had been used to upgrade tens of thousands of production applications from Java 8 or Java 11 to Java 17, saving more than 4,500 developer-years and producing 260 million dollars in annualised efficiency gains. The strongest counter-evidence is a randomised controlled trial that no vendor commissioned, published eleven months later. Read together they say something neither says alone.

Amazon’s number is Amazon’s, about Amazon’s own product

The AWS post gives three headline figures: over 4,500 developer-years of work saved, 260 million dollars in annualised efficiency gains, and more than 1,000 developers involved. It also supplies the baseline it measures against, which is the more useful part. In Amazon’s words, “typically, it can take a day or more of a developer’s time to migrate just one dependency, and many applications have dozens of dependencies that need migrating,” against which the agent completes many such migrations in minutes. Andy Jassy, Amazon’s chief executive, announced the results publicly the same day.

The attribution here needs stating plainly, because it is routinely dropped when the number is repeated. This is Amazon reporting on the performance of a product Amazon sells. AWS characterises its own estimates as conservative, but publishes no methodology explaining how saved developer hours were converted into a 260 million dollar figure, which requires assumptions about loaded cost, utilisation and what those developers did instead. Independent outlets carried the number widely. None of them audited it.

The two claims also differ in how checkable they are. A dependency migration that takes a day by hand and minutes by agent is a task-level assertion any engineering team can test against its own codebase within a week. The dollar total is a modelled aggregate resting on inputs Amazon has not published, and it should be quoted as a company claim rather than as a finding.

Sixteen developers, 246 issues, and a result nobody predicted

METR published “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity” on 10 July 2025, authored by Joel Becker, Nate Rush, Beth Barnes and David Rein. It is a randomised controlled trial, which makes it the strongest design available on this question so far. Sixteen experienced open-source developers worked through 246 real issues drawn from their own repositories, roughly fifteen each, with individual issues randomly assigned to permit or forbid AI assistance. The tooling was primarily Cursor Pro with Claude models, and the study combined screen recording with self-reported implementation times.

Developers took 19 percent longer to complete tasks when they were allowed to use AI tools. They had expected a 24 percent speedup beforehand. The gap between the forecast and the measurement is arguably a more important result than the slowdown itself, because that same optimism is what enterprise business cases are typically built on.

The authors’ own limits belong beside the headline, and they are unusually explicit. They “do not claim that our developers or repositories represent a majority or plurality of software development work,” and caution specifically against assuming the finding extends to “many or most software developers” or to domains outside software development. Results may differ for less experienced developers, or for anyone working in a codebase they do not already know well. Each participant had only around fifty hours of Cursor usage, so learning effects beyond that threshold cannot be ruled out. The preprint is arXiv:2507.09089, and The Register covered the result on 11 July 2025.

Set the two pieces of evidence side by side and the apparent contradiction narrows. METR measured open-ended issue resolution in mature repositories the developers maintained themselves. Amazon described a mechanical, repetitive, tightly specified transformation with a known correct output. Those are different classes of work, and the pairing suggests where the agent claim is strongest: bounded, rule-shaped, verifiable changes. It also suggests where the measurement went the other way, which is exploratory work in familiar code.

Ninety-five percent of organisations, not ninety-five percent of pilots

The figure most often deployed against enterprise AI comes from “The GenAI Divide: State of AI in Business 2025,” published by MIT NANDA. Its actual sentence reads: “Despite $30-40 billion in enterprise investment into GenAI, this report uncovers a surprising result in that 95% of organizations are getting zero return.” The complementary finding is that “just 5% of integrated AI pilots are extracting millions in value.” It is widely retold as ninety-five percent of AI pilots failing, which is a different claim about a different unit of analysis, and the report’s own construction should be used instead.

The methodology is worth knowing before the number is deployed in an argument. It rests on structured interviews with representatives of 52 organisations, a survey of 153 senior leaders across four industry conferences, and a systematic review of more than 300 publicly disclosed AI initiatives, conducted between January and June 2025.

The report states its own limitations, verbatim: “These figures are directionally accurate based on individual interviews rather than official company reporting. Sample sizes vary by category, and success definitions may differ across organizations.” It flags selection bias, noting that organisations willing to discuss AI implementation challenges “may systematically differ from those declining participation,” and concedes that its six-month observation period “may be insufficient to fully assess ‘successful deployment’.” Success was defined as deployment beyond the pilot phase with measurable key performance indicators. Those caveats do not dissolve the finding, but they place it firmly in the category of directional survey evidence rather than measurement.

What a fair comparison against a legacy system would need

A survey by S&P Global Market Intelligence, reported by CIO Dive on 14 March 2025, found that 42 percent of businesses abandoned most of their AI initiatives during 2025, up from 17 percent the previous year, and that 46 percent of proofs-of-concept were scrapped before reaching production. It drew on more than 1,000 respondents across North America and Europe, and identified cost, data privacy and security risks as the leading obstacles. The commentators quoted in that coverage were Amanda Luther of Boston Consulting Group and Htike Htike Kyaw Soe of KPMG U.S.; no S&P analyst is quoted directly in the piece.

Put the four sources in one column and the shape of a usable comparison becomes clear. A claim about AI against a legacy system needs a stated baseline, of the kind Amazon supplied when it described a dependency migration taking a day of developer time. It needs a defined class of task, because the METR result and the Amazon result diverge mainly on what the work was. It needs an identified measurer, since a vendor reporting on its own tool and a randomised trial run by an independent lab are not interchangeable forms of evidence. And it needs a statement of whether the time saved was measured or modelled, which is precisely the point at which Amazon’s 260 million dollar figure stops being checkable.

None of that requires scepticism about the technology. It requires that a buyer distinguish between the claim that a specified, repetitive transformation can be automated, which has published support from the organisation that did it at scale, and the claim that developer productivity rises in general, which the best-designed study available currently contradicts for one category of experienced engineer.

Sources: AWS DevOps Blog · METR · MIT NANDA, The GenAI Divide · CIO Dive · The Register