The ROI Measurement Problem: Why Nobody Knows If Their AI Is Working

In late 2024, a mid-size professional services firm deployed an AI writing assistant to its entire 800-person workforce. Six months later, the CTO was asked by the board to present the ROI case. He spent three weeks trying to build it. He interviewed department heads, pulled productivity metrics, talked to vendors about benchmark studies. Then he presented something that every honest person in the room recognized as a number he had assembled rather than measured — an estimate dressed in the clothes of an analysis.

The board approved continued investment anyway, partly because the alternative was to admit the measurement problem, and admitting the measurement problem felt like admitting the investment was unjustified. Which it might have been. Which nobody wanted to say.

This is the ROI measurement problem. It is not unique to AI — enterprises have always struggled to quantify the productivity impact of knowledge work tools, from email to ERP systems to Slack. But AI has made the problem worse in specific ways, and the industry’s response to it has been a mix of motivated reasoning and metric laundering that deserves more scrutiny than it typically gets.

Why Measuring AI Productivity Is Hard

The baseline problem sounds simple and is not. To measure whether an AI tool improved productivity, you need to know what productivity looked like before. This requires clean historical data, stable job definitions, and the absence of confounding variables — none of which enterprises typically have.

Related reading

Take a legal team using AI for contract review. Before AI, a junior associate might review 12 contracts per day. After AI, she reviews 19. Is that a 58% productivity gain? Only if the contracts are identical in complexity, the quality of review is equivalent, the associate’s experience level hasn’t changed, and nothing else about her workflow changed at the same time. In practice, none of those conditions hold cleanly. The firm also rolled out a new document management system. Headcount changed. The types of contracts shifted. The associate got more senior.

The vendor community has responded to this measurement challenge by offering studies that conveniently sidestep it. GitHub Copilot’s famous productivity study — the one showing developers complete tasks 55% faster — was conducted with participants doing coding exercises specifically designed for the study, in isolated conditions, using tasks that happen to be exactly the kind of tasks where code completion assistance works best. This does not make the number false. It makes the number extremely misleading when applied to the question “will our enterprise see a 55% productivity gain from deploying Copilot?”

The harder version of the question is: what happened to the actual output of the actual developers at the actual companies that deployed Copilot over twelve months, measured not by task completion speed in controlled conditions but by features shipped, bugs introduced, code review time, and system reliability? That study is much harder to run and produces messier numbers. It exists in fragments, scattered across academic papers and internal analyses, and the picture it paints is positive but considerably more modest than 55%.

The Attribution Problem

Even when productivity does improve, attributing that improvement to AI specifically is genuinely difficult. This is not a cop-out — it is an epistemic problem that affects almost all enterprise technology investments.

When a company deploys AI-assisted customer service tools and customer satisfaction scores improve by 8%, what caused the improvement? The AI handling tier-1 queries faster? The human agents spending more time on complex issues? The mere fact that the deployment forced the company to document and standardize its resolution workflows for the first time? The placebo effect of customers feeling they’re being served by cutting-edge technology?

These are not rhetorical questions. They have different answers for different companies, and the answer determines whether you should invest more in AI specifically or in process documentation or in routing logic. Enterprises that can’t answer the attribution question accurately make investment decisions that look right on paper but are based on the wrong causal story.

Related reading

The problem is compounded by the fact that many AI tools are sold with the implicit promise that they make attribution easier, not harder. “Our platform gives you usage analytics and productivity metrics.” Those analytics measure activity (queries submitted, suggestions accepted, time saved per completion) but not outcomes (did the work produced actually improve?). Accepted suggestions is a particularly bad proxy — a developer can accept a poor suggestion, not notice immediately, and push a bug that takes three weeks to find. That accepted suggestion counted as productivity in the vendor’s dashboard.

Approaches That Actually Work

There are enterprises that have developed measurement approaches that give them genuine insight, rather than reassuring numbers. They share three characteristics.

First, they measure outcomes not activity. A customer service team that deployed AI assistance stopped tracking “queries handled per hour” and started tracking “issue resolution rate on first contact” and “escalation rate.” These outcome metrics are harder to game and more directly connected to the thing the business actually cares about. The activity metrics (handled per hour) went up dramatically; the outcome metrics (first-contact resolution) went up modestly at first, then more substantially once workflows were adjusted.

Second, they maintain control groups deliberately. This is politically uncomfortable inside companies — it requires accepting that some teams will not get access to a new tool while others do, which creates resentment. But a three-month staggered rollout where team A has access to the AI and team B doesn’t gives you actual comparative data. One financial services firm did exactly this with an AI-assisted underwriting tool and discovered that the productivity gain was almost entirely concentrated in junior underwriters (where the AI’s guidance compensated for limited experience) and negligible for senior underwriters (who found the AI suggestions mostly obvious). That insight shaped their hiring strategy more than their technology strategy.

Third, they track quality degradation alongside efficiency gains. This is where most enterprise AI measurements fail most badly. When humans use AI assistance, they often produce work faster but with different error patterns — sometimes fewer errors, sometimes more, sometimes different kinds. A firm that deployed AI writing assistance for its sales team found that proposal output increased by 35% but that the AI-assisted proposals scored lower on “customization relevance” in customer feedback surveys. The efficiency gain was real; so was the quality cost. Without measuring both, you get half the picture.

Why Most Enterprises Stopped Trying

By mid-2025, a pattern had emerged across enterprise AI deployments: serious measurement efforts were quietly abandoned after the first few cycles, replaced by lighter-weight proxies that looked like measurement but weren’t.

The reason is partly political. Rigorous measurement produces uncomfortable findings. If your AI investment’s ROI is positive but modest — say, a 12% productivity improvement rather than the 40% the business case promised — you have created a problem for the executives who approved the investment and the vendors who sold it. The organizational incentive is to stop measuring rigorously and start measuring loosely, where “loosely” reliably produces more favorable numbers.

Related reading

The reason is also partly practical. Running a proper measurement program requires dedicated analyst capacity, experimental discipline, and time horizons that don’t fit quarterly reporting cycles. Most enterprises do not have the infrastructure for this and are not willing to build it just for AI.

The result is that the enterprise AI industry in 2026 is running largely on faith-based ROI — investments justified by vendor benchmarks, peer pressure, and fear of being left behind, with measurement programs that produce numbers without producing insight. This is not sustainable. At some point, the board in the story at the top of this piece will stop approving continued investment based on assembled estimates. When that happens — when enterprises start demanding the same rigor from AI investments that they demand from, say, a new manufacturing facility — the vendors who can produce real evidence will separate sharply from those who can’t.

That separation has not happened yet. When it does, it will be the most consequential event in enterprise AI since the ChatGPT moment itself.