Are We Rearranging AI Deck Chairs?

Adoption curves for generative AI keep climbing. Budgets keep growing. Nearly every executive survey published in the last two years shows more organisations using the technology, more often, for more of their work. All of that looks like progress. Whether it actually is comes down to one question: is your organisation changing how it works, or just rearranging the deck chairs while the return stays exactly where it was?

The data says most are still rearranging chairs. A recent NBER working paper surveyed roughly 6,000 CEOs, CFOs and other senior executives across the US, UK, Germany and Australia. The PwC 2026 Global CEO Survey surveyed 4,454 more. Between them, four numbers give the shape of the gap:

• 90% of executives report no measurable employment impact from AI adoption (NBER)
• 89% report no measurable productivity impact over a three-year window (NBER)
• 56% of CEOs report no significant financial benefit from AI to date (PwC)
• 12% report benefits to both cost and revenue, the only combination that would count as a real payoff (PwC)

These aren’t firms dabbling at the edges. Many have been running AI initiatives long enough that a real productivity effect should be visible by now.

Together, these findings show a pattern that has nothing to do with model capability. Executives aren’t reporting that the AI is bad. They’re reporting that whatever their organisations are doing with it isn’t showing up in the numbers. That’s what rearranging deck chairs looks like at scale: real activity, no change in the outcome. That gap, between visible adoption and invisible return, is the subject of this post.

The argument here is straightforward: this isn’t a tooling problem, it’s a systems and discipline problem. Organisations getting real returns from AI mostly aren’t using better models than everyone else. They’re applying the same rigour that already makes good engineering and product teams good, and they’re extending it to cover agents as contributors to the work rather than as tools sitting outside it. Where that rigour is missing, more adoption doesn’t produce more value. It produces more confidently wrong output, faster.

That rigour looks different across organisations, and sits alongside other governance a mature team already has. But one pattern recurs often enough, among the teams actually seeing a return, to be worth calling out on its own, applied at every stage of a pipeline rather than bolted on where code gets written:

• Living references that encode what a team knows and get corrected as gaps surface.
• Scoped roles with a clear boundary on what a given contributor owns and what it hands off.
• Review gates that don’t bend regardless of whether the work in front of them came from a person or an agent.

Teams getting a real return haven’t rebuilt how they work. They’ve adapted the workflow they already had so it holds an agent to the same standard as everyone else on the team. Where any one of these three is missing, it doesn’t matter how good the model is underneath. They’re not rearranging chairs. They’re steering.

All the Gear and No Idea

Lessons Nobody Learned

The most useful diagnosis of this gap comes from MIT’s NANDA initiative, whose “GenAI Divide” report found that 95% of enterprise generative AI pilots show no measurable effect on profit and loss. That figure alone reads like evidence the technology isn’t ready. The more interesting part of the report is its diagnosis of why.

The researchers describe the core issue as a “learning gap”, not a model quality problem, and the data backs that framing up: projects run with an external partner succeeded at roughly twice the rate of purely internal efforts. The models available to both groups were the same. What differed was the discipline brought to deploying them, the willingness to design workflows around what the tool actually does rather than what an internal team assumed it would do.

The report also names the concrete failure modes behind the 95%:

• Bolting AI onto legacy processes that were never designed to accommodate it.
• Chasing visible “hero” projects instead of the unglamorous workflow changes that would actually move a P&L.
• Treating AI procurement and deployment like a traditional software rollout, a tool you install and wait for value from, rather than a contributor whose output needs the same scaffolding a new hire would need.

Amplified, Not Improved

Google’s DORA 2025 report supplied a sharper version of the same story from inside engineering. Individual measures like pull request throughput are up roughly 98% year on year among AI-adopting teams. Organisational delivery performance, the metric that actually reflects whether software ships reliably and value reaches users, is flat.

Thoughtworks’ analysis of the same report puts the framing plainly: AI is an amplifier of whatever engineering maturity already exists in an organisation, not a substitute for it. Teams with strong systems, clear ownership and rigorous review turn that individual throughput gain into real delivery improvement. Teams without those things get more code moving faster through a pipeline that was already the bottleneck, and the bottleneck doesn’t move.

Feeling Fast, Going Slow

There’s a third strand, and it’s the most uncomfortable one: it isn’t about organisations at all, it’s about individual perception. A METR study covered by Aviator asked developers using AI coding assistants how much faster they believed the tools made them, then measured actual task completion time. Developers believed they were roughly 20% faster. They were measured at roughly 19% slower.

That gap between felt and actual productivity doesn’t show up until someone measures the P&L, which is exactly what the NBER and PwC surveys did, at scale, across thousands of firms. Feeling productive and not producing a return are, apparently, entirely compatible.

None of these three findings blame the models. All three point at the same thing: the return is won or lost by what surrounds the model, not the model itself. Three things in particular decide which way it goes:

• The workflow it’s dropped into.
• The review it’s subjected to.
• The assumptions under which its output gets trusted.

Same Rules, New Player

If the problem isn’t the model, reaching for a better model, a longer prompt, or a more capable agent framework aims at the wrong target. What closes the gap is applying the same discipline that already governs good human contributors on a well-run team to agents too. That discipline shows up in three things, and they need to stay separate in your head, because each fails independently of the other two.

1. Living references. A document, or set of documents, that encodes what the team knows, how it works, and what it has learned the hard way, corrected as gaps are found rather than written once and left to rot. Anthropic’s own framing of Agent Skills captures why this matters for agents specifically. A skill isn’t a one-off prompt fired at the start of a session and forgotten. It’s a modular, iteratively refined instruction set the agent consults as it works, the same way a competent new hire keeps referring back to a team’s documented conventions rather than relying on what they were told in an onboarding call three weeks ago. A briefing given once decays. A reference that’s actively maintained doesn’t.

2. Scoped roles. A clear boundary on what a given contributor, human or agent, owns, and what it hands off to someone or something else. Practical guidance on sub-agent orchestration converges on the same short list every time: define roles and handoffs explicitly, put guardrails around what an agent is permitted to touch, and monitor completion and intervention rates so scope creep is visible rather than discovered after the fact. This is unglamorous organisational design. It’s also what makes a team of humans work, and there’s no reason to expect it to matter less once some of the contributors are agents rather than people.

3. Review gates that don’t bend. Work gets checked against the same standard regardless of who or what produced it, and that check happens before the work is trusted, not after something has gone wrong downstream. This needs no separate framework, only the discipline to apply it consistently, which is where it tends to go missing under pressure.

These need to stay three separate things, not one fuzzy idea of “AI governance”, because they fail independently. An agent can be handed an excellent, current, well-maintained reference and still overstep the boundary of what it was scoped to do, because a good reference tells you what the team knows, not what this specific task is allowed to touch. Equally, an agent can stay perfectly within a tightly scoped role and still produce output that shouldn’t pass review, because staying in your lane says nothing about whether the work you did in that lane was any good.

Knowledge, boundary and verification are three different failure points. A team that has solved one or even two of them can still watch the third one quietly eat the return on their AI investment, and because the failures are independent, fixing one doesn’t make the others go away.

Amplifiers Have No Taste

DORA and Thoughtworks frame AI as an amplifier of existing engineering maturity, usually in terms of good systems getting better. The reverse is just as true, and it’s where most of the damage in the numbers above comes from. An amplifier has no opinion about what it amplifies. Feed it a strong process and the gains compound. Feed it an ambiguous one and the ambiguity compounds instead.

This matters most upstream of code, in requirements, epics and user stories, where ambiguity has always lived and always caused damage, just slowly enough for a human to absorb it. A Deloitte-sourced statistic discussed by Synodus quantifies the old version of this problem: 41% of IT project failures trace back to unclear requirements, a finding that predates generative AI by years. A vague requirement handed to a developer used to cost a slow week: some clarifying questions, maybe a wrong first attempt caught in code review before it went anywhere.

That slow week was itself an informal feedback loop, and agile development formalised the idea: short iterations and fast feedback, so a misunderstanding got caught within days rather than months, because someone saw working software before too much got built on a wrong assumption.

Much of what gets practised as agile today has drifted a long way from that. Standups, story points and fixed-length sprints have hardened into ritual at plenty of organisations, followed because they’re the process, not because anyone is still using them to surface ambiguity faster. That’s well removed from what the Agile Manifesto actually asked for: people talking to each other and adapting quickly, not a fixed process for its own sake. Even done properly, though, the fix is calibrated to a human cadence, a stand-up the next morning, a demo at the end of the sprint. An agent doesn’t wait for either.

The same vague requirement handed to an agent doesn’t get a slow week. It gets a confident, complete, plausible-looking piece of work in minutes, built on a guess that’s never flagged as a guess. Several more decisions can get made on top of it before anyone checks: the same gap in the brief, compounding at a completely different speed.

This is the second thread running through the return-on-AI problem. Living references, scoped roles and review gates catch amplified ambiguity too, but only if they’re applied upstream, where a requirement or user story gets written, not just downstream where code gets reviewed. An organisation with rigorous code review but a loose process for writing epics has simply moved the failure point earlier in the pipeline. The amplifier is still running. It’s just amplifying ambiguity that arrived a stage earlier than anyone was watching.

The next two sections apply this to both ends of that pipeline, in the same order, so the fix mirrors the problem: first to product refinement, where the ambiguity originates, then to implementation, where it would otherwise land.

Garbage In, Upstream

This has traditionally been a product team’s job, and that’s no longer the whole picture. When an agent does most of the implementation, what it builds is set almost entirely by how well the work was defined beforehand, which is context and prompt design as much as product management in the traditional sense. Whoever shapes what an agent gets asked to build is doing this work now, whether or not “product” is in their title. Engineering and product definition stop being two functions that hand off to each other and start looking like the same job, seen from two different angles.

Start with the living reference. Most teams already have the raw material for this and don’t need to invent a new artefact: architecture decision records, business decision records, contributing guides, and existing product and engineering standards all capture exactly this knowledge, provided they’re kept current and consulted rather than filed away as one-off onboarding reading. The gap most teams have isn’t a missing reference. It’s a team’s personas, its non-negotiable constraints, the decisions it has already made and the reasons behind them, briefed once in a kickoff meeting and then left to drift out of date. Treating that material as something to maintain rather than something briefed once means writing it down somewhere an agent doing refinement work can consult it, and correcting it the moment a gap is found rather than letting the same misunderstanding recur across the next five tickets. A refinement agent working from a living product reference produces stories that reflect what the team has decided. One working from a stale slide deck produces stories that reflect what the team decided eight months ago, confidently, and without flagging the difference.

Then the scoped role. The contractor framing described by Udacity is the right mental model here, and it applies just as well to refinement as it does to code: an agent doing this work needs a clear scope, explicit acceptance criteria, and a defined way to verify the work is complete, in the same way a contractor works from a brief rather than sitting beside you waiting for line-by-line feedback. For product refinement specifically, that means being explicit about the split:

• What the agent owns: turning a rough idea into a structured candidate story, surfacing open questions, flagging ambiguity it can’t resolve on its own, and proposing a correction to the living reference itself when the work in front of it doesn’t match what’s documented.
• What it doesn’t own: deciding priority, making trade-off calls that belong to a product owner, resolving genuine ambiguity by guessing, or merging that correction into the reference unreviewed.

An agent with a scoped role that includes “flag what you can’t resolve” produces stories with visible gaps in them. An agent left to fill every gap itself produces stories that look finished and aren’t. The reference-correction part matters for the same reason: an agent working from the document all day is often the first to notice where it’s gone stale, and proposing the fix is what keeps it living rather than labelled that way. It still goes through the same review gate as everything else. Being the one to spot a gap in the team’s knowledge doesn’t mean getting to also approve the fix.

Finally, the review gate, applied here with the same rigour a code review gets and for the same reason. Spec-driven development approaches exist to give this stage something concrete to check against, structured acceptance criteria and explicit constraints rather than prose a human can read generously and an agent can’t read at all. A story doesn’t move from refinement into a sprint because an agent produced something that reads well. It moves because a human owner has checked it against the same acceptance-criteria standard every story is checked against, and that check happens regardless of whether the first draft came from a product manager typing at 2am or an agent generating candidates in ten seconds.

The value of doing this upstream is that it’s cheaper to catch an ambiguous requirement before it’s built than after. A story that reaches a sprint with a hidden assumption baked into it doesn’t get more correct by being implemented well. It gets built accurately wrong, at speed, by whatever contributor picks it up next, which is exactly the amplification problem from the previous section arriving on schedule. Applying it at refinement is what stops that handoff from happening in the first place.

Garbage Out, Downstream

The same three things apply to the point where a scoped, reviewed story becomes code, though each does a different job here than it did upstream, because implementation is where “good” stops being a matter of judgement and starts being something you can check.

The living reference stops being about what the team believes and starts being about what this codebase has already decided. General engineering wisdom, keep functions small, avoid duplication, don’t repeat yourself, is true everywhere and specific to nowhere, which means there’s nothing in it an agent can be held to. What closes that gap is a reference tied to this system’s own history: the patterns it has settled on, and, more importantly, the mistakes it has already made and doesn’t want repeated.

A team that never writes those lessons down isn’t just failing to onboard its agents properly, it’s discarding its own institutional memory every time a session ends, and paying to relearn the same lesson at whatever it cost the first time. That cost falls harder on an agent than on a person. A developer who got burned by a flaky dependency six months ago carries the caution forward even after forgetting the detail. An agent carries forward nothing that isn’t written down.

The scoped role does different work here too. Upstream, the boundary was about which decisions belonged to a person and which could be handed to the agent doing the drafting. Downstream, it’s about which part of the system a given change is allowed to touch, this layer and not that one, this service and not the one behind it. That boundary matters more, not less, once an agent is doing the implementing, because an agent under no constraint takes the shortest path to something that runs, and the shortest path often cuts straight through a boundary that has nothing to do with whether today’s feature works. Code that reaches across a boundary it wasn’t scoped to touch can pass every test in front of it and still make the next six months of change to that system harder, and “it works” doesn’t catch that on its own.

Which is what the review gate exists to catch, and why it can’t soften at this stage. A gate that only checks whether code looks reasonable, was written quickly by someone experienced, or came from a source the team trusts has already stopped being a gate. The only version of it that holds is one that checks the work itself, against tests, against behaviour, against whatever the team already uses to decide correctness, applied the same way regardless of whether a person or an agent produced what’s in front of it.

This is also where it becomes obvious that the three things are separate failure points rather than one problem wearing three names. An agent can be fully briefed on the system’s history and never leave the part of it it was scoped to touch, and still ship something subtly wrong, an off-by-one, a missed edge case, a race condition, because neither the reference nor the boundary has anything to say about correctness. Only the gate does. That’s the whole reason it’s the one that isn’t allowed to bend.

Built In, Not Bolted On

Everything so far has been about applying this discipline to a single piece of work, a story, a feature, a task. The amplification effect described earlier doesn’t stop at individual tasks, though. It applies to whole systems: the pipeline a change moves through, the architecture that change lands in, the process a team follows when something breaks in production. A pipeline, an architecture, or an incident response process built around human-paced iteration doesn’t just slow an agent down once it’s working inside it. It can fail in ways that compound faster than a human team can catch, because the checks that made the system safe were sized for the speed and volume of human work, not machine-speed work.

Pipelines Built for Yesterday

This is the mechanism the CI/CD-for-agents argument and the related analysis of the software supply chain both describe. A pipeline designed around a handful of human-authored pull requests a day, reviewed by people who each bring their own scepticism and pattern recognition to the diff in front of them, behaves differently once it’s receiving a much higher volume of agent-generated changes arriving much faster. The gates that used to be adequate weren’t built with that volume or that speed in mind:

• A single reviewer’s sign-off.
• A test suite sized for the pace changes used to arrive at.
• An incident process that assumes a human is the one making each remediation decision.

None of them degrade gracefully when the assumption underneath them stops holding. They degrade suddenly, at exactly the moment an organisation has the least slack to absorb it.

Bolted On, Not Built In

This is also the “bolting AI onto legacy processes” failure mode the MIT NANDA report names as one of the core reasons pilots fail to reach the P&L. It isn’t that the process was inadequate before. It’s that a process adequate for human-paced iteration is a different thing from a process adequate for machine-paced iteration, and installing an agent into the first without redesigning it into the second doesn’t make the process worse in the way a slow human contributor would make it worse.

It makes it fail differently, in ways the existing checks weren’t built to catch, because those checks were never tested against this speed or this volume.

The implication is that the same three things, a living reference, a scoped role, a review gate, have to be designed into a pipeline, an architecture or an incident process from the outset, not retrofitted once an organisation notices agents are already moving through it. Retrofitting means discovering the gap through an incident.

Designing for it from day one means the review gate is sized for the actual volume and speed agents will produce before that volume and speed ever arrives, the ownership boundaries between systems are drawn explicitly rather than assumed, and the reference material an agent needs to operate safely inside a given system exists before the agent is given access to that system, not after something in it has already gone wrong.

Engineering Isn’t Special

None of this is specific to software. The same pattern, a living reference that’s corrected as gaps are found, a scoped role with a clear ownership boundary, and a review gate that doesn’t bend regardless of who or what produced the work, applies anywhere an organisation hands ambiguous input to a person and relies on that person’s judgement to quietly clean it up before it becomes a decision, an answer, or an action. Legal, sales, customer success, and finance all run on that kind of judgement today, absorbing vague inputs and turning them into something usable, largely invisibly, largely without anyone naming it as a discipline worth protecting.

The moment agents start doing pieces of that work too, the same three failure points open up in those functions that have opened up in engineering, and they open up for the same reason: the judgement was never written down as a reference, the role was never scoped as a boundary, and the review was never a gate that held regardless of source. Engineering is where the evidence surfaced first, because it has spent the longest time measuring itself well enough to notice the gap.

Turns Out, This Was the Job

The research surveyed at the start of this piece tells a consistent story from four independent angles: adoption is real and growing, and measurable return mostly isn’t following it. That gap isn’t evidence that the technology has been overstated. It’s evidence that most organisations are deploying it the way they’d deploy a faster tool rather than the way they’d onboard a new contributor, and a faster tool doesn’t need a living reference, a scoped role, or a review gate. A contributor does, whether that contributor is a person or an agent, and the organisations closing the gap are the ones that noticed the second thing was true before the numbers forced them to.

None of the three components of that discipline are new. Good engineering teams have maintained living documentation, drawn clear ownership boundaries, and enforced review gates that don’t bend for years, long before agents were part of the picture. What’s changed is the cost of skipping any one of them. A missing reference, a fuzzy boundary, or a soft review gate used to produce a delay and a correctable mistake. The same gaps, with an agent working inside them, produce confidently wrong output at a volume and a speed no team can review its way out of after the fact.

At Wyrd Technology, this is the work we do with engineering organisations: we don’t sell a faster route to AI adoption. We help teams design the workflow underneath it properly: the references, the boundaries, and the review gates that let the system hold once agents are doing real work inside it. Otherwise, they get another pilot that never reaches the P&L.

The return on AI was never going to come from a better model. It comes from the same systems and discipline that were always the real work, applied to one more kind of contributor.