I can't point to one moment this clicked for me. It was more that the same pattern kept repeating across different systems until it stopped looking like a coincidence. The industry often talks about human-in-the-loop as if it's a feature with an expiration date: essential at first, unnecessary once the model becomes "good enough."
That framing falls apart the first time an automated system executes against real money before anyone's had a chance to catch it, and at scale, that happens regularly enough to plan for, not just acknowledge. Models hallucinate and misread, and a human in the loop catches some of that. But the failure that hurts most often is the opposite: the model was right, and did exactly what it was built to do. The failure sits upstream of the model, in a decision about what it was allowed to do without a person looking first. Human-in-the-loop isn't only a hallucination filter you bolt on. It's a control on what the system is allowed to do alone. That distinction matters more than almost anything else in this article, so let's start there.
Where AI in commerce actually carries risk
Most of what gets written about AI in e-commerce lives on the customer-facing side. Recommendations, search ranking, a chatbot that helps you find the right size. It's visible work, and it's genuinely useful work. A bad recommendation gets scrolled past and nobody remembers it by lunch.
The harder half sits underneath, in the systems nobody outside the company ever sees. Pricing logic, inventory allocation, fulfillment routing, returns processing, the quiet reconciliation work that decides whether what a system says actually happened. AI is moving into this layer fast, and it's a fundamentally different kind of problem than the storefront, because the systems here are already autonomous in ways the storefront isn't. Money moves on its own. A truck leaves the dock without a person signing off. A seller's listing disappears from in front of buyers who were about to buy it. By the time someone notices, the only thing left to do is damage control.
This shows up most often in returns and pricing, less often in something like search ranking, mostly because the first two have a direct line to money moving and the third usually doesn't. That's the entire reason this topic deserves more attention than "how confident is the model."
Stop asking how much autonomy; start with reversal cost
The question that matters most is reversal cost. It's Amazon's one-way versus two-way door framework, pushed one layer down: the door an action opens, not the model's confidence in opening it, determines whether a person should see it first.
Some actions cost almost nothing to reverse. A listing flagged and held for review hasn't hurt a buyer, and you can release it quickly if the flag was wrong. Other actions are expensive, or impossible, to undo. A refund that's already settled. A price that's already been seen and acted on by ten thousand people. A shipment that's already on a truck.
What counts as reversible isn't fixed. It's a property of how the platform underneath is built. At Shopify, reversibility was designed into the order primitives: an order edit is a first-class operation, cheap to undo by construction. At Amazon, the same move hits physical logistics: once a fulfillment decision commits inventory and the truck is rolling, no data model walks it back for free. At TikTok Shop, a wrongful takedown may be technically reversible, but the GMV and seller trust lost while a listing stays dark are not. Same word, three price tags, because the architectures differ.
Reversal cost is the anchor, but in production it travels with three related questions: what is the consequence of one wrong action, how much exposure can accumulate before anyone intervenes, and how quickly will the failure be detected? Those variables don't replace reversal cost. They tell you how tightly to control the actions that appear reversible on paper.
Confidence and reversal cost are not the same variable. I won't put a number on how often this bites teams; it's too domain dependent to fake one. The shape of the problem holds up everywhere I've seen it: a model can be 98 percent confident and still land in the 2 percent that costs you real money, and at scale, that 2 percent shows up often enough to matter.
Here's roughly how to think about routing this. Not production code, just the shape of the decision.
function routeDecision(decision):
reversal_cost = classify_reversal_cost(decision.action_type)
aggregate_exposure = estimate_aggregate_exposure(decision)
detection_latency = estimate_detection_latency(decision)
confidence = model.get_confidence(decision)
if reversal_cost == "high":
return "escalate_to_human"
if aggregate_exposure > EXPOSURE_LIMIT:
return "queue_for_review"
if detection_latency > MAX_DETECTION_LATENCY:
return "queue_for_review"
if reversal_cost == "low" and confidence >= AUTO_EXECUTION_THRESHOLD:
return "auto_execute_with_limits_and_audit_sample"
return "queue_for_review"
The important branch is the first one. A high-cost action still requires review even when the model is highly confident. Confidence can help route decisions inside an authority boundary, but it should not erase that boundary.
Even a low-cost action needs limits, because something cheap to reverse once may become expensive when repeated ten thousand times before anyone notices.
Why narrow authority holds up better than broad mandates
There's a real temptation to build a single agent that handles "operations" as a category. One system that triages returns, flags pricing anomalies, and routes fulfillment exceptions, all under one roof. It sounds efficient on a slide. In practice, it tends to fall apart in a specific way.
There's a live version of this debate right now. Cognition's "Don't Build Multi-Agents" (2025) warns that splitting tightly coupled work across agents can fragment context and produce conflicting decisions. Anthropic's multi-agent research system showed the other side of the trade-off: multiple agents can improve breadth when work is highly parallel, but at the cost of coordination and compute overhead.
For commerce, the more important question is not how many agents are involved. It's how broadly the system is authorized to act when money moves.
The problem is accountability. Scope an agent’s authority to one judgment call, like reading a return and classifying why it happened, and you can define exactly what "good" looks like, build a tight feedback loop, and trace any failure straight back to its source. Authorize that same system to classify the return, issue the refund, alter inventory, and penalize the customer, and the moment something goes wrong, you're untangling which task, which input, which edge case. Nobody can tell you cleanly anymore.
The honest cost on the other side is that you end up maintaining more of these narrow authority boundaries, and someone has to own the seams between them. I don't think that cost is bigger than the alternative.
A useful rule of thumb: if you're describing what an agent is allowed to commit and you need the word "and," its authority is probably too wide.
Reviewers need more than a yes-or-no button
Deciding you want a human in the loop is the easy part. Most teams stop there, like checking a box, and that's exactly where it quietly stops working.
What the reviewer sees matters enormously. Hand someone a decision to approve or reject with none of the context that produced it, and you get a reviewer who's either rubber-stamping everything or fighting everything, and neither tells you much about whether the system's actually working. Give them the surrounding context, and you get an actual judgment.
How decisions reach the reviewer matters too. Sending every low-confidence case to a person one at a time, the moment it happens, doesn't scale and doesn't produce better decisions. It produces interruption. Group similar low-stakes cases and review them in a batch, and you get someone who can actually see the pattern, not just the case in front of them.
Batching does not solve the deeper problem. It trades one kind of fatigue for another. Instead of being worn down case by case, reviewers can become worn down by pattern matching all day.
The failure mode nobody warns you about enough is reviewer fatigue. Human-factors researchers were documenting this failure mode at least as far back as Parasuraman and Riley's 1997 review of automation misuse. Experts are not immune, and training alone does not reliably eliminate the problem. The real risk in many human-in-the-loop systems isn't the model making a bad call. It's a tired person clicking approve on everything because the volume routed to them crept up slowly enough that nobody noticed. If your escalation rate quietly grows month over month and nobody's tracking it, you've rebuilt full automation. You just added a click in the middle to make it feel safer.
A simple version of batching low-confidence decisions instead of firing them off one by one:
function batchForReview(decisions):
low_confidence = filter(decisions, d => d.route == "queue_for_review")
grouped = group_by(low_confidence, d => d.action_type)
for group in grouped:
if group.size >= BATCH_THRESHOLD or time_since_last_batch(group) > MAX_WAIT:
send_to_reviewer(group)
Batching still needs an escalation-rate metric. If the share of cases requiring review keeps growing, the system is not becoming safer. It is quietly transferring the automation problem to the reviewers.
Where pilots actually break down
The same story plays out across enough teams that it's worth naming directly. An agentic system performs beautifully in a pilot. Clean data, a small team watching closely, a curated set of cases. Then it goes live, and it breaks.
MIT's NANDA initiative reported in 2025 that only about 5 percent of task-specific enterprise GenAI tools in its study reached successful implementation. The figure is directional rather than universal, but the underlying finding matters: integration, adaptation, and workflow fit were major differentiators, not model capability alone.
The model is often not the primary failure. The handoff is. In a pilot, the people who built the system are also the ones reviewing its output, in an environment they control. In production, that same decision has to flow into systems nobody fully owns anymore, hit edge cases nobody thought to test for, and produce consequences that someone has to answer for after the fact, usually not the person who built the agent.
So before launch, somebody needs an actual answer for who finds out when it's wrong in production, how fast, and what the rollback looks like. Whether it worked in the pilot tells you almost nothing about that. If nobody can answer it cleanly, the question doesn't go away. It just gets answered later, badly, by whoever's stuck with it.
Build the audit trail and the rollback path on day one, not as the lesson learned after the first incident. That is harder to fund before anything has gone wrong, but much more expensive to add after it has.
Start small: here's how
If you're sitting down to design one of these systems this week, here's where I'd start.
Pick one decision, not a category of decisions, and ask what it costs if it is wrong, how easily it can be undone, how much exposure can accumulate, and how quickly anyone will detect the failure. Those answers, not the model's confidence score alone, tell you whether a human needs to see it before it executes.
Resist the urge to give one agent authority over several consequential judgment calls at once, even when it looks more efficient on paper.
Whatever review step you put in place, design it as something a person actually looks at, not a checkbox that happens to have a human's name attached.
Where this actually leaves us
Human-in-the-loop isn't a phase your AI system automatically grows out of as the model gets better. In commerce operations, it is one part of an authority architecture tied directly to the consequences, reversibility, exposure, and detection time of a given action.\ \ Grant autonomy by action, not by agent. Use confidence to route work inside an explicit authority boundary, not to erase that boundary.\ \ The point is not to shrink the human's role over time. It is to keep people exactly where their judgment is worth more than the speed gained by removing them, and to keep checking whether that remains true instead of assuming it from the last time you checked.\ \
About the author
Anwesha Roy has spent over a decade building commerce, operational, and AI systems at Amazon, Shopify, and TikTok Shop. Her work sits in the layer most people never see: pricing, inventory, fulfillment, returns, and the trust systems that hold a marketplace together at scale. She writes about agentic AI's collision with that layer, where the stakes are different and the easy answers usually don't hold up.
