Claims agents are already right the vast majority of the time. The question carriers now face is what it takes to let them act on their own, and the answer lies in the small share of decisions they get wrong. Once agents move from answering coverage questions to approving claims and endorsing policies, the readiness test is whether a team can take any one of those calls apart years later and show which agent made it and what evidence it used. For a carrier, that paper trail determines whether an agent gets to deny a claim on its own or still has to hand the file to an adjuster.
Harsha Patnam is an independent AI Product Leader with more than 17 years of experience building products across insurance technology and fintech. His career has run through data platforms, machine learning products and insurance automation, including senior leadership of product and technology as a Vice President. He builds AI-driven products and sees governance as the foundation that lets carriers expand them across the business. His background in rules-based automation gives him a useful dividing line, since once a system stops producing the same answer every time, the controls around it have to change.
“We shouldn’t be worried about the 90 or 95% of the times that AI is right in automating decisions. We should be concerned about the 5 or 10% of times that it gets it wrong, and whether there’s sufficient explainability behind it,” Patnam says. In a claims workflow, that explainability means tracing why a claim was approved or denied all the way back to the data and approval gates behind it. Executives weighing whether an agent can decide on the organization’s behalf need that answer long before they need a benchmark score.
The 5% is the real production test
That bar is where agents start delivering real value. Patnam draws a clear line between adopting a model and deploying it well. The second is the harder and more valuable work because it means embedding the agent in live workflows where its decisions carry weight. “Taking one of the frontier models and saying yes, we are utilizing AI within an organization is easy, and that gets a round of applause within boardrooms,” he says. The returns that outlast the applause come from the deployment work that follows.
He has spent years building automation that runs on fixed rules and returns the same answer to the same input every time, and agents give up that guarantee the moment they go live. “Deterministic automation is all about if X, then do Y. AI is different. AI is more probabilistic. How it interprets the given contextual information and what decisions it makes could be totally different in two different attempts,” he explains. A rules engine can be validated once and trusted to repeat itself. An agent gets judged by its misses, and every one of them has to be reconstructable and containable on demand.
Autonomy gets earned with evidence
His deployment path starts with assistive work on rote, low-consequence tasks such as answering policy questions or pulling coverage details from a policy document. Agents then move up to recommending and drafting next actions for a human to approve, and fully automated decisions come last. Each rung requires proof that the organization can see what the agent did and catch it when it was unsure, and the top rungs belong to decisions that can’t be undone. Roadmaps that skip straight to the top tend to stall, while those that climb each rung deliberately can expand autonomy with confidence.
“You don’t want a really amazing brand to have reputation damage because of a denied claim, because of an agent decision that looked at the information provided for a claim and automated a decision that is adverse to the claim being submitted,” Patnam says. A misapplied bill payment or a botched policy endorsement lands on the same complaint desk, so an agent moves up a rung only after the carrier can pull its recent calls and show they held up under review. Authority handed out early tends to get clawed back, with roughly 40% of enterprises expected to demote or decommission autonomous agents by 2027 once governance problems surface in production.
Every decision needs a chain of evidence
“Six months down the line, am I able to look at an action that an agent took and triangulate the factors that led to the AI agent making this decision? It’s all the log history,” Patnam says. That question works as a practical test for any agent up for more authority, and it gets much harder to answer after the fact.
Answering it takes what he calls seven layers of trust, which together record who the agent is, what it was allowed to touch and how it reached each decision.
Identity: Which agent took the action.
Authorization: Which systems the agent could access and whether sensitive data stayed cordoned off.
Data provenance: The sources the agent pulled and where that evidence came from.
Evaluation: The model version and system prompt behind the decision, and whether its outputs hold up under review.
Observability: The incoming request, how the agent interpreted it and how certain it was.
Auditability: A retained record of the decision and its outcome that can be reconstructed years later.
Human escalation: The point where the agent hands a decision to a person.
Those records have to survive far longer than any model version. “The data retention standards vary from state to state. In some states it’s three years, in some states it’s five. In some states it’s as high as seven years. And in the fintech world it’s decades,” Patnam notes. Regulators already expect that record, with roughly half of U.S. states adopting the NAIC model bulletin that asks insurers to document how their AI systems are governed and monitored.
Patnam saves his heaviest emphasis for the final layer. “When does the AI agent actually bring or defer for human escalation when there is uncertainty? That is the most critical aspect,” he says. When an agent can tell it’s unsure and hands the claim to a person, a likely mistake gets caught before it ever reaches the policyholder. Regulators and courts are increasingly scrutinizing that handoff, including a 2025 ruling that let a lawsuit over algorithm-driven claim denials move forward.
Bad context can make a good agent wrong
A model can behave exactly as designed and still produce a bad decision when the evidence it retrieves is stale or skewed. Patnam sees that risk most clearly in pricing, where carriers file updated rates with state insurance departments so often that a quote expires within weeks. A premium prediction tool working from old rates can be wrong before anyone even gets to the question of bias.
“If the underlying data already had bias introduced into it in the form of demographic or geographic or risk exposure, and the AI agent isn’t built with the right guardrails, the same bias is introduced and even more so amplified by the AI agent,” Patnam says. An agent estimating premiums from last year’s rate environment, or from historically skewed data, hands customers a confident number built on evidence nobody should trust. New York regulators already expect insurers to test whether external pricing data works as a proxy for a protected class.
His fix is to match data freshness to the consequence of the decision. A retrieval system answering policy questions on customer calls can run for long stretches without retraining because it simply looks up a specific policy and returns what it finds. Premium prediction gets a much tighter window. “I should be only using data within the last 30 to 90 days, nothing beyond that,” he explains. Claims and catastrophe modeling sit at the other end, where a decade of history reveals seasonality and expected loss ratios, and the right window will vary by carrier and line of business while the principle holds everywhere.
All of that infrastructure costs money before it prevents a single bad decision, and competitors bragging about agents in production make the bill harder to defend. Patnam’s answer to the CFO who balks comes straight from his own industry. “Why do you keep spending thousands of dollars each year insuring your assets when you could be putting that in the bank and letting it compound and cover it yourselves?” Patnam asks. Every carrier sells protection against a loss it knows will eventually come, and the controls around an agent run on the same math. Those same controls are what carry an agent past the pilot stage and into wider production.
With that foundation in place, a carrier can hand an agent more authority with confidence. Once the organization can reconstruct why an agent made a call and reverse it before it does lasting damage, the agent can take on decisions that once sat with a human reviewer. “You have to always have controls for when things do not go right,” Patnam says.
The views and opinions expressed are those of Harsha Patnam and do not represent the official policy or position of any organization.
