Every operations review has the slide: uptime, five nines-ish, a reassuring line that only goes up. For twenty years that slide meant something, because for twenty years “the system responded” and “the system worked” were close enough to the same thing.
For AI agents, they’ve come apart. AgentStatus’s March 2026 reliability report fired 4.5 million outside-in tests at 6,259 production agents from real residential networks, the way actual users reach them. The findings: 56.6% of agents held perfect uptime all month, and 89% of responses failed every answer-quality check they were given. Only 0.2% of test executions came back reachable, on time, and correct. The agents are online. The answers are wrong.
(You may have seen that 56.6% number going around as an agent “success rate.” It’s the uptime share, and half the posts citing it got it backwards. The real success figure is two orders of magnitude worse, which rather proves the report’s point about nobody checking answers.)
Why your tools can’t see it
Every tool in a standard monitoring stack is doing exactly the job it was hired for: watching the machinery. Status codes, latency, error rates, restarts. The problem is that an agent which confidently gives a customer the wrong refund policy returns the same healthy status code as one that gets it right. The failure is in the content, and the content is the one thing the dashboard doesn’t read.
And it doesn’t stay fixed once you’ve fixed it. AgentStatus’s follow-up on drift found 88% of agents started giving worse answers at least once inside a 30-day window, often after some upstream change nobody connected to them. Answer quality moves like a tide, and most teams measure it exactly once, at low water, on launch day.
Questions that find the gap
You don’t need to build anything exotic to get ahead of this, and you don’t need to be the technical one in the room to ask for it:
- Do we test answers, or just endpoints? A dozen questions with known correct answers, asked automatically every day, from where your users actually are. If the agent starts flubbing them, you find out from a test instead of a customer.
- Who owns “the agent said something wrong”? Support logs it as a ticket, engineering sees a healthy service, and the complaint dies in the gap between them. Somebody’s name goes on that failure mode or it belongs to no one.
- What’s the rollback? When quality drifts after an update (a model swap, a prompt tweak, a vendor’s silent improvement), can you get back to yesterday’s behavior, and how fast?
- If it’s a vendor’s agent: how do they measure correctness? Ask exactly that, then listen. “We monitor uptime and latency” is the wrong answer wearing a straight face.
None of this is hard. Most of it is a spreadsheet, a scheduled job, and a name in a runbook. What it mostly requires is accepting the uncomfortable premise: for software that talks, up tells you almost nothing.
“Is it up?” was the last decade’s question. “Is it right?” is this one’s. The companies that ask it will find out from their own tests, quietly, on a Tuesday. Everyone else will find out from their customers, and the customers will not file it as a P3.