Evaluating AI Agents: Why 95% Per-Step Accuracy Is a Failing Grade (Part 1 of 2)

Why agent evaluation breaks the tools built for prompts, and how to measure outcomes instead of vibes.

Estimated reading time: 9 minutes

Key Takeaways

• Reliability compounds against you: An agent that gets each step right 95% of the time completes a 20-step task 36% of the time. Per-step accuracy is not the number you ship on.
• Grade the outcome, not the path: Check the environment's end state, not the transcript. Agents find valid approaches you never anticipated, so asserting a fixed sequence of tool calls just makes tests brittle.
• Agents lie about finishing: In a 2026 study, 45-48% of failures in single-control τ²-bench domains were cases where the agent declared success while the environment said otherwise.
• pass^k beats pass@k: A 70% agent run three times succeeds at least once 97% of the time, and every time 34% of the time. Production doesn't get retries.
• Start absurdly small: 20 to 50 tasks drawn from real failures is a legitimate starting suite. Waiting for a thousand means shipping blind for a year.

Table of Contents

• The Compounding Math: Why 95% Per Step Is a Failing Grade
• Grade the Outcome, Not the Story
• The Reliability Metric Almost Nobody Reports
• Building Your First 20 Tasks

Your agent works. You've run it fifteen times, watched it pick the right tools, and it did the thing. So you ship it.

Three weeks later it's quietly failing on a chunk of real requests, and nobody can tell you which change caused it - because nothing in your test suite was ever measuring the thing that broke.

I've been on the wrong end of exactly this. I built a hybrid retrieval system once, keyword search fused with vector search, and I had tests. Green across the board. What those tests actually asserted was that a search returned results. Not that the right result came back first. So when a scoring bug inverted the ranking and put the weaker match above the stronger one, every test stayed green. It shipped, and it sat in production for three weeks until I noticed by eye.

That wasn't a testing failure. It was an evaluation failure. My tests proved the code ran. Nothing proved the output was any good.

Here's the thing: that gap gets dramatically worse when your system stops being one prompt and becomes an agent. A prompt eval scores one input against one output. An agent produces a trajectory - model turns, tool calls, results, changes to the world - and ends in a modified environment rather than a string. Almost none of the intuitions you built on prompt evals survive that transition.

Let's break down what changes, and what to measure instead.

The Compounding Math: Why 95% Per Step Is a Failing Grade

Start from first principles. If an agent's steps succeed independently with probability p, then finishing an N-step task means getting every step right: p^N.

That exponent is unforgiving.

Read the 95% row. Five steps and you're at 77%. Twenty steps, 36%. Fifty steps, 7.7%. A number that looks excellent on a prompt eval dashboard produces a system that fails two times in three.

Now invert the question, because this is the version that changes how you build. To hit 90% end-to-end, what per-step reliability do you need? Over 10 steps, 98.95%. Over 100 steps, 99.89%.

Is 100 steps realistic? OSWorld 2.0, published June 2026, builds 108 long-horizon computer-use workflows with a median human completion time of about 1.6 hours. Its tasks average 318 tool calls. At a 500-step budget, the best system on the board completes 20.6% of tasks end to end, scoring 54.8% on partial credit.

To be honest about the limits: this is an intuition pump, not a measurement. Real steps aren't independent. Failures correlate, since one bad plan dooms everything after it, and agents also retry and self-correct, which pushes real performance above p^N.

But the direction is right, and the direction is what matters: long-horizon reliability cannot be extrapolated from single-step accuracy. You have to measure it end to end.

Grade the Outcome, Not the Story

So what do you assert against?

The instinct most engineers have - and I had it too - is to check that the agent followed the right steps in the right order. Anthropic's engineering team pushes back on this directly in Demystifying evals for AI agents:

"We've found this approach too rigid and results in overly brittle tests, as agents regularly find valid approaches that eval designers didn't anticipate. So as not to unnecessarily punish creativity, it's often better to grade what the agent produced, not the path it took."

Their flight-booking example is the clearest statement of the principle I've seen: the agent might end its transcript with "Your flight has been booked," but the outcome is whether a reservation exists in the environment's SQL database.

This is not hypothetical. τ-bench was built on exactly this idea - it "compares the database state at the end of a conversation with the annotated goal state." And a 2026 paper, From Confident Closing to Silent Failure, put numbers on why transcripts can't be trusted: agents assert completion when the environment says otherwise in 45-48% of failures in single-control τ²-bench domains, and 75.8% of self-assessing coding-agent trajectories on AppWorld.

Here's the part that should change your tooling choice. That same paper tested LLM judges at detecting these false successes. The judges never beat AUROC 0.65 on τ²-bench and managed only 0.54 on AppWorld. A plain TF-IDF detector hit 0.83 and 0.95 respectively, at 3,300 times lower latency.

On the specific question of "did the agent actually finish," a cheap deterministic detector beat the expensive language model badly. Reach for the state check first.

From a practical standpoint, a useful agent assertion has five layers:

def test_refund_task(agent, env, task):
    run = agent.run(task.instruction, tools=env.tools)

    # 1. OUTCOME - the environment must match the annotated goal state
    assert env.db_hash() == task.gold_db_hash

    # 2. REQUIRED ACTIONS - the side effect must actually have happened
    assert env.called("issue_refund", order_id="W123", amount=52.40)

    # 3. COMMUNICATION - required information reached the user
    assert "5 to 7 business days" in run.final_message

    # 4. FORBIDDEN ACTIONS - the negative case matters just as much
    assert not env.called("cancel_order")

    # 5. BUDGET - cost and latency are results, not footnotes
    assert run.n_turns <= 12
    assert run.cost_usd <= 0.35

Assertion 4 is the one people skip, and the one I'd argue hardest for. Anthropic's guidance is blunt: test "both the cases where a behavior should occur and where it shouldn't," because "one-sided evals create one-sided optimization." An agent that never refuses anything will ace a suite made entirely of things it should do. That's not a capable agent. That's an unguarded one, and you built the suite that rewarded it.

The Reliability Metric Almost Nobody Reports

Now the metric that changed how I think about shipping agents.

Most people know pass@k from code benchmarks: out of k attempts, did at least one succeed? It's the right metric when a human filters the output, like a coding assistant offering suggestions you review.

It is exactly the wrong metric for an agent acting on its own.

τ-bench introduced pass^k for this: out of k attempts, did all of them succeed? That's the probability your agent handles the same task correctly every single time a customer asks.

The gap is brutal. Take an agent that succeeds 70% of the time on a single attempt. Run it three times: pass@3 is about 97%. pass^3 is about 34%.

The published τ-bench numbers show the same shape on real systems. In the retail domain, claude-3-5-sonnet goes 0.692 at pass^1 down to 0.462 at pass^4. In airline, gpt-4o falls from 0.420 to 0.200. The paper's own abstract puts it plainly: state-of-the-art function-calling agents "succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail)."

Here's where it gets interesting. Measured pass^k decays slower than simply raising pass^1 to the power of k. For that retail row, 0.692^4 would be 0.229, but the measured pass^4 is 0.462 - double. That gap means task difficulty is heterogeneous: some tasks the agent nails every time, others it never gets.

Which gives you a free diagnostic. If your measured pass^k lands close to (pass¹⁾k, your task suite is too uniform and probably isn't stressing anything.

Building Your First 20 Tasks

Most teams have no agent evals because they picture a thousand hand-labelled cases and quietly decide to do it later. So let me kill that objection with the actual recommended number.

Anthropic's guidance: "20-50 simple tasks drawn from real failures is a great start."

Four rules make those tasks worth having.

1. Two experts must agree. A good task is one where two domain experts would independently reach the same pass/fail verdict. If you and a colleague can argue about whether a run passed, the task is underspecified and will produce noise forever.

2. Don't write the rubric first. This is the counterintuitive one, and it comes from Shankar et al.'s "Who Validates the Validators?". They named a phenomenon called criteria drift: "users need criteria to grade outputs, but grading outputs helps users define criteria." It's a genuine catch-22. Some criteria simply cannot be defined until you've looked at what your system actually produces. So look at real traces first, then write the rubric the failures suggest.

3. Review enough traces to hit saturation. Hamel Husain's evals FAQ gives a clean heuristic: "you should aim to review at least 100 traces," and "if ~20 traces don't turn up a new category, you can stop." Group what you find into a handful of named failure modes, then count them. The counts tell you what to fix. The failure modes tell you what to measure.

4. Make the verdict binary. Not a 1-5 score. "Binary evaluations force clearer thinking and more consistent labeling." A 1-5 scale invites annotators to park everything on 3, and nobody can tell you what separates a 3 from a 4.

One number for perspective on the investment: Husain reports spending 60-80% of development time on error analysis and evaluation. That sounds enormous until you remember that the alternative is what I did with that retrieval bug - shipping a silent regression and finding it by accident three weeks later.

Ask Yourself What You'd Bet On

Every engineer building agents right now is making an implicit claim: that watching a system work a dozen times tells you how it behaves a thousand times. The compounding math says that claim is false, and says so quantitatively.

What makes this worth the effort isn't the dashboard. Evaluation is the only thing that converts "I think it got better" into "it got better, and here's the number." Non-deterministic software doesn't give you that for free the way a compiler does. You build the instrument yourself.

So start small, this week. Pull twenty real failures out of your logs. Write the outcome assertion for each - the end state, not the transcript. Run them three times and look at your pass^3, not your pass@3. That's a weekend of work, and it will tell you more than every demo you've run so far.

Part 2 covers what code assertions can't reach: building an LLM judge that isn't fooling you, evaluating RAG without debugging blind, which frameworks are worth adopting in 2026, and wiring it all into a CI gate. Read it here: Building an Eval Stack That Catches Regressions (Part 2 of 2).

Measure the outcome, or you're just watching a demo.

Frequently Asked Questions

Do I need an evaluation framework to start?

No, and starting with one is often a mistake. Your first agent eval can be plain pytest with the five assertions above. Frameworks earn their place once you need dataset management, judge alignment, or a shared dashboard - Part 2 covers which ones are worth it.

Can I just use a public benchmark like τ-bench or SWE-bench instead?

Use them to eliminate weak models, not to choose your system. Public benchmarks are contaminated and frequently buggy - a 2025 audit of ten major agent benchmarks (the ABC checklist paper) found flaws that shift reported performance "by up to 100% in relative terms," including τ-bench counting empty responses as successes. Your own 20 tasks are worth more than any leaderboard.

Is it safe to run agent evals against real systems?

Treat the eval environment as a place where a badly-behaved agent will do real damage, because that's exactly what you're testing for. Run against a sandboxed copy with seeded data, never live credentials or production databases. And include tasks where the correct outcome is that the agent refuses and escalates - that's the negative case from assertion 4, and it catches an agent willing to issue a refund it shouldn't.

How often should I redo the error analysis?

Whenever something material changes: a model upgrade, a prompt rewrite, a new tool, an incident, or a spike in complaints. Between those, a weekly pass over 10-20 traces keeps you honest without eating the sprint.

My agent passes everything. Is that good?

It means your suite is too easy. A 100% pass rate carries no information about where the system is fragile. Go find twenty harder failures and add those.

P.S. If you want the single best primary source on this, read Anthropic's Demystifying evals for AI agents - it's the densest practical guidance published on the topic.

Don't miss out on future posts and exclusive content—subscribe to my free newsletter today.

Ready to connect or explore more? Head over to my LinkedIn profile