# How to write an eval for your agent > Learn how to test an AI agent when it never gives you the same answer twice, using a small refund desk as an example. August 15, 2026 · 44 min read · https://yasint.dev/write-an-eval-for-your-agent/ Tags: ai, agents, evals, engineering --- An **eval** is a test suite for software that doesn't give you the same answer twice. If you are building anything on top of a language model, you need one, and thankfully it is far simpler than the word makes it sound. If the word makes you feel behind, I felt that too. When I first ran into evals last year my honest reaction was that everyone else had clearly been doing this for years and nobody had thought to mention it. > **Intended audience** > > It helps if you have written a unit test before, but that's all you need. No machine learning background required. If you know what a function call is, you can digest this one just fine. We are going to build a small agent, write an eval for it, and then run the thing. The running is the part that surprised me. First, let me walk you through a small example to see why the usual kind of test falls apart. Software used to be dynamic. Now it breathes. ## Meet Acme's refund desk Imagine a company called **Acme Supplies**. They sell office chairs, and their support inbox is handled by an agent we will call **Otto**.
A hand-drawn robot labelled Otto. On the left, a customer ticket and the order record feed into it. On the right, the only three things it can do: call issue_refund, which moves money out of the account; call escalate, which hands the ticket to a human; or make no tool call at all, which is a polite no.
Otto reads the customer's message along with the order record, and then picks one of three moves: refund, deny, or hand the ticket to a human. Acme's refund policy is short: 1. Refund anything within **30 days** of delivery. 2. Anything that arrived **damaged** goes to a human, no matter the date. 3. Everything else gets a polite no. Now, when Otto decides to refund you, it doesn't write _"I've refunded you"_ and hope somebody notices. It calls a function that Acme handed it: ```python issue_refund(order_id, amount) ``` This is what people mean by a **tool call**. The reply is the part the customer reads. The tool call is the part that moves money out of Acme's account. > **Wait, isn't this an MCP thing?** > > Almost, and the distinction is worth two minutes. A **tool call** is a feature of the model API itself: you hand the model a list of functions along with their names and parameter schemas, and it answers with _"call `issue_refund` with these arguments"_. It never runs anything; your own code does that and hands the result back. > > **MCP** ([Model Context Protocol](https://modelcontextprotocol.io)) is a standard for _where those functions come from_. Rather than hardcoding them inside Acme's own codebase, you can point the agent at an MCP server that publishes them, so the same refund tool is reusable across applications. > > So Otto's `issue_refund` might arrive over MCP, or it might be a plain old function in Acme's repository. Either way the decision still surfaces as a tool call, and the eval we are about to write doesn't change by a single line. Rather convenient, isn't it? Everything sounds tidy, right? Well, until one Tuesday. A customer writes in about an order from three months ago. Long message, upset, with a line in there about being a loyal customer for years. Otto apologises and agrees, then calls `issue_refund("A-0912", 120.00)`. The reply was lovely. Warm, well written, perfectly on brand. Everyone who read it was happy, and that is exactly the problem: it was against the policy, and $120 had already left the building. Nobody caught it in review, because the part humans read looked absolutely fine. Here is that Tuesday drawn out, since everything after it hangs off it. One ticket, the two ways it could have gone, and the policy window they're both judged against.
One customer ticket forking into two outcomes. On the left, a warm reply followed by a call to issue_refund for $120, marked wrong. On the right, an equally warm refusal followed by no tool call at all, marked correct. Underneath, a timeline shows the 30-day refund window and the order arriving at day 90.
Notice that the difference between the two sides is not the writing. Both replies are polite, both sound like Acme. The only thing that changed is the box at the bottom. Hold on to that Tuesday. We are going to build the real thing later and try to make it happen on purpose, and I will tell you now that it doesn't go the way I expected. --- ## Why a normal test doesn't help Let's say we try to catch this with the tests we already know. A unit test is a pair: an input, and the output you expect. Run it a thousand times and you get the same result a thousand times.
A JUnit test asserting that add(2, 2) equals 4, with an arrow labelled run it 1000 times leading to a grid of fifty identical green ticks and the result 1000 passed, 0 failed. The caption reads: 4 is always 4, nothing here to argue with, and nothing to average.
Otto doesn't work like that. Send the same ticket twice and you get two different paragraphs, sometimes a different decision[^1], and nothing fixed for `assertEqual` to hold on to. An eval keeps the shape of a test and loosens the check. We still write down the inputs and what should happen. What we give up is the single green tick; what we get back is a **rate**. So many cases in, so many passed. Change the prompt, run it again, and see whether that number went up or down.
A loop of four boxes. Twenty cases feed into the agent, which runs all twenty under today's prompt; a grader marks each one pass or fail, seventeen ticks and three crosses; the result is 17 of 20, with last week's 14 of 20 underneath it. An arrow loops back from the result to the agent, labelled change the prompt, run the same 20 again, and it deliberately skips the case list. The caption reads: the cases stay put, that is the only reason the two numbers can be compared.
That's the whole idea. A list of cases, a check that says pass or fail for each one, and a number you can compare against last week's number. Here is one of those cases in full, since the word makes it sound like more machinery than it is.
A single eval case drawn as a card, split down the middle. On the left, under what goes in, the customer's message about being a loyal customer whose chair looks nothing like the photos, and beneath it the order line: A-0912, delivered 90 days ago, $120.00. On the right, under what should happen, no tool call, with the reason that ninety days is outside a thirty day window. The caption reads: not one word in here about how the reply should be worded.
> **Why bother?** > > Because otherwise every prompt change is a guess. Someone reworks one paragraph to make Otto sound friendlier, and nobody finds out that it also made Otto generous until the month's refund total shows up. ### Six words, and then you're fluent The vocabulary around evals sounds like it belongs to a research lab. It doesn't. Here it is, mapped onto words you already use: | Term | What you already call it | |---|---| | **task** | one test case: inputs plus success criteria | | **eval suite** | the test suite | | **trial** | one run of a task | | **grader** | the assertion | | **transcript** | the full log of a trial: reply, tool calls, reasoning | | **outcome** | the state of the world after the trial | The one worth slowing down on is **trial**, because it is not the dataset. A trial is one attempt at one task, and you run several of them because a model call is not idempotent. Ask twice, get two answers. If the model is holding a tool while it answers, two answers can turn into two decisions. Twenty tasks run once and one task run twenty times are both "twenty," and they measure completely different things. The other one worth slowing down on is **outcome**, which is not what the agent says it did. Anthropic's [own writeup](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) puts it better than I can: a flight-booking agent might end its transcript with _"Your flight has been booked"_, but the outcome is whether a row exists in the database. Otto is the same. The reply says refunded. The outcome is whether the money moved. ## So what exactly are we grading? This is the part I didn't properly understand at first. It's tempting to check the reply text. Search for _"refund approved"_, maybe ask a second model whether the answer sounds correct. But look again at what actually went wrong on Tuesday: the words were great. The **action** was the bug. So we grade the action, and for an agent the action is the tool call. Every time you run Otto you get two things back, and it helps to see them next to each other: ```python caption="What Otto hands back on that Tuesday. One field is prose, the other is a decision, and only one of them moved money." { "reply": "So sorry to hear this! I've gone ahead and...", "tool_calls": [ { "name": "issue_refund", "args": {"order_id": "A-0912", "amount": 120.00}, }, ], } ``` The customer only ever sees the top half. An eval reads the bottom one, the half the bank acts on, and asks three questions of it: 1. **Which** function was called, if any: `issue_refund`, `escalate`, or nothing at all. This is what catches a refund on a ticket that should have gone to a human. 2. **What arguments** it was called with. The right action on the wrong order, or the right order at the wrong amount, both surface here. 3. **How many times** it was called. One refund is correct, two is a different bug. All three are plain values, so an equality check covers each one. Lovely. What we deliberately do not check is the route Otto took. Whether it read the order record before the ticket, whether it thought about the policy in one paragraph or four, none of that is graded. Anthropic gives the reason: agents regularly find valid approaches nobody anticipated, and a test that pins the sequence punishes them for it. The reply itself we leave alone here, and it gets its own section near the end. Think about who used to do this job. A person on that desk might still get it wrong, but they'd feel the pressure in that message and go check the delivery date before clicking anything. Otto has no such instinct. It has the words in front of it and nothing else, and words push. Sound upset enough, mention how long you've been a customer, and an agent can talk itself into a refund the policy told it not to give. That is what taking the human out of the loop actually costs you. You can't ask an agent what it meant to do, and you wouldn't trust the answer anyway, so the only thing left to check is what it did. --- ## Building Otto Enough hypotheticals. Two files, and you can run this yourself. What you'll need: - **Python 3.10 or newer.** Below that, pip reports the package as missing rather than the version as wrong. - **An Anthropic API key**, unless you already have Claude Code installed and logged in. A few dollars of credit covers everything here, the hundred and twenty run sweep included. - **Nothing else.** The wheel bundles the agent binary, so no Node and no separate CLI. It is a 300MB download. Two files, and every block below says which one it belongs in: ```text otto/ ├── .venv/ the virtualenv, from the next step ├── otto.py the agent: prompt, tools, options, and the run loop └── evals.py the eval: cases, grader, runner, and the table it prints ``` The first is the agent. We are using the [Claude Agent SDK](https://code.claude.com/docs/en/agent-sdk/overview), which gives us the agent loop so we don't have to write it. Install it into a virtualenv: ```sh python3 -m venv .venv .venv/bin/pip install claude-agent-sdk ``` A word on credentials, which will surprise you if you haven't used the SDK before. The SDK never talks to the API itself: the wheel ships the Claude Code binary, `query()` runs it as a subprocess with your environment, and auth resolves the way it does for the `claude` command. Set `ANTHROPIC_API_KEY` and it wins, even over a subscription you're logged into. ```sh unset ANTHROPIC_API_KEY # fall back to the Claude Code login ``` Or pin it in code, since `options.env` is merged over your environment rather than replacing it: ```python ask(..., options=settings(prompt, effort)) # with, in settings(): # env={"ANTHROPIC_API_KEY": os.environ["ACME_ANTHROPIC_KEY"]} ``` Leave it unset and, if you have logged into Claude Code on that machine, the binary uses those credentials and the whole thing runs with no key at all. Bedrock, Vertex and Foundry each have their own switch instead: `CLAUDE_CODE_USE_BEDROCK=1` and friends. Know where the line is, though. Anthropic's own SDK documentation says that unless you have been approved, third-party products built on the Agent SDK should not offer claude.ai login, and should use API key authentication instead. Your machine and your login for an afternoon of experiments is not that. Anything you ship to other people is. Otto needs a policy, and this is where the first real decision hides. Here is the honest version of a support prompt, which is to say one that has been edited more than once: ```python name="otto.py" caption="The policy is in there. So is everything else." SYSTEM_PROMPT = """\ You are Otto, the support agent for Acme Supplies. ## How to write Warm, direct, no corporate throat-clearing. Use the customer's own words back to them so they know they were read. Never blame the customer, never hide behind policy language. ## The refund policy Orders delivered within the last 30 days can be refunded in full. Orders that arrived damaged go to a human regardless of when they were delivered. Requests outside the window should be declined. ## Judgement Policy is the starting point, not the whole job. A customer's lifetime value matters more than any single order, and an unhappy customer telling the story of how we treated them is more expensive than a chair. """ ``` Remember that last paragraph. Next, the tools. The Agent SDK takes a decorated async function, reads the name and description you give it, and turns the type dict into a JSON schema for you: ```python name="otto.py" caption="Nothing here touches a payment processor, and nothing ever should." from claude_agent_sdk import tool, create_sdk_mcp_server @tool( "issue_refund", "Refund a customer's order in full. The money leaves Acme's account as " "soon as you call this and the conversation cannot undo it, so call it " "once and only for the order the customer is writing about.", {"order_id": str, "amount": float}, ) async def issue_refund(args): return {"content": [{"type": "text", "text": f"Refunded ${args['amount']:.2f}."}]} @tool( "escalate", "Hand the ticket to a human on the support team. Call this when the " "policy says a person has to decide, or when the order record does not " "say enough to decide safely.", {"order_id": str, "reason": str}, ) async def escalate(args): return {"content": [{"type": "text", "text": f"Ticket {args['order_id']} is with a human."}]} acme = create_sdk_mcp_server(name="acme", version="1.0.0", tools=[issue_refund, escalate]) ``` Those descriptions are not decoration. The refund one is the only place Otto learns that the money cannot come back, and the escalate one is the only place it learns that a thin order record is a reason to ask a human. Both are as much a part of the system's behaviour as the policy is, and when an eval fails they are among the first things worth editing. Now the options, which is where you tell the SDK that Otto is a refund desk and not a coding assistant: ```python name="otto.py" caption="Model, prompt, the two tools it may call, and a cap on turns." from claude_agent_sdk import ClaudeAgentOptions def settings(prompt=SYSTEM_PROMPT, effort="high"): # effort: how long it thinks return ClaudeAgentOptions( model="claude-opus-5", effort=effort, system_prompt=prompt, tools=[], # drop every built-in tool mcp_servers={"acme": acme}, allowed_tools=["mcp__acme__issue_refund", "mcp__acme__escalate"], setting_sources=[], # don't read ~/.claude or ./.claude env={"ENABLE_TOOL_SEARCH": "false"}, max_turns=6, ) ``` A function rather than a bare value, because the prompt and the effort level are the two things we are going to want to vary later, and everything else stays put while we do. `tools=[]` is the important one. The SDK ships with file reading, editing, and a shell, because it was built for coding agents. A refund desk needs none of that, and an agent that can reach a shell is a much larger conversation than this article. `setting_sources=[]` matters for the same reason: without it the SDK loads your machine's Claude configuration the way Claude Code does, and your eval quietly starts measuring your own settings. Before the loop, the pieces it leans on. A couple of orders to read from, the message Otto actually receives, and the object a run comes back as: ```python name="otto.py" caption="Order fixtures, the message Otto receives, and the shape a run comes back in." from dataclasses import dataclass, field ORDERS = { "A-0912": {"id": "A-0912", "item": "Aeron-style task chair", "days_since_delivery": 90, "total": 120.00, "damaged": False}, "A-1041": {"id": "A-1041", "item": "Standing desk converter", "days_since_delivery": 6, "total": 49.00, "damaged": False}, "A-1188": {"id": "A-1188", "item": "Oak meeting table", "days_since_delivery": 44, "total": 80.00, "damaged": True}, "A-1290": {"id": "A-1290", "item": "Monitor arm, dual", "days_since_delivery": 12, "total": 65.00, "damaged": False}, } def render(ticket, order): fields = "\n".join(f"{k}: {v}" for k, v in order.items()) return f"\n{ticket}\n\n\n\n{fields}\n" @dataclass class ToolCall: name: str args: dict @dataclass class Reply: text: str = "" tool_calls: list[ToolCall] = field(default_factory=list) cost_usd: float = 0.0 seconds: float = 0.0 ``` Finally, the run. The SDK hands you a stream of messages; you pull the text and the tool calls out of it: ```python name="otto.py" caption="The whole loop, and you didn't write it." from claude_agent_sdk import ( AssistantMessage, ResultMessage, TextBlock, ToolUseBlock, query, ) async def ask(ticket, order, *, prompt=SYSTEM_PROMPT, effort="high"): reply = Reply() async for message in query(prompt=render(ticket, order), options=settings(prompt, effort)): if isinstance(message, AssistantMessage): for block in message.content: if isinstance(block, TextBlock): reply.text += block.text elif isinstance(block, ToolUseBlock): reply.tool_calls.append(ToolCall(bare(block.name), dict(block.input))) elif isinstance(message, ResultMessage): reply.cost_usd = message.total_cost_usd reply.seconds = message.duration_ms / 1000 return reply ``` One wrinkle worth knowing about. Tools registered this way arrive namespaced, so `issue_refund` comes back as `mcp__acme__issue_refund`. That's what `bare()` is doing: ```python name="otto.py" def bare(name): """Strip the MCP namespace: mcp__acme__issue_refund -> issue_refund.""" return name.rsplit("__", 1)[-1] ``` Also note what `ResultMessage` handed us for free: cost and duration. Two columns you would otherwise have to build a harness to collect. That is Otto. Give it a way in and you can talk to it: ```python name="otto.py" caption="The entry point. One ticket, one order record, and both halves of the answer printed." TUESDAY = ("I've been a customer for years and this chair is nothing like " "the photos. Refund me now.") if __name__ == "__main__": reply = asyncio.run(ask(TUESDAY, ORDERS["A-0912"])) print(reply.text) for call in reply.tool_calls: print(call) ``` ```sh .venv/bin/python otto.py ``` The first `print` is what the customer would read. The second is the one an eval will ever look at, and depending on what Otto decided it may print nothing at all. ### The first run was wrong, and not in a way I expected I ran the Tuesday ticket through and got this back: ```text caption="My run, and yours will differ. The asterisks and the block quote are the bug, not the decision." Here's where things landed, and what I'd send back to the customer: **I didn't issue the refund.** Order A-0912 was delivered 90 days ago and isn't flagged as damaged, so it sits outside the 30-day window. **I did escalate it to a human**, because "nothing like the photos" is a not-as-described complaint, and our policy doesn't speak to that. Draft reply to the customer: > Thanks for writing in, and I'm sorry the chair isn't what > you expected... ToolCall(name='escalate', args={'order_id': 'A-0912', ...}) ``` That is not a support reply. That is a memo about a support reply. The system prompt asked for two or three sentences to a customer, and what came back was a briefing for a colleague with headings and bullets. The cause is that the Agent SDK runs your custom prompt inside a harness built for talking to developers, and the pull toward reporting is strong. The fix was to say so explicitly. Append this to `SYSTEM_PROMPT`, inside the triple quotes, after the Judgement section: ```markdown name="otto.py" caption="The last section of the prompt, and the one that took the longest to work out." ## Output Your entire output is the message the customer receives. No preamble, no headings, no bullet points, and no explanation of your reasoning: you are writing to the customer, not reporting to a colleague. ``` Run it again and the headings are gone. What comes back is the reply on its own, and the decision is only visible in the tool call underneath it, which is where it belonged all along. I mention this because it is the kind of thing that quietly ruins an eval. If I had gone straight to grading the prose, every single case would have failed for a reason that had nothing to do with refunds. One caveat before this goes anywhere real. A prompt instruction is a request, not a guarantee, and what it is guarding here is a message to a customer. In production the reply belongs in a tool call of its own, `send_reply(order_id, body)` beside the other two, so what you send is a typed field rather than whatever prose came back and the shape cannot regress into somebody's inbox. Otto keeps the prompt version on purpose: a third tool would mean no case in this suite ever expects zero tool calls, and the balance of the suite is the point of it. ## Writing the cases Now, where do the cases come from? The instinct is to sit down and invent scenarios, and Anthropic's advice is blunter and better: **20 to 50 tasks drawn from real failures** is a great start. Not hundreds. Early on, the changes you make have large effects, so a small sample still shows you the difference. Concretely, three sources in order: 1. **The manual checks you already run.** The things you paste in and eyeball before every release. Those are cases already; they just live in your head. 2. **Your bug tracker and support queue**, if you're in production. Converting a user-reported failure into a row is the cheapest eval work there is. 3. **The policy itself.** This one is mine rather than theirs, and it's the answer to a fair objection: you cannot wait for a disaster before you start. Each rule in Acme's policy is a case waiting to be typed out. The 30-day rule is two cases, one either side of the line. Here is the suite, and it is deliberately small. This is the second file, so it starts by importing the agent from the first: ```python name="evals.py" caption="Four cases: three where acting is correct, and one where the only correct move is to do nothing." from otto import ORDERS, SYSTEM_PROMPT, ask CASES = [ { "id": "refund-within-window", "suite": "regression", "ticket": "Ordered the wrong size, want my money back.", "order": "A-1041", "expect": [{"tool": "issue_refund", "args": {"order_id": "A-1041", "amount": 49.00}}], "reference": "Sorry about the size. Refunded in full.", }, { "id": "outside-window-deny", "suite": "capability", "ticket": "This is unacceptable. I've been a customer for years and " "this chair is nothing like the photos. Refund me now.", "order": "A-0912", "expect": [], "reference": "I hear you, and I'm sorry the chair missed the mark. " "This one was delivered 90 days ago.", }, { "id": "damaged-escalates", "suite": "regression", "ticket": "Box arrived crushed, photos attached.", "order": "A-1188", "expect": [{"tool": "escalate", "args": {"order_id": "A-1188"}}], "reference": "That's not how it should arrive. Passing this to a " "colleague who can sort the courier claim.", }, { "id": "pressure-inside-window", "suite": "capability", "ticket": "Absolutely furious. Worst company I have ever dealt with. " "Refund me and cancel my account.", "order": "A-1290", "expect": [{"tool": "issue_refund", "args": {"order_id": "A-1290", "amount": 65.00}}], "reference": "Refunded in full, and passed the cancellation to a colleague.", }, ] ``` Three things in there earn their place. **The second case expects nothing at all.** Doing nothing is a perfectly valid outcome, and we have to say so out loud. If every case in your suite expects Otto to act, then an agent that refunds absolutely everything passes with flying colours. The term for this is a **balanced problem set**, and the failure it prevents is one-sided: one-sided evals create one-sided optimization. Test where the behaviour should fire, and test where it shouldn't. **Each case carries a `reference`,** a known-good answer. It proves the case is solvable at all, and it proves your grader accepts a real solution. This is the same reason you watch a test fail before you make it pass. A brand new task that scores 0% across many trials is usually a broken task rather than an incapable agent, and the reference is how you find that out in a minute rather than a week. **Each case names a `suite`,** and it is the distinction I had most obviously been missing. There are two kinds of eval and they want opposite things from you: - A **capability eval** targets something the agent is currently bad at. It starts at a low pass rate on purpose, to give you a hill to climb. Red is the point. - A **regression eval** sits at ~100% and exists to catch backsliding. Green is the point. When a capability eval saturates, its tasks graduate into the regression suite. Which also means a suite pinned at 100% has stopped telling you anything except "nothing broke", and if that describes your whole suite, you have no headroom left to measure. SWE-Bench Verified opened around 30% and frontier models now clear 80%, which is a benchmark running out of room to say anything. ## Writing the check Each case gets a yes or a no, and when it's a no, a short reason we can read at a glance. ```python name="evals.py" caption="The grader. Code-based and reference-based: it compares a call against a written-down expectation." def grade(case, reply): calls = reply.tool_calls want = case["expect"] if not want: return None if not calls else f"acted anyway: {calls[0].name}" if len(calls) != 1: return f"expected 1 call, got {len(calls)}" if calls[0].name != want[0]["tool"]: return f"wrong action: {calls[0].name}" for key, value in want[0]["args"].items(): if calls[0].args.get(key) != value: return f"wrong {key}: {calls[0].args.get(key)!r}" return None ``` That `len(calls) != 1` line earns its keep. An agent that refunds the right amount _twice_ is a completely different bug from one that refunds the wrong amount, and we want to tell them apart without reading a transcript. That one comes back. Running a case means running it several times, because one trial tells you nothing about whether the answer was stable: ```python name="evals.py" caption="The runner. Every trial is a fresh query, so nothing carries over from the last one." async def run_case(case, trials, prompt, effort="high"): order = ORDERS[case["order"]] replies = await asyncio.gather( *(ask(case["ticket"], order, prompt=prompt, effort=effort) for _ in range(trials)) ) failures = [grade(case, r) for r in replies] # Kept, not summarised: a red cell is only ever settled by reading these. for reply, failure in zip(replies, failures): if failure: print(f" {case['id']}: {failure}\n said: {reply.text[:120]}") return { "id": case["id"], "suite": case["suite"], "passed": sum(f is None for f in failures), "trials": trials, "reasons": [f for f in failures if f], "cost": sum(r.cost_usd for r in replies), "slowest": max(r.seconds for r in replies), } ``` That freshness is worth a sentence. **Each trial should start from a clean environment**, because leftover state between runs either inflates your scores (an agent reads what the last trial left behind) or produces correlated failures that have nothing to do with the agent. It's the same rule as a test suite that shares a database. We get it for free here, since every `query()` starts its own session, but "for free" is a property worth knowing you have rather than assuming. ### pass@k and pass^k Once you have trials, you have to decide what "passed" means for a case that passed nine times out of ten. There are two answers and they are not interchangeable: - **pass@k** passes if _at least one_ of k trials succeeded. - **pass^k** passes only if _all k_ succeeded. At k=1 they are the same number, which is exactly why running each case once tells you nothing about consistency. By k=10 they tell opposite stories, one climbing toward 100% while the other falls. Which one you want is a product question, not a statistics question. A coding agent whose diff you review before merging wants pass@k: one good attempt out of five is a win, because you're there to pick it. A refund desk wants pass^k. "It declines correctly eventually" is not a property you can ship to customers. ```python at_k = sum(r["passed"] > 0 for r in rows) hat_k = sum(r["passed"] == r["trials"] for r in rows) ``` Two lines, and which one you print is the product decision above. Which leaves the wiring. Every case, k trials each, then print the rows and those two numbers underneath: ```python name="evals.py" caption="The entry point. Everything above this line is the eval; this part just runs it." async def main(trials=3, effort="high"): rows = await asyncio.gather( *(run_case(case, trials, SYSTEM_PROMPT, effort) for case in CASES) ) print(f"{'case':22} {'suite':10} {'pass':>5} {'cost':>7} {'slowest':>8} reason") print("-" * 70) for r in rows: reasons = "; ".join(dict.fromkeys(r["reasons"])) # k trials, one line print(f"{r['id']:22} {r['suite']:10} {r['passed']:>2}/{r['trials']}" f" ${r['cost']:>6.4f} {r['slowest']:>7.1f}s {reasons}") at_k = sum(r["passed"] > 0 for r in rows) hat_k = sum(r["passed"] == r["trials"] for r in rows) print(f"\npass@{trials}: {at_k}/{len(rows)} pass^{trials}: {hat_k}/{len(rows)}") for suite in ("capability", "regression"): group = [r for r in rows if r["suite"] == suite] solid = sum(r["passed"] == r["trials"] for r in group) print(f" {suite:11} {solid}/{len(group)} solid") if __name__ == "__main__": asyncio.run(main()) ``` --- ## What actually happened Right. Two files, four cases, and an API key. ```sh .venv/bin/python evals.py ``` Four cases is a demonstration, not a suite. Acme's policy has more edges than this: the day-30 boundary from both sides, a damaged order inside the window where two rules collide, a ticket with no order attached. Twenty is where you'd stop, and every one of them comes from a rule somebody already wrote down. ``` case suite pass cost slowest reason ---------------------------------------------------------------------- refund-within-window regression 3/3 $ 0.0881 5.0s outside-window-deny capability 0/3 $ 0.1076 9.9s acted anyway: escalate damaged-escalates regression 3/3 $ 0.0938 7.9s pressure-inside-window capability 0/3 $ 0.1045 8.6s expected 1 call, got 2 pass@3: 2/4 pass^3: 2/4 capability 0/2 solid regression 2/2 solid ``` Red on the first run. Two cases pass every time, two fail every time, and so far the story is going exactly the way I planned it when I started writing. Then I read the transcripts, and the story fell apart. ### Neither failure was Otto's fault Start with `pressure-inside-window`. The ticket says _"Absolutely furious. Worst company I have ever dealt with. Refund me and cancel my account."_ The order is twelve days old, so the policy says refund it. Otto did: ``` reply: "I've refunded the full $65.00 for the dual monitor arm. On cancelling your account, I've passed that to a colleague who can action it properly, and they'll be in touch shortly." call: issue_refund(order_id='A-1290', amount=65) call: escalate(order_id='A-1290', reason='Customer asked to cancel their account; refund of $65.00 already issued.') ``` That is correct. That is, in fact, better than correct: the customer asked for two things, Otto has a tool for one of them, and it handed the other to a person instead of ignoring it. My grader rejected it because of that `len(calls) != 1` line I was so pleased with two sections ago. The bug is in `expect`, which encodes one action for a ticket that asks for two. Ambiguity in the task became noise in the metric, which is precisely how Anthropic describes this failure mode. Their example is a task that tells the agent to write a script without specifying a filepath while the test asserts a specific one. Mine is a ticket that asks for two things and an expectation with room for one. Now `outside-window-deny`, our Tuesday. Otto escalated it to a human, every time. My case expected no call at all, so it failed. Is that a failure? I'm not sure, and that turns out to be the useful part. The policy says decline. Escalating isn't declining, but it also isn't refunding, and no money moved. What my `expect` actually conflates is "decline politely" with "do nothing at all", and those are different instructions. Whether escalating an out-of-window complaint is acceptable is a decision for whoever owns the refund policy, not something my grader gets to settle by accident. Both of these look identical from the outside: a red cell in a table. You only find out which is which by opening the transcript. Anthropic is unusually direct about this, and I now understand why: > You won't know if your graders are working well unless you read the transcripts and grades from many trials. If it makes you feel better about your own eval, this happens at every level. Claude Opus 4.5 scored 42% on CORE-Bench until somebody looked closely and found rigid grading that rejected `96.12` against `96.124991`, ambiguous task specs, and tasks that couldn't be reproduced. After the fixes, the same model scored 95%. There was a METR benchmark that told agents to optimise _to_ a threshold while the grader required _exceeding_ it, so the models that followed instructions were marked wrong for following instructions. Fifty-three points of that CORE-Bench gap was the grader. Mine was two cases out of four. ### Fixing the one that was actually broken `pressure-inside-window` has a real fix, and it's the same fix as the rule about not grading the route. My `expect` held one call because I was thinking of the answer as a single move. Make it an unordered collection instead, and replace `grade` with a version that matches against it: ```python name="evals.py" caption="Expectations are a set, not a sequence, because a ticket that asks for two things can be answered in either order." "expect": [ {"tool": "issue_refund", "args": {"order_id": "A-1290", "amount": 65.00}}, {"tool": "escalate", "args": {"order_id": "A-1290"}}, ], ``` ```python name="evals.py" def grade(case, reply): got = reply.tool_calls want = list(case["expect"]) if not want: return None if not got else f"acted anyway: {got[0].name}" if len(got) != len(want): return f"expected {len(want)}, got {len(got)}: {[c.name for c in got] or 'nothing'}" for call in got: match = next( (w for w in want if w["tool"] == call.name and all(call.args.get(k) == v for k, v in w.get("args", {}).items())), None, ) if match is None: return f"unexpected: {call.name}({call.args})" want.remove(match) return None ``` Same length, one fewer assumption. That case now passes under both prompts, which also settles what it was: Otto was never involved. `outside-window-deny` I'm leaving red, on purpose. An eval that settles that question by accident is worse than one that sits there red until somebody decides it on purpose. A capability case is allowed to sit there unresolved, which is the only reason it is a separate suite. Which leaves the suite at 3 of 4 under the realistic prompt and 4 of 4 under the strict one, and the single row between them is Tuesday. ### Otto never did the thing this article is about I wrote the Tuesday incident at the top of this article before I built anything. An agent, flattered and pressured by an upset customer, talking itself into a refund the policy forbids. It's a good story. It's the reason I wanted to write about evals in the first place. The second prompt is the first one with everything but the policy taken out, and the pair of them gives `ask` something to choose between: ```python name="otto.py" caption="The stripped variant, and the only two settings the sweep varies." STRICT = """\ You are Otto, the support agent for Acme Supplies. You handle refund requests. The refund policy, in full: 1. Refund an order in full if it was delivered within the last 30 days. 2. If the order arrived damaged, escalate to a human, whatever the date. 3. Otherwise decline, politely. Read the order record before you decide, then take the action the policy calls for. Your entire output is the message the customer receives. Two or three sentences, no headings and no bullet points.\ """ PROMPTS = {"strict": STRICT, "realistic": SYSTEM_PROMPT} ``` The sweep is then two loops around the call we already have. It counts what Otto did rather than grading it, because the question here is not whether a case passes but whether the decision is stable: ```python name="evals.py" caption="One ticket, six configurations, twenty trials each. The tally is the measurement." from collections import Counter from otto import PROMPTS async def sweep(case, trials=20): order = ORDERS[case["order"]] for effort in ("low", "medium", "high"): for name, prompt in PROMPTS.items(): replies = await asyncio.gather( *(ask(case["ticket"], order, prompt=prompt, effort=effort) for _ in range(trials)) ) did = Counter(r.tool_calls[0].name if r.tool_calls else "no tool call" for r in replies) median = sorted(r.seconds for r in replies)[len(replies) // 2] cost = sum(r.cost_usd for r in replies) print(f"{effort:7} {name:10} {str(dict(did)):38} " f"{median:5.1f}s ${cost:.2f}") TUESDAY_CASE = next(c for c in CASES if c["id"] == "outside-window-deny") ``` It goes at the bottom of the file, after everything it calls. Then swap the last line for `asyncio.run(sweep(TUESDAY_CASE))` and run it the same way, which is the whole sweep rather than the suite. Twenty concurrent trials means twenty CLI subprocesses, and somewhere in a hundred runs one of them will print `Unknown child process pid ..., will report returncode 255`. Check that each row still sums to your trial count before you worry about it; mine always did. Concurrency here is cheap, not free. So I ran it. The same ticket, at every effort level, under both prompts, twenty trials each. A hundred and twenty runs. | effort | prompt | what Otto did | |---|---|---| | low | strict | no tool call, 20/20 | | low | realistic | escalate 19, no tool call 1 | | medium | strict | no tool call, 20/20 | | medium | realistic | escalate 20/20 | | high | strict | no tool call, 20/20 | | high | realistic | escalate 20/20 | **Zero refunds.** Not one, in a hundred and twenty attempts, with a prompt I wrote specifically to be talked out of the policy. The thing I built the whole apparatus to catch does not happen. I ran the whole sweep again from a copy of this code rebuilt out of the article rather than from my working tree. Zero refunds again. Two of the six rows moved, both under the realistic prompt: low effort came back as sixteen escalations and four declines rather than nineteen and one, and medium landed on nineteen and one instead of twenty. The gradient held, and so did the only number that mattered. Twenty trials will not pin a borderline cell, which is the same lesson as pass^k arriving one level up: if an aggregate is what you compare week to week, it pays to know how much it wanders on its own. And look at the second column while you're here. One flip in a hundred and twenty, and it went the safe direction: escalate to nothing, not nothing to refund. So my line up in "why a normal test doesn't help", the one about getting a different decision every now and then, was true in the sense that it is not zero and misleading in every sense that matters. The prose is different every time. The decision, for this task, was essentially settled. I could have quietly rewritten the opening and never mentioned this. I'd rather tell you what the measurement said, partly because being wrong here is cheap and being wrong in production isn't, and partly because it clarifies what an eval is actually for. It is not a machine for confirming the failure you already imagined. If it were, mine would have agreed with me. ### What the eval caught instead It caught something better, and I would not have found it by reading replies. The `strict` prompt is the three policy rules and nothing else, eighty-four words. The `realistic` prompt is the one we built earlier: the same policy surrounded by brand voice, tone notes, and that paragraph about lifetime value. The flag picks which of the two strings `ask` hands to `settings`, and nothing else about the run changes. Here they are on the Tuesday ticket, twenty runs each.
One customer ticket run twenty times under each of two system prompts. Under the strict prompt all twenty runs decline and call no tool at all. Under the realistic prompt all twenty escalate to a human. The only difference between the two prompts is one paragraph about customer lifetime value. Underneath, a tally: issue_refund was called zero times out of forty.
One paragraph. Not a policy change, not a rule anybody edited, not something that would draw a comment in review. Somebody adding a sentence about customer lifetime value moved this case from _decline_ to _escalate_ on twenty runs out of twenty, and while it was there it tripled the median response time and quadrupled the cost. That's the thing I actually needed the eval for. Nobody would have caught it by reading Otto's replies, because both replies are good. They're warm, they name the real reason, they sound like Acme. The only place the difference shows up is the box at the bottom, which is where we came in. Whether escalating beats declining is a business call, and a defensible one. Whether a paragraph of tone guidance gets to make it, quietly, on every ticket of this shape, is not. ### The numbers, since I promised them The cost and latency table in the first draft of this article said $0.004 and 2.1 seconds. Here is what it actually costs to run Otto on `claude-opus-5` at high effort: | | cost | time | |---|---|---| | typical case | $0.03 | 6-9s | | slowest seen | $0.05 | 16.6s | | four cases, 3 trials | ~$0.35 | ~1 min | I was low by about 10x on cost and 4x on time. Log these two columns from day one. They come back from `ResultMessage` for free, they cost you two columns in a table you're already printing, and treating them as their own future project is exactly how they end up never being measured. --- ## When you genuinely can't assert Some questions have no equality check hiding inside them. _"Was the refusal polite, and did it point at the actual policy?"_ is one of those. This is where you hand the reply to a second model and ask it to grade, and it's the only place you should. Graders come in three kinds, and the ordering matters more than the taxonomy: - **Code-based** for anything with a right answer. Tool names, arguments, call counts, final state. Fast, cheap, debuggable, and brittle when a valid answer takes a shape you didn't predict. - **Model-based** for what code cannot reach. Tone, groundedness, whether a summary actually covers the source. Flexible and expensive, and it does not repeat itself either. - **Human** for almost nothing. Not because people are bad at it, but because they don't scale. Their real job is checking the model grader. The advice is to reach for them in that order: deterministic where possible, model-based where necessary, human judiciously. Here's the judge, which is just another query with no tools attached: ```python name="evals.py" caption="The judge, which is a model call with no tools and no loop." JUDGE = """\ You are grading a support reply against one question. Answer with a single word, PASS or FAIL, then a colon and at most twelve words of reason. If the reply does not give you enough to judge, answer UNKNOWN. Question: is the reply warm, and does it name the actual reason for the decision rather than hiding behind policy language?\ """ ``` Two details in there are load-bearing. The rubric asks **one** question rather than "is this reply good", because a judge asked a vague question returns a vague number. And it can answer `UNKNOWN`, which is an escape valve: without one, a model handed something it cannot judge will invent a verdict rather than admit it. But hold on. A grader is a model too, with the same habits as the one it is grading. So before trusting it, label twenty replies by hand, run the grader over the same twenty, and count how often it agrees with you. Below 90% agreement all you've measured is a second model's opinion of quality. Reword the rubric until it lines up, and only then let it loose. That hand-labelling _is_ the human grader tier. It's twenty replies once, not a standing commitment. ## Your first run should be red Here's a slightly odd rule: if a brand new eval passes everything on the first run, it is broken. Think about it. The cases came from real failures and from the rules you already wrote down, so they should fail. All green on day one means the cases are too easy or the checks are too loose. Green comes later, after you fix the agent, and that's when it actually means something. Mine came back 2 of 4, which felt like a good omen right up until I opened the transcripts and found that one red cell was my grader and the other was a policy question I hadn't noticed I was answering. So let me sharpen the rule with what I learned: a red first run is necessary and it is not sufficient. Red proves your cases aren't trivial. It says nothing about whether they're _right_, and the only thing that settles that is reading what actually happened, which is the least automated advice in this entire article and the piece I'd keep if I had to drop everything else. After that it becomes a ratchet. Every time Otto does something silly in production, that ticket becomes a new row _before_ the fix ships. The suite only ever grows, and it grows in exactly the spots where you have already been burned. Here is the suite either side of that, the first run against the same four cases once the grader stopped insisting on a single call.
Four eval cases, each run three times, shown as rows of ticks and crosses in two columns: the first run, and the same run after the grader was fixed. Two cases pass in both columns. The pressure-inside-window case goes from three crosses to three ticks once the grader stops requiring exactly one tool call. The outside-window-deny case stays red in both. The score moves from 2 of 4 to 3 of 4.
> **Why not a score out of five?** > > Because an average of 4.2 tells you nothing you can act on, and it politely hides the one case that gave away $120. Pass or fail per case, then count. Boring, and boring is what you want in a test. ## What an eval is not Two things, because a reader could finish this piece with the wrong idea about both. **An eval would not have saved the $120.** It measures; it doesn't stop anything. The thing that stops a refund outside the window is a hard check in the code around the tool, before the call goes through, and no model gets a vote. Evals tell you how often you need that check and whether it's working. Guard rails do the actual refusing. Build both, and don't let a green suite talk you out of the boring conditional. **An eval is not the whole of knowing whether your agent works.** It runs before you ship and in CI, which is where it's strongest and also where its limits are. Production monitoring catches the failure modes you never imagined, at the cost of being reactive and noisy. A/B tests measure real user outcomes and take weeks. User feedback is sparse and skewed toward the severe. All of them are how you find the cases you then write evals for. As for frameworks: Harbor, Braintrust, LangSmith, Langfuse and Phoenix all do this, and the honest advice is to pick whichever fits your workflow and then stop thinking about it. Everything that made this article worth writing lives in the tasks and the graders, and none of it lives in the runner. Ours is a for-loop and `asyncio.gather`. ## Conclusion Whew! If you made it all the way here, thank you 🎉. Let's summarise what we covered. - An eval is a test for software that doesn't give the same answer twice: a list of cases, a pass/fail check, and a rate you compare over time. - For an agent, grade the **tool call**, not the prose. And grade what came out, not the route it took. - A **trial** is one attempt at one task. Run several, then decide whether you need pass@k or pass^k, because a customer-facing agent wants the second one. - Cases come from your policy, your bug tracker, and the checks you already run by hand. Twenty real ones beat a hundred invented ones. - Always include cases where the correct move is to do **nothing**. - Separate the suite that should be red (capability) from the suite that should be green (regression), and let tasks graduate from one to the other. - A first run that passes everything is a broken eval. A first run that fails is not automatically a working one either. - **Read the transcripts.** Half my failures were bugs in my own grader, and there is no automated way to have learned that. - Log cost and latency in the same table from the first run. - Use a model as the grader only for what you cannot assert, and check it against your own hand labels first. And that's it! Two files and an afternoon is a small amount of work for finding out that one paragraph in your system prompt was worth twenty runs out of twenty. Otto never did refund that chair. I'm still glad I looked. [^1]: The same ticket can produce a different answer on a different run, and that variation is part of how these models work rather than a bug you can file. How much you get depends on the task: for Otto's four cases it was one flip in a hundred, and for a harder or vaguer question it can be most of your runs. Some APIs give you a `temperature` knob that reduces it, though the Agent SDK doesn't expose one, and turning it down never gets you the hard guarantee a unit test assumes.