CCDV-F · module 8 of 8 · 2.6% of the exam

Eval, testing, and debugging

Weight: 2.6 percent of the exam. Tests whether you can build an evaluation that catches regressions, choose the right grader, diagnose model behaviour from response metadata, and instrument a production system.

What the exam expects

Six questions, focused on engineering discipline rather than statistics. The through-line is that prompts are code and deserve the same treatment: a test suite, a pass criterion, one change at a time, and enough observability to explain a production incident after the fact.

The distractors in this domain cluster into three shapes. There is the manual workaround (more eyeballing, more reviewers, a staging soak). There is the metric that looks like progress but hides the problem (a single aggregate number, a higher threshold on the same metric, a larger denominator). And there is the diagnostic that changes something before understanding anything (upgrade the tier, raise effort, apply four fixes at once).

Building an eval

An eval is a set of inputs with known-good outputs or gradeable criteria, run automatically, with a defined pass threshold agreed before the run.

Four properties matter:

• Coverage over polish. Volume and breadth beat a small hand-curated set. Most production regressions land on inputs nobody thought to check, which is exactly what a broad set catches and a notebook of five favourites does not.
• Held out. Cases used to iterate on the prompt stop being an honest measure of it. Keep a set you do not tune against.
• Edge cases included deliberately. Missing fields, ambiguous inputs, mixed intent, unusually long documents, and the categories your examples under-represent.
• A threshold set in advance. Deciding what counts as a pass after seeing the numbers is how a regression gets rationalized into a ship.

Run the suite on every prompt change, every model change, and every parameter change. That is what makes it a regression test rather than a one-time benchmark: a model upgrade is an input to the system, and the suite runs against the new model, prompt held constant, before any traffic moves. Version prompts alongside code so a result can be attributed to a specific revision.

Source new cases from real production failures first. Every incident, support ticket, and confusing output becomes an eval case, so the suite grows toward the failure modes the system actually has rather than the ones someone imagined. The stable, verified core of the suite is often called a golden dataset; keeping it fixed is what makes this week's score comparable to last week's. To A/B compare two prompt candidates, run both against the same set and compare pass rates. Before a custom harness exists, the Anthropic Console's evaluation tooling covers this loop at low setup: build a suite of test cases, run a prompt against all of them in one pass, grade the outputs, and compare prompt versions side by side on the same cases.

Choosing graders

Match the grader to what is being measured.

A mixed suite is normal and usually correct. Grading an invoice number with an LLM judge is slow, noisy, and expensive when a string comparison is exact. Grading a customer-facing explanation with a substring check is a proxy so weak that a bad explanation containing the right number passes.

Rubric quality determines judge quality. "The summary is good" produces noisy grades. "The summary states the decision, names the responsible team, and introduces no claim absent from the source" produces criteria a judge can score independently and a human can audit.

Judges also carry known biases worth checking for: position bias (favouring whichever answer appears first in a pairwise comparison; swap the order and confirm the verdict follows the answer, not the slot), self-preference (scoring output from their own model family higher), and a tilt toward longer answers. Calibrate the judge against human grades on a sample before trusting its numbers.

Agentic tasks are graded differently from single-turn ones. A single-turn task is graded on its output. An agentic task is graded on its outcome, the end state after the run: the file is correct, the tests pass, exactly one record was written. Transcript matching is brittle because two valid runs can reach the same result by different tool paths, so assert on final state, and add trajectory checks (which tools ran, how many steps, token spend) only where the path itself matters, such as a rule that a destructive tool is never called.

Reading the metric honestly

Aggregate accuracy is the number most likely to mislead. A change can raise overall accuracy while a specific segment gets materially worse, and that segment is where the support tickets come from. If the business cares about a distinction (enterprise versus self-serve, language, document length, customer tier), break the eval down along it. A single headline number cannot show a subgroup regression underneath it, and raising the pass threshold on that same number does not make it visible.

Two related failures worth naming. Denominator manipulation is growing the eval set until the failing cases are a smaller share of the total; the metric improves and the same users keep hitting the same failures. Metric-only confidence is treating a five point gain as proof that a change was an improvement, which forecloses the investigation that would find the regression.

Debugging model behaviour

Start with response metadata before changing anything.

stop_reason is the first thing to read. It distinguishes failure modes that look identical from the outside:

A great deal of confused debugging comes from code that only catches exceptions. A truncated answer, a refusal, and a paused turn are all successful HTTP responses. Server tool errors behave the same way: they arrive as a result block whose content is an error object, not as a raised exception.

Change one variable at a time. When quality is poor on hard cases, the candidate fixes are usually a prompt change, retrieval, more effort, or a different model tier. These address different causes: a missing-fact failure is a retrieval problem, a shallow-reasoning failure is an effort or tier problem, and a misread-instruction failure is a prompt problem. Collect the failing cases, read what actually went wrong in each, then change one thing and measure. Applying all four at once means a gain cannot be attributed and a regression in one is masked by a gain in another.

Observability

Log per request, at minimum:

• The request id, which is what lets Anthropic trace a specific request end to end when you report a problem
• The usage fields: input, output, cache creation, and cache read token counts. These carry both cost attribution and the direct signal for whether prompt caching is working
• The stop_reason, so refusals and truncations are visible in aggregate rather than as scattered anomalies
• The full tool call and result history for agents, which is the ground truth for what the agent actually did

Two anti-patterns. Estimating tokens client side with a third-party tokenizer built for another model family produces systematically wrong numbers when the response already carries exact counts. And logging only the final user-facing response keeps a summary written by the system under investigation, omitting every intermediate action that would explain the incident. For the same reason, asking the model afterwards what it did is not an audit trail, and the raw chain of thought is not available on current models to serve as one.

How to think through the question

Signal words to look for. "Inputs nobody thought to check" points at missing eval coverage. "Cut off mid-sentence" with no exception points at stop_reason. "Overall accuracy holds but one segment complains" points at a subgroup regression under an aggregate. "Changed the prompt and the model" flags a confounded comparison. "Clear and appropriately toned" flags an output only a rubric-graded judge can score. "Track spend" or "debug after the fact" points at logging usage and the request id.

1. Decide whether this is measurement, diagnosis, or instrumentation. Building or grading an eval is measurement. Explaining an observed behaviour is diagnosis. Deciding what to log is instrumentation. The three have different correct-answer shapes.
2. For measurement, ask what the metric cannot see. Subgroups, edge cases, and dimensions outside the metric are where the hidden answer usually lives.
3. For diagnosis, read the metadata before changing anything. stop_reason and usage settle a surprising share of these outright.
4. Eliminate options that change several things at once, and options that scale a manual process rather than replacing it.
5. Eliminate options that improve the number without improving the system. Growing the denominator, raising a threshold on the same aggregate, and reviewing the same five examples more carefully all belong here.

Worked example. A support summarizer's responses started getting cut off mid-sentence after a prompt change that added more context. Error handling only checks for exceptions, and no exception is raised. What should the team inspect first?

Step 1: an observed behaviour needs explaining, so this is diagnosis. Step 3 is the whole answer: mid-sentence truncation on a response that raised no exception is the signature of the max_tokens stop reason, since the request succeeded and the model simply ran out of room, which is consistent with a prompt change that added context and left less headroom. Step 4 and 5 clear the rest: an HTTP client timeout would produce a transport error rather than a well-formed response; sampling parameters such as temperature are rejected on current models and randomness does not truncate output anyway; and malformed XML can confuse the model but does not stop generation at the output cap. Read stop_reason first, then raise max_tokens or shorten the input.

Exam traps

• Confusing a soak period with a test. Running a change in staging for a week generates no signal unless something exercises the edge cases.
• Scaling manual review. More examples reviewed by more engineers keeps missing the same unconsidered inputs.
• One aggregate number as proof. It cannot show a subgroup regression, and raising its threshold does not help.
• Growing the eval set to dilute failures. The metric improves; the product does not.
• LLM judges for objectively checkable facts. Slower, noisier, and more expensive than a comparison.
• A weak proxy grader. "Contains the invoice number" is not a measure of explanation quality.
• Catching only exceptions. Truncation, refusals, paused turns, and server tool errors all arrive as successful responses.
• Changing several variables at once. No attribution, and offsetting effects hide regressions.
• Third-party tokenizers. Wrong for Claude, and unnecessary when usage and the token counting endpoint exist.
• Logging only the final answer. The intermediate trace is what explains an incident.

Quick reference

• Evals need coverage, held-out cases, deliberate edge cases, and a threshold set in advance. Run on every prompt, model, and parameter change.
• Graders: code-based for objective checks, LLM judge with an explicit rubric for open-ended quality, human for calibration and adjudication.
• Break results down by the segments the business cares about; aggregate accuracy hides subgroup regressions.
• Read stop_reason first: end_turn, max_tokens, tool_use, pause_turn, refusal.
• Change one variable at a time; match the fix to the failure category.
• Log the request id, the usage fields, stop_reason, and the full tool trace.
• Use the token counting endpoint rather than a third-party tokenizer.