CCAR-P · module 4 of 7 · 16% of the exam

Evaluation, testing, and optimization

Covers the 6 official objectives: define evaluation metrics; design eval datasets/test frameworks (mixed methodologies); conduct A/B testing and iteration; diagnose system issues (prompt failure/hallucination/model mismatch); optimize token/latency/cost trade-offs; monitor via logging/observability.

Official sample theme shared with Domain 3: RAG retrieval as first suspect after a document refresh — a diagnosis question, not a design question, in this domain's framing.

Objective: Define evaluation metrics (accuracy, latency, cost, safety, security)

• Core concept: "Is this good?" must be decomposed into named, measurable metrics before you can test or optimize anything — a single fuzzy "quality" score hides which dimension actually failed.
• Anthropic-platform specifics:
  • Accuracy — task-specific correctness (exact match, rubric-graded, or LLM-judge scored). For Managed Agents, Outcomes (user.define_outcome + a gradeable rubric) operationalize this: an independent grader model scores each iteration against explicit, independently-gradeable criteria and the harness iterates until satisfied or max_iterations.
  • Latency — time-to-first-token and total completion time; measured per model tier/effort combination, not assumed constant across configurations.
  • Cost — driven by usage.input_tokens/output_tokens/cache_*_tokens at the model's per-token price; cache hit rate is a first-class cost metric, not just a latency one.
  • Safety — refusal rate, harmful-output rate on adversarial inputs; distinct from security.
  • Security — resistance to prompt injection, tool-misuse, and unauthorized data access; tested via adversarial/red-team-style eval sets, not covered by a standard accuracy eval.
• Decision heuristic: A scenario naming a specific failure ("responses are correct but slow," "responses are fast but wrong") is telling you which metric to optimize — don't propose a fix that improves a metric the scenario didn't flag as broken (e.g., don't raise effort/model tier to fix a pure latency complaint).

Objective: Design evaluation datasets and test frameworks using mixed methodologies

• Core concept: No single eval methodology covers every failure mode — combine deterministic checks, rubric/LLM-judge grading, and human review depending on what's being measured.
• Anthropic-platform specifics:
  • Deterministic/exact-match — for tasks with a verifiable correct answer (structured extraction validated against a schema, classification against a labeled set). Cheapest and fastest to run at scale.
  • Rubric-graded (LLM-judge) — for open-ended output (summaries, analysis, generated documents) where exact match doesn't apply. This is the same mechanism as Managed Agents Outcomes: explicit, independently-gradeable criteria in a rubric, scored by a separate grader context.
  • Human review — for the highest-stakes or most subjective judgments, and as a calibration check on the LLM-judge itself (does the automated grader agree with human graders on a sample?).
  • Eval datasets should be sourced from real production failures wherever possible (not only synthetic cases) — synthetic-only eval sets systematically miss the failure modes that actually occur.
  • Mixing methodologies also means mixing eval timing: pre-deployment (offline eval suite) and post-deployment (production monitoring/online eval) — a framework limited to pre-deployment checks misses regressions introduced by data drift or model updates after launch.
• Decision heuristic: For a subjective/open-ended output type, proposing pure exact-match testing is a mismatch (it can't grade prose quality); for a task with one correct structured answer, proposing an expensive LLM-judge when deterministic validation would suffice is unnecessary cost/complexity — match methodology to task shape.

Objective: Conduct A/B testing and iterative improvements

• Core concept: Changes to prompts, models, or effort levels should be validated against a held-out eval set / live traffic split before full rollout, not shipped on the strength of a few manual spot-checks.
• Anthropic-platform specifics:
  • A/B comparisons must hold everything except the variable under test constant — e.g., comparing Sonnet vs. Opus while also changing the prompt confounds the result; isolate one variable per test.
  • Effort-level sweeps (low/medium/high/xhigh/max) are a first-class A/B dimension on current models — measure the actual intelligence/latency/cost curve for your workload rather than assuming a monotonic "higher is always better" relationship (diminishing returns and occasional overthinking are documented at the top end).
  • Iterative improvement loops should feed eval failures back into the prompt/tool design (the feedback-loop stage of the architecture from Domain 1) — A/B testing without a mechanism to act on the losing/winning signal isn't complete iteration.
• Decision heuristic: "We changed the model and the prompt at the same time and accuracy went up — ship it" is a confounded test, a common distractor; the best-answer response isolates variables before attributing the improvement.

Objective: Diagnose system issues (prompt failure, hallucinations, model mismatch)

• Core concept: Different failure symptoms point to different root causes and different fixes — matching symptom to cause correctly is the core "best-answer" skill tested here.
• Anthropic-platform specifics:
  • Prompt failure (ambiguous instructions, missing constraints, conflicting guidance) → symptom is inconsistent or off-target output even though the model had the right information; fix is prompt clarity, not a bigger model.
  • Hallucination (model states something false/unsupported) → check first whether the answer should have come from retrieved context (RAG) that either wasn't retrieved or was ignored, before assuming it's an inherent model-reliability issue; citations and structured outputs reduce ungrounded hallucination on document-based tasks.
  • Model mismatch (task exceeds the chosen tier's capability, e.g., complex multi-step reasoning on Haiku) → symptom is a capability ceiling that no amount of prompting fixes; fix is a model-tier or effort upgrade, not more prompt engineering.
  • RAG retrieval as first suspect after a document refresh — official sample theme. When "the agent gives outdated/wrong info about something that was recently updated," the diagnostic order is: (1) was the source re-indexed? (2) is retrieval actually surfacing the new chunk? — before concluding it's a hallucination or a model-capability problem. Jumping straight to "the model is hallucinating" or "switch to a better model" skips the far more likely and far cheaper root cause.
• Decision heuristic: Build the diagnostic order into your answer selection: verify data/retrieval freshness → check prompt clarity/constraints → only then consider model-capability mismatch. A distractor that proposes a model upgrade for what is actually a stale-index problem is the domain's signature trap.

Objective: Optimize token usage, latency, and cost-performance trade-offs

• Core concept: Optimization levers are ordered by where the waste actually is — fix the biggest, cheapest win first (usually caching or right-sizing the model) before reaching for structural rewrites.
• Anthropic-platform specifics:
  • Prompt caching is almost always the first lever for token-cost and latency on any repeated-context workload — verify cache_read_input_tokens is actually non-zero before assuming caching is "already handled."
  • Model right-sizing — routing simple sub-tasks to Haiku and reserving Opus/Fable-tier for the hard sub-tasks (a workflow "routing" pattern from Domain 1) often beats using one model tier for everything.
  • Effort tuning — dropping from xhigh/max to medium on tasks that don't need the extra depth is a direct token/latency reduction with measurable, testable impact (A/B it, don't assume).
  • Batches API (50% cost) for non-latency-sensitive bulk workloads is a distinct lever from caching/model choice — applies when the SLA tolerates async delivery.
  • Context pruning (context editing / compaction) reduces token spend on long-running agents by not re-sending stale, no-longer-relevant tool results on every turn.
• Decision heuristic: When a scenario presents a cost/latency problem, check for a caching miss or an over-provisioned model tier before proposing a structural change (rewriting the architecture, switching integration protocol) — the cheapest fix is usually a configuration fix, not a redesign.

Objective: Monitor system performance using logging and observability tools

• Core concept: Continuous production monitoring closes the loop between the eval framework (Domain 4) and the architecture's feedback loop (Domain 1) — evals validate before ship; observability validates after ship.
• Anthropic-platform specifics:
  • usage fields (token counts, cache stats) and _request_id on every response are the baseline structured signal to log for every call, not just failures.
  • Managed Agents session traces (Console URL per session) and the event stream (span.model_request_end.model_usage, error events) are the built-in agent-level observability surface — prefer them over hand-rolled logging for agentic workloads.
  • Webhooks for state-change monitoring (session.status_idled, vault_credential.refresh_failed, etc.) let you monitor at scale without polling every session.
  • Monitoring should feed back into the eval dataset — production failures caught by monitoring become new eval cases (mixed-methodology principle from earlier in this domain), closing the loop.
• Decision heuristic: "How do we catch regressions after a prompt/model change ships" → production monitoring + webhook/logging-driven alerting feeding new eval cases, not just periodic manual spot-checks.

How to think through the question

Domain 4 questions are mostly diagnosis under a stated symptom. The scenario tells you which metric is broken, and the wrong answers improve a different one.

Signal words to look for

Reasoning procedure

1. Name the metric the scenario says is broken, and immediately discard any option that improves a different one.
2. For a diagnosis question, walk the order deliberately: is the data or retrieval fresh and actually reaching the model, is the prompt clear and constrained, and only then is the model tier the limit. Answer with the earliest step that has not been ruled out.
3. For a testing question, check that exactly one variable moved, that the methodology matches the output shape, and that the result was confirmed on data the change was not developed against.
4. For a cost or latency question, look for a configuration fix first, meaning caching, model right sizing, effort tuning, batching, or context pruning, before endorsing any structural rewrite.
5. Ask whether the framework spans both before and after deployment, and whether production failures flow back into the eval set.
6. Ask what the reported number could be hiding, which usually means slices, subgroups, or a grader that changed.

Worked example

A company refreshed its product documentation last week. Since then, a support assistant has been confidently telling customers about a return window that no longer exists. A proposal on the table is to move to a more capable model, since the current one is evidently unreliable.

Read the timing first. The failure began immediately after a content change, which puts the content pipeline at the top of the suspect list and pushes model reliability to the bottom.

Now walk the diagnostic order rather than the option list. Step one is whether the source was actually re-indexed, and step two is whether retrieval is surfacing and passing through the new text. Only if both hold does the question become one about prompting or capability. The proposed model upgrade is the most expensive option available and would not place the current policy text in front of the model, so it cannot fix the described failure even if it improved the system in general. The answer is to verify re-indexing and retrieval first, and note that a confident wrong answer sourced from a stale index is not a hallucination in any useful sense, it is a retrieval defect wearing a hallucination's clothes.

Exam traps (Domain 4)

• Model-upgrade-first diagnosis: jumping to "switch to a more capable model" for a symptom that's actually a stale-RAG-index or prompt-clarity problem — the domain's signature distractor, sharing the official RAG-retrieval theme with Domain 3.
• Confounded A/B tests: changing model and prompt simultaneously, then attributing the delta to either one.
• Wrong methodology for output shape: exact-match grading for open-ended prose, or an expensive LLM-judge for a task with one deterministically-checkable answer.
• Ignoring the stated failure metric: proposing an accuracy-maximizing fix (bigger model, more effort) for a scenario that specifically flagged a latency or cost problem, or vice versa.
• Structural rewrite before the cheap fix: redesigning the pipeline when the actual issue is a caching miss or an over-provisioned model tier.
• Pre-deployment-only eval framework: an eval suite with no production/online monitoring component, missing post-launch drift or regressions.