CCAO-F · module 1 of 7 · 21% of the exam

Output evaluation and validation

Weight: 21 percent of the exam. This domain tests whether you can look at something Claude produced and decide, with a repeatable method, what to trust, what to check, what to fix, and what must not leave the building without an expert signature.

What the exam expects

This is the largest domain on the CCAO-F exam, and it is the one that separates people who use Claude from people who are accountable for Claude's output. The exam is not testing whether you know that AI can be wrong. It is testing whether you can name which part of a given output is likely wrong, why it is the risky part, and what specific check would catch it.

Five capabilities show up repeatedly:

1. Detect the failure. Distinguish a fabricated fact from an incomplete summary from a biased framing from an internally inconsistent document. These are different failures with different fixes.
2. Choose the right check. Match the verification technique to the claim type. Checking a quotation against an uploaded PDF is a different operation from checking a market statistic against a published report.
3. Decide who reviews. Know when your own review is sufficient, when a peer is needed, and when a qualified subject matter expert must sign off before anything ships.
4. Iterate effectively. Turn dissatisfaction into a specific instruction rather than regenerating and hoping.
5. Choose the output form. Decide when a result should be inline text, a table, an artifact, or a diagram, and understand that format is part of whether the output is usable and checkable.

Questions are scenario driven. You will be given a marketing manager, a compliance officer, a teacher, or an operations lead who has an output in hand, and asked what they should do next.

The four evaluation dimensions

Almost every Domain 1 question is secretly asking which dimension has failed. Learn to sort outputs along these four axes, because a fix aimed at the wrong axis is the most common wrong answer on the exam.

The trap the exam loves: an output that passes accuracy and fails completeness. Every sentence checks out, and an entire topic is missing. You cannot catch that by reading the output carefully. You catch it only by going back to the source and asking what should have been there.

Where errors concentrate

Not all parts of an output carry equal risk. Errors cluster in a predictable set of places, and knowing the cluster is what lets you verify efficiently under time pressure.

Highest risk, always check:

• Specific numbers. Percentages, totals, dollar amounts, counts, thresholds. Numbers stated in prose are especially suspect, because prose arithmetic is reasoned about rather than computed. If a total matters, compute it in a spreadsheet or a file rather than accepting it from a paragraph.
• Named sources. Citations, report titles, book titles, author attributions, case names, standard numbers. A fabricated reference blends a real author with a plausible title, which is exactly what makes it hard to spot and easy to propagate.
• Quotations. Text presented in quotation marks. Search the source for the exact wording. A paraphrase presented as a quote is a real and common failure.
• Time sensitive facts. Interest rates, prices, org charts, product capabilities, regulations, conference dates. Anything that changes is a candidate for being recalled correctly as of training and being wrong today. Nothing in the phrasing distinguishes a current fact from a stale one.
• Causal claims. "X drove the change in Y" goes beyond what most business data can establish, and it will be repeated in meetings as though it were measured.

Lower risk, usually fine:

• Structure, organisation, and phrasing
• Restating something you supplied in the prompt
• Suggested next actions and recommendations, which are judgments to accept or reject rather than facts to verify

A note on grounding. Uploading a document to a project, using a connector, or running research mode all substantially reduce fabrication, because the model has real material to work from. None of them guarantee that every sentence comes from a source. General knowledge and inference can still enter the output. The practical control is to require a pointer: ask which document or section supports a claim, and treat any claim with no locatable source as unverified.

Verification techniques that actually work

And three that feel like verification but are not:

• Asking Claude whether it is sure. Self reported confidence is not calibrated to whether a fact exists. You will get a plausible answer in either direction.
• Running it twice and comparing. Two runs of the same model are not independent observations. Shared tendencies produce shared errors, so agreement is weak evidence.
• Adding a disclaimer. A note that content was AI assisted discloses risk without reducing it, and the author remains accountable for what they distribute.

When a human must review, and which human

Review effort should scale with three things: consequence of error, checkability of the claims, and whether the producer has the expertise to spot a mistake. Notably, none of those three is "which model produced it." Model choice moves the error rate somewhat; it does not move the cost of being wrong, which is what a review policy exists to manage.

Subject matter expert sign off is mandatory, not advisory, for: medical and clinical content, legal advice and contractual language, financial figures and filings, and anything with a safety dimension. In these domains, a non expert's sense that the output "reads accurate" is not review. The exam will offer you tempting substitutes: a peer read for clarity, a self check by Claude against published guidelines, a disclaimer footer. All three are wrong when the correct answer is that a qualified professional signs off.

A second reviewer is warranted when output reaches customers, regulators, or the public; when it commits the organisation to dates, pricing, or scope; or when the producer cannot independently evaluate the domain content.

Self review is sufficient when the output is internal, exploratory, low consequence, and the producer can recognise an error in it.

Iterating on outputs

When an output is not right, the exam's correct answer is nearly always specific, targeted feedback, and nearly never regenerate and hope.

Effective iteration names three things: what is wrong, what right looks like, and what to trade away to make room. "Underweight the methodological debate, expand the two competing frameworks to roughly equal length, and compress the historical background to a paragraph" converges in one turn. "Make it better" and "try harder" convey no information, so the revision is a guess.

Two related patterns worth recognising:

• Leading premises. If your prompt asserts the conclusion ("our churn rose because of the pricing change, explain the link"), the output will build the case for it. Ask what the data supports, or ask for competing explanations.
• Authorship signals. If you present work as your own and ask for weaknesses, you will tend to get a gentle response. Ask instead for the strongest objections a hostile expert reviewer would raise.

Choosing the output format

Format is an evaluation concern because the right form makes an output both usable and checkable.

Artifacts on Claude.ai can be published (Free, Pro, and Max, generating a public link) or shared within an organisation (Team and Enterprise, where viewers must be signed in). That distinction matters for governance, but for Domain 1 the point is simpler: an artifact is the right surface when a deliverable will be revised repeatedly and handed off.

How to think through the question

Domain 1 questions are constructed the same way. Someone has an output, something is subtly off, and four plausible actions are offered. Work the problem in this order.

Step 1: Identify what the output actually is and where it came from. Was a source document uploaded? Was web search or research on? Is the content grounded, or is it recall? Scenarios name this deliberately. "Web search was not enabled" and "she uploaded the contract" are not scene setting, they are the answer key.

Step 2: Name the failed dimension. Accuracy, completeness, consistency, or audience suitability. Ask specifically: is anything stated wrong, or is something missing, or does the document contradict itself, or is it correct but unusable by its reader? Most wrong answers on this domain are correct actions aimed at a dimension that did not fail.

Step 3: Identify what the question is really asking. "What should they do first?" wants the earliest correct step, not the most complete programme. "What is the root cause?" wants a diagnosis, not a fix. "What is the durable control?" wants a process change, not a one time correction.

Step 4: Eliminate the non checks. Cross off self assessment by the model, regeneration and comparison, disclaimers, tone changes, and settings changes (effort, extended thinking, model tier) offered as if they were verification. These appear in almost every question, and they are almost never the answer to a validation question.

Step 5: Eliminate over and under reactions. "Stop using Claude for this entirely" is nearly always wrong: it removes the tool rather than the gap. So is "accept it, that is what the tool is for."

Step 6: Pick the option that puts evidence or accountable human judgment in the path. The surviving answer is usually the one that either grounds a claim in a source or routes the output past the person qualified to catch the error.

Worked example

A finance analyst asks Claude to produce a quarterly variance commentary from an uploaded spreadsheet export. The narrative is well written, and the totals cited in the prose do not match the totals in the file.

Step 1: The source is in hand and it is a spreadsheet. This is grounded work, so wholesale fabrication is unlikely.

Step 2: Nothing is missing and nothing contradicts itself internally. The prose contradicts the source. This is accuracy, specifically numeric accuracy.

Step 3: The question asks for the interpretation and the response, so both a diagnosis and a fix must be present in the answer.

Step 4: An option proposing a more capable model is a settings change dressed as a control. Eliminate.

Step 5: An option calling the difference rounding waves away a material discrepancy. An option assuming the export is stale inverts the evidence, treating the generated narrative as ground truth and the source file as suspect. Eliminate both.

Step 6: The remaining answer diagnoses the cause (figures were reasoned about in prose rather than computed) and fixes it by moving the arithmetic into the file and re-grounding the commentary on computed values. That is the answer.

Exam traps

• Treating fluency as reliability. A polished, confident, well organised output feels verified. Fluency is a property of the writing and carries no information about whether the content is true. The exam builds scenarios where a "well written and clear" draft is precisely the one with the invented figure.
• Asking Claude to check Claude. Tempting because it is fast and stays in one window. It produces another ungrounded answer rather than independent evidence.
• Consistency across runs read as confirmation. Tempting because reproducibility genuinely is evidence in other contexts. Two runs of the same model share the same tendencies, so a repeated fabrication looks like corroboration.
• A disclaimer treated as a control. Tempting because it feels responsible and is nearly free. It transfers blame rather than reducing error, and the author is still accountable.
• Fixing the dimension that did not fail. Tempting because the offered action is genuinely good practice. Fact checking a summary that is accurate but missing a section will pass every statement and miss the problem entirely.
• Settings offered as verification. Extended thinking, higher effort, and a more capable model all sound like they should improve correctness. They change how the draft is produced, not whether anyone checked it, and a more thorough draft can be confidently wrong in new places.
• Abandoning the tool after one failure. Tempting because it looks appropriately cautious. It is an overcorrection that discards real value; the right answer adds the control.
• Accepting an agreeable critique. When you ask for weaknesses in your own draft and get compliments, the framing produced the answer. The exam tests whether you notice.
• Precision mistaken for rigour. Percentages attached to unverified groupings, and likelihood and impact scores on a risk register the model knows only from a paragraph, look more rigorous than prose and are less trustworthy, because they get carried forward unquestioned.

Quick reference

• Four dimensions: accuracy, completeness, consistency and logic, audience suitability. Name the failed one before choosing a fix.
• Verify first: numbers, named sources, quotations, time sensitive facts, causal claims.
• Grounding reduces fabrication; it does not eliminate it. Require a source pointer and drop what cannot be traced.
• Not verification: model self assessment, regeneration and comparison, disclaimers, tone changes, effort and thinking settings.
• Expert sign off is mandatory for medical, legal, financial, and safety content. No substitute counts.
• Review scales with consequence of error, checkability, and producer expertise. Not with output length or model tier.
• Iterate with specific feedback naming what to change and what to trade away. Never "try harder."
• Avoid leading premises and authorship signals in prompts; both bias the response toward agreement.
• Format for the job: inline for short answers, artifacts for revisable deliverables, tables for comparisons, diagrams for branching processes, computed files for defensible numbers.
• Under time pressure, verify what carries the most consequence and is least checkable by the reader.