CCAR-P · module 5 of 7 · 14% of the exam

Governance, safety, and risk management

Covers the 5 official objectives: implement guardrails/safety controls; identify risks/limitations/failure modes; apply human-in-the-loop validation; ensure regulatory compliance (GDPR/HIPAA/FedRAMP); address ethical AI considerations (bias, fairness, transparency).

Objective: Implement guardrails and safety controls

• Core concept: Guardrails should be layered and structural where possible — prose instructions alone are the weakest layer and should not be the only control for anything high-stakes.
• Anthropic-platform specifics:
  • System-prompt-level guardrails (explicit behavioral constraints) — necessary but not sufficient; current models follow instructions closely, so calibrated, specific guardrail language works better than vague "be careful" or over-aggressive "CRITICAL: NEVER..." phrasing (which tends to overtrigger or underspecify).
  • Tool-level permission_policy — always_ask forces human confirmation before a risky tool executes; this is a structural control independent of prompt wording, and can't be bypassed by a cleverly-crafted user input the way a prose instruction sometimes can.
  • Structured outputs / strict schemas — constrain the shape of output at the API level (output_config.format, strict: true tool schemas), preventing malformed or off-contract responses from reaching downstream systems regardless of what the model "decided" to say.
  • Refusal handling — stop_reason: "refusal" is a built-in safety signal from Anthropic's classifiers (most relevant on frontier/high-capability models for cyber/bio-adjacent content); application code must branch on stop_reason before reading content, and should treat a refusal as a governance event to log, not silently retry with a jailbreak-style rephrase.
  • Least-privilege tool configuration (see Domain 3) is itself a safety control, not just a security one — fewer capabilities means fewer ways a guardrail failure can cause real-world harm.
• Decision heuristic: For any control protecting against a high-consequence action (financial transaction, data deletion, PII exposure), the correct answer layers a structural control (permission policy, schema validation) on top of prompt-level guidance — a prompt-only answer is the plausible-but-inferior choice.

Objective: Identify risks, limitations, and failure modes of LLM systems

• Core concept: LLM systems fail in known, nameable ways — an architect should be able to identify which failure mode a scenario describes and pick the mitigation matched to it, not a generic "test more."
• Anthropic-platform specifics:
  • Hallucination — confident but false/unsupported output; mitigated by grounding (RAG + citations), not by prompting the model to "be more careful."
  • Prompt injection — untrusted content (a fetched web page, a tool result, a user-uploaded document) contains instructions that hijack the agent's behavior. Mitigated by least-privilege tools, always_ask on consequential actions, and treating retrieved/tool content as data, not instructions, in the system prompt framing.
  • Non-determinism / inconsistency — same input, different outputs across runs; relevant to compliance/audit contexts where reproducibility matters — mitigate with structured outputs, lower effort where appropriate, and logging every response for audit trail rather than expecting bit-identical repeatability (removed sampling params like temperature=0 never guaranteed determinism even on older models).
  • Capability limitations — a model tier genuinely cannot do the requested reasoning; distinct from a fixable prompt/retrieval problem (see Domain 4 diagnostic order) — the fix is a tier/effort upgrade, and if no tier can do it reliably, the correct governance answer is to add a human-in-the-loop check, not to keep prompting.
  • Context/window limitations — degraded quality or dropped information at very long context; mitigated by compaction/context editing/decomposition (Domains 1-2), not by assuming the model "remembers everything perfectly" at 1M tokens.
• Decision heuristic: Match the failure mode named/implied in the scenario to its specific mitigation — a security-flavored failure (injection) calls for a security control (least privilege, always_ask), not a prompting fix; a capability-flavored failure calls for HITL or a model upgrade, not more guardrail prose.

Objective: Apply human-in-the-loop validation strategies

• Core concept: HITL is a deliberate architectural checkpoint, sized to the actual risk of the action — not a blanket "always ask a human" (which defeats the purpose of automation) nor "never ask" (which removes the safety net for consequential actions).
• Anthropic-platform specifics:
  • permission_policy: {type: "always_ask"} on specific tools (not the whole toolset) is the standard mechanism — gate the consequential subset (e.g., bash, a send_email custom tool, a payment API) while leaving read-only/low-risk tools (read, grep, web_search) on always_allow so the agent isn't blocked on trivial steps.
  • Custom tools naturally support HITL by gating inside the tool's handler (return a "pending approval"/"declined" result) before performing the real side effect, or in the manual/tool-runner loop by inspecting a pending call and overriding it before execution.
  • Managed Agents cross-posts subagent tool-confirmation requests to the primary thread — so a HITL design in a multiagent system only needs one human-facing approval surface, not one per subagent.
  • HITL should be reserved for genuinely hard-to-reverse or high-consequence actions (the same reversibility criterion used in Domain 1's tool-design guidance) — over-applying it to every tool call creates approval fatigue and defeats automation value (a productivity/business-value trade-off from Domain 1).
• Decision heuristic: "Which actions in this scenario need a human checkpoint" → the hard-to-reverse or high-stakes ones specifically (deletion, payment, external communication, PII disclosure) — a scenario proposing HITL on every single tool call, or on none of them, is likely describing the plausible-but-inferior extreme rather than the calibrated answer.

Objective: Ensure compliance with regulations (e.g., GDPR, HIPAA, FedRAMP)

• Core concept: Regulatory compliance constraints shape where data can go, how long it's retained, and what must be auditable — architecture decisions (data retention config, hosting/platform choice, logging) must be made consciously against the applicable regime, not bolted on after the fact.
• Anthropic-platform specifics:
  • Data retention is a configurable org-level setting; some capabilities have minimum retention requirements (e.g., a frontier model requiring at least 30-day retention will hard-fail under a zero-data-retention configuration) — a compliance-driven ZDR requirement can directly conflict with a feature requiring longer retention, and the architect must resolve that conflict explicitly (e.g., pick a different model/feature, or negotiate the retention policy) rather than ignore the mismatch.
  • FedRAMP / government workloads — point toward the platform offering that meets the compliance boundary (e.g., a cloud-provider-hosted first-party offering with the provider's own compliance certifications) rather than assuming the standard consumer API endpoint automatically satisfies a specific certification; per-feature availability differs by platform (some server tools, Managed Agents, Batches, and Files are unavailable on some third-party platforms) — verify feature support against the compliance-required platform before designing around it.
  • HIPAA (healthcare/PII) — drives requirements around what data is sent to the model at all (redaction/minimization before the call), audit logging of every access, and often HITL validation before any PHI-adjacent action is taken.
  • GDPR (EU personal data) — drives data residency/locality expectations, right-to-erasure implications for any persisted memory/logs containing personal data, and minimization principles similar to HIPAA's.
  • Memory stores and session event history persist data (including anything written into them) — a compliance-relevant design point: never write secrets, and be deliberate about what personal data is allowed into a persisted memory store versus an ephemeral session, since redaction/deletion of a memory version is possible (memory_versions.redact) but is a remediation, not a preventive control.
• Decision heuristic: A scenario naming a specific regulation is telling you to check retention config, data minimization, platform/hosting choice, and audit logging against that regime's requirements — a generic "add more guardrails" answer that doesn't address the specific compliance lever named (retention, residency, audit trail) is the weaker choice.

Objective: Address ethical AI considerations (bias, fairness, transparency)

• Core concept: Ethical AI risk shows up as unequal treatment across groups (bias/fairness) or as opacity about how a decision was reached (transparency) — both are testable, not just aspirational.
• Anthropic-platform specifics:
  • Bias/fairness — evaluate outcomes across relevant subgroups in the eval dataset (Domain 4's mixed-methodology principle applied to fairness specifically), not just aggregate accuracy; a system can hit a high overall accuracy number while systematically underperforming for a subgroup, which aggregate metrics hide.
  • Transparency — citations (tracing an answer back to its source document/location) and structured outputs with explicit rationale fields are architectural transparency mechanisms; for agentic systems, the session/event trace itself (Managed Agents Console trace, or logged tool-call history) is what makes a decision auditable after the fact.
  • Human oversight as an ethical control, not just a safety one — HITL on consequential decisions (loan approval, hiring screen, medical triage support) is often required for both risk-management and ethical-AI reasons simultaneously; a scenario invoking a high-stakes decision about people is very likely testing whether you reach for HITL + transparency together, not either alone.
• Decision heuristic: When a scenario involves a decision that materially affects people (approval/denial, resource allocation, risk scoring), the best answer combines subgroup-aware evaluation, transparency (traceable rationale), and human oversight — a technically-accurate-only answer that ignores fairness evaluation or auditability is incomplete for this domain.

How to think through the question

Domain 5 questions describe a risk and offer four controls. The decisive question is almost always whether the proposed control is structural or merely worded, and whether it is matched to the failure mode the scenario actually names.

Signal words to look for

Reasoning procedure

1. Name the failure mode: hallucination, prompt injection, non determinism, capability ceiling, or context limitation. Each has a matched mitigation, and mismatching them is the domain's most common wrong answer.
2. Ask whether each candidate control is structural or prose. For anything hard to reverse, require a structural control layered on top of the prose rather than choosing between them.
3. Size human review to reversibility and consequence. Both extremes, gating everything and gating nothing, are usually the distractors.
4. When a regulation is named, go straight to the lever that regulation implies rather than to a generic "add more guardrails" answer, and check for a conflict between a compliance posture and a capability that has its own requirements.
5. When the decision materially affects a person, expect the best answer to combine three things: subgroup aware evaluation, a traceable rationale, and human oversight. An answer with only one of the three is incomplete for this domain.
6. Before settling, ask what happens when the control fails. If the answer is nothing catches it, keep looking.

Worked example

A customer service agent can issue refunds up to a set limit. The only control is a system prompt instruction stating that it must never issue a refund on an unverified account. The team asks whether this is sufficient before launch.

Name the exposure first: issuing money is hard to reverse and the consequence is direct, so this sits at the top of the risk scale.

Now classify the control. It is a single prose instruction, which is the weakest layer available, and it can be pushed on by a crafted or unusual input in a way a structural control cannot. Notice the question is not whether the instruction is well written, because a better worded instruction is still one layer of the same kind.

The structural options are a human confirmation policy on the refund tool so a person approves before it executes, verification enforced inside the tool handler rather than requested of the model, and a strict schema so the tool refuses inputs that do not carry the verification result. The best answer keeps the prompt guidance and adds at least one of these, since layering is the point. An answer that only rewrites the prompt more forcefully, or that gates every tool including read only lookups, is the plausible but inferior choice on either side of the calibrated one.

Exam traps (Domain 5)

• Prose-only guardrails for high-consequence actions: relying solely on system-prompt wording ("never do X") instead of a structural control (always_ask, schema validation, least privilege) for anything hard to reverse.
• Wrong mitigation for the failure mode: applying a prompting fix to a security failure (injection) or a security fix to a pure capability-ceiling failure.
• HITL at the extremes: gating every single action (kills automation value) or gating nothing (removes the safety net) instead of calibrating to reversibility/consequence.
• Retention/compliance mismatch ignored: choosing a feature or model with a minimum retention requirement while operating under a stricter (e.g., zero-data-retention) compliance posture, without flagging or resolving the conflict.
• Assuming platform parity: assuming a compliance-required hosting platform supports the same feature set as the standard API without checking per-feature availability.
• Aggregate-metric-only fairness claims: citing a single overall accuracy number as evidence of fairness without subgroup-level evaluation.
• Auditability treated as optional: proposing an agentic design for a high-stakes decision with no traceable rationale/logging, relying only on the final output being "correct."