How we address hallucination in regulatory updates

A hallucination is fluent output that isn’t faithful to a real source. In compliance, that’s a nonstarter: a confident citation to a rule that doesn’t say what the model claims is worse than no answer at all. So before we ask AI to touch a single policy, we constrain it. Three disciplines do the work, and they are the whole method.

This page is the short version. The full write-up, with the research behind each move, is the white paper “A Methodology for Leveraging AI Across Your Policies and Procedures.” Read the PDF → · Word

1. Grounding: work from the actual regulatory text, never from memory

Grounding ties every output to a provided source of truth, so answers stay inside the supplied evidence rather than the model’s recollection. It’s the move the field converged on as retrieval-augmented generation, and it is the single most important defense against hallucination.

For a regulatory update, we don’t hand the AI a PDF of the new rule and hope. We convert the regulation into a structured, machine-readable corpus, a table rather than prose. Text is grouped under a register (one regulation or source), and each row is a single obligation that carries:

• A stable ID (obligation_id) — a permanent handle for that one requirement, so a reference points to a specific, retrievable unit of text, not a vague gesture at “the regulation.”
• The verbatim regulator text (paragraph_quote) — the exact words of the rule, and the only thing the model is permitted to cite.
• A type classifying the obligation’s force: do, don’t, prohibition, refrain, requirement, notice.
• Tags and a section label — subject area and lifecycle markers, such as an effective date.
• A change classification — for an update, a change_action of added / changed / removed, plus what it supersedes.
• Provenance — the citing authority, a source_url, a snapshot_id, and the date it was fetched.

The model is then instructed to work only from these rows: cite a rule only by quoting an actual row, verbatim, tagged with its ID. The change classification is what makes this specific to updates: a rule change is itself structured data the model lines up old-against-new, not something it has to infer. You can see this in practice in the FHA Loss Mitigation update kit, where HUD’s change entries ship linked to the Handbook text they supersede.

2. Verification: gate the output against the source

Grounding and verification are the two ends of one guarantee. Grounding is the front-end constraint, work only from the verified source and quote it. Verification is the back-end check, confirm after the fact that the quote actually matches. A constraint you don’t check is only a hope, so the method runs both.

The check itself is blunt and reliable: a string comparison of the quote the AI returns against the regulatory corpus. The quote is either a literal substring of the verified source or it is not. Paraphrase fails the gate. The durable finding in reliable-AI research is exactly this, that a separate check on a model’s output catches errors the generating pass misses.

And when a claim cannot be grounded, the model is told to take the exit, “outside this kit,” rather than fabricate. An explicit abstention path is a known hallucination reducer. Every change the model proposes is logged with its provision ID and the verbatim text that drove it, so the output isn’t just a revised document but an audit trail an examiner can follow.

3. Context engineering: a scratchpad in every chat

Context engineering is the discipline of deciding what context a model carries between steps. It addresses the stray hallucinations that creep into multi-stage work, the kind that appear when a model holds too much in its head and leaps to a conclusion. Multi-chat accuracy improves markedly when each chat writes its findings down and separate chats with clean context can read them back.

So each P&P under review ends by writing a structured findings record: a typed block, not a prose summary, with each finding tagged by a stable ID, an inherent- and residual-risk rating, and a named owner for every open question. Two things follow. First, the record is inspectable, you see the reasoning, not just the verdict. Second, it hands off: because each chat leaves behind structured knowledge another chat can query, dozens of isolated chats reconcile into one result. A single regulatory change reviewed across every affected P&P, and a master policy synthesized from what each chat found, without re-reading all of them.

Why Skills matter: enforcing all three, every time

Grounding, verification, and context engineering are disciplines. A Skill is what makes every chat in a Project apply them identically, the same way in chat #1 and chat #40. Without it, the discipline lives in a prompt you have to type out or paste perfectly each time.

• Operating rules. Cite verbatim by ID and never assert a fact the document doesn’t support (grounding); apply the substring check and the “outside this kit” exit (verification); end every run with the exact findings block (context engineering).
• The verified rule kit. The grounded corpus itself, a CSV, ships inside the Skill, so every chat reads the same source.
• The output contract. The exact findings block every run must end with, so many chats produce results that merge cleanly.

The result: when a regulatory update lands, you drop it into the Project, each chat decides whether its policy is affected and what must change, and the results reconcile into one coordinated review instead of forty separate errands. The compliance officer sets the rules the Project enforces and reviews what it surfaces, the gate, never the bottleneck, never bypassed.

This page is educational and is not legal advice. Compliance officers make compliance decisions. Independent; not affiliated with or endorsed by Fannie Mae, Anthropic, NIST, or any regulator.