Why a model fills in a detail your document never stated, and how to stop it

You asked for a summary of a policy, and it stated a coverage term as a plain fact. You went back to check, and that term wasn't in the document anywhere. The model filled in a gap with something that sounded exactly right.

Why rewording won't fix this

A model generates the most likely next words given everything it's seen, and when a document is silent on a point, the most likely continuation is often a plausible answer pulled from everything similar the model has encountered before, not a note that the source doesn't cover it. Telling it to only use what's in the document narrows this but doesn't remove it, because the model isn't comparing each claim against the source sentence by sentence unless the prompt forces that comparison directly.

Require a quote, and check that the quote is real

Ask for the exact sentence behind every claim, not a paraphrase of it. A quote can be checked against the source text in code; a paraphrase can drift from what the document actually says without that drift being visible to anyone reading the summary.

A claim with no quote, or a quote that doesn't appear in the source, comes back unverified.

function extractCoverageTerms(policyText) {
  const claims = callModel(buildPrompt(policyText));   // each claim should carry a quote

  return claims.map((claim) => ({
    ...claim,
    verified: claim.quote !== null && policyText.includes(claim.quote),
  }));
}

Worked example: wind damage, and silence on flood

Say a policy explicitly excludes wind damage from a named storm, and simply never mentions flood at all. Asked to summarize what's covered, a model handed only the wind clause can still produce a confident line about flood coverage, built from what flood coverage usually looks like in policies generally, not from anything on this page.

Forced to quote, there's no sentence to point to, and the correct answer becomes 'not addressed in this document' instead of a guess dressed up as a fact.

Silence is itself an answer worth returning

A document that doesn't mention something is giving you information, not withholding it. A system that can say the document doesn't address this is more useful than one that always finds something to say, because the gap is exactly what a person needs to go check elsewhere.

If you're already asking a model to cite a page for every finding, the way the playbook on reading a long document covers, add this one extra check on top: confirm the quote it returns is actually in the source, word for word, before anyone trusts it.

Verify before it reaches a client

An unverified claim should never leave the system labeled the same as a verified one. Route anything without a matching quote to a person instead of sending it forward automatically, the same staged check the playbook on a human approval step describes for any output that needs a second look before it goes anywhere.

What stays manual

Deciding which claims are important enough to need a quote versus general framing text.
Reading the flagged, unverified claims yourself before deciding what to do with them.
Updating the check if the policy template or form changes shape.