Real vs “On-Point” Legal Citations: Can AI Check Support?

Real vs “On-Point” Legal Citations: Can AI Check Support?

Introduction: When a citation is real—but not on point

A citation isn’t just a decoration in legal writing. It’s supposed to signal that the cited authority actually supports the proposition you’re claiming. And that’s exactly where things can go wrong with AI assistance.

New research from the original paper looks at a subtle failure mode: not fake citations that invent nonexistent cases, but misrepresented citations—cases and pinpoint pages that are real, yet the specific page cited doesn’t back up the legal claim being made. This is the kind of mistake that can slip past basic “does this exist?” checks.

The paper also connects to real-world consequences. In 2023, a New York judge sanctioned attorneys in Mata v. Avianca after they filed a brief containing hallucinated citations generated by ChatGPT. Database lookup can catch many of those. But the tougher problem is detecting citations that are real and properly formatted—while still not supporting the statement for which they’re used.

Why This Matters: Proposition-level citechecking is the new weak link

This research is significant right now because citation volume is exploding inside drafting workflows. LLMs can generate a brief in minutes and toss in dozens of citations. Even if a tool checks that cases exist, the attorney still has to verify meaning: “Does page X say what we claim it says?” That’s a different task than “Is this a legitimate source?”

A scenario where this can bite today: imagine a junior associate drafts a section arguing that a procedural requirement is “notice reasonably calculated to apprise interested parties,” and an LLM supplies Mullane with a pinpoint page. The citation looks perfect—correct case, correct reporter, correct Bluebook format. But if the pinpoint page doesn’t contain the support for the specific proposition (maybe it discusses a different factor or a limitation), the argument quietly rests on sand.

How is this different from earlier AI research on legal reliability? A lot of the legal-aid evaluation work has focused on detecting fabricated citations or general factuality problems. This paper argues that once fabrication gets easier to catch, the more consequential failure mode is support verification: distinguishing “topically related” from “actually supports the claim at this exact page.” The core claim is blunt: current models often conflate the two.

What the researchers actually measured: “On point” means page-level support

The paper formalizes the task as: given legal text containing a target citation, and given the content of the cited source, determine whether the citation is on point—meaning the cited authority supports the proposition offered in the surrounding passage.

To make this test measurable, they wrap the citation under review in <CITE>...</CITE> tags and feed the model:
1. the surrounding paragraph (and, for court opinions, preceding context), and
2. the cited content—either the full document or just the “pinpoint” page/range if the citation includes a pinpoint reference.

The model’s output is simply YES/NO (plus a rationale), which lets the researchers compute two key metrics:
- Recall: of the corrupted (wrong) citations, how many the model correctly flags.
- False Positive Rate (FPR): of the valid citations, how many the model incorrectly flags.

The two ways citations can fail: fabrication vs misrepresentation

They split failure into two buckets:
- Fabrication: the cited authority doesn’t exist.
- Misrepresentation: the authority exists, but it doesn’t support the proposition being claimed.

Their focus is the misrepresentation case—especially pinpoint mismatches where the model has every surface cue that a human would usually trust.

Why “topical overlap” is a trap in legal reasoning

Legal writing can involve analogy and doctrine-specific language that overlaps across different propositions. So even when the cited page is wrong, the case-level topic can still “feel” relevant. The researchers found that models frequently rely on that feeling—treating topical coherence as sufficient evidence—rather than verifying what the cited page actually says.

How they built the hardest test: controlled “wrong page” corruptions

The study uses two legal datasets:
- CLERC: court opinions, about ~2,000 citations
- BriefMe: legal briefs, about ~750 citations

Each test example is generated from real citations extracted from these sources, paired with the full text of cited documents.

Three corruption levels (from easy to brutal)

They create “negative” examples by changing a citation in a controlled way, producing three difficulty levels:

The hardest setting is exactly what you’d worry about in real life: the citation “looks right” because it points to the same case and often the same doctrine—just the wrong location.

They verify the corruptions are really “not on point”

The paper doesn’t just assume their corruption strategy worked—they do human validation. Across 23 citations per dataset (mix of on-point and off-point), three legal analysts labeled each as on point or not. Agreement with the heuristic labels was high:
- ~88% for CLERC
- ~84% for BriefMe
(They report additional agreement stats too; the key point is that the corruptions reliably create not-on-point examples.)

Which models passed—and which still miss page-level support

The researchers evaluate 14 frontier model configurations across three model families (including multiple versions and prompt styles). For GPT-5.4, they test both standard inference and a high reasoning effort setting, and they use two prompt styles:
- a baseline prompt
- a page-grounded prompt intervention designed to push models to verify support at the cited page (described later in more detail)

Performance: wrong cases are mostly caught; wrong pinpoints are not

The results show a clear pattern: models do great when the citation points to the wrong case, but struggle when only the pinpoint page is wrong.

On CLERC (court opinions):
- Easy: recall roughly 93–100%
- Medium: recall drops to 73.6–93.0%
- Hard (wrong pinpoint pages): recall 36.5–60.6%

On BriefMe (briefs):
- Easy: recall roughly 93–100%
- Hard: recall 51.5–82.7%

So yes: model capability helps, but there’s a stubborn gap specifically for page-level proposition support.

Bigger/newer models help—but the gap doesn’t vanish

Within the OpenAI family, they report:
- GPT-5 beats GPT-4.1 on Hard examples:
- CLERC: 55.5% vs 38.4% recall
- BriefMe: 77.0% vs 51.5% recall

But improvements aren’t monotonic with “newer” or “more capable.” They note examples where certain newer models didn’t win on Hard recall. For instance:
- GPT-5.4 without reasoning trails GPT-5 on both datasets.
- Claude Opus 4.6 trails its other variants in Hard-example recall.

Recall vs false positives: you can get more catching power, but not cleanly

They emphasize the tradeoff using recall and FPR. If a model gets more aggressive about flagging citations, it can catch more wrong pinpoints but incorrectly reject valid citations too—meaning your citechecking tool becomes less practical.

They also report a specific comparison on CLERC:
- The highest Hard recall among matched models: up to 60.6% baseline recall for GPT-4o, but with 34.9% FPR
- The lowest FPR example: Claude Opus 4.6 with about 4.1% FPR, but with lowest Hard recall about 36.5% on CLERC and 54.0% on BriefMe

This is a practical problem: a “citecheck” system that wrongly flags lots of correct citations creates reviewer fatigue.

Document type gap: briefs are easier than court opinions

Across models, Hard-example recall is higher on briefs than on court opinions by 12–26 percentage points.

For GPT-5.4, high reasoning effort yields its largest improvement on briefs:
- Recall rises from 64.0% to 82.0% on BriefMe
- vs 48.5% to 59.8% on CLERC
(They suggest a plausible reason: briefs often make narrower propositions, so a wrong pinpoint page creates a sharper mismatch. They treat this as a hypothesis, not something directly measured.)

What reasoning effort does: helps, but still leaves many misses

Adding extended reasoning (GPT-5.4 with high reasoning effort) increases recall substantially:
- CLERC Hard recall from 48.5% → 59.8%
- BriefMe Hard recall from 64.0% → 82.0%
It also slightly improves (reduces) FPR in their report:
- FPR 15.6% → 13.1% on CLERC
- FPR 14.6% → 14.4% on BriefMe

But even under this stronger setting, they report a striking remaining failure:
- GPT-5.4 still misses about 40% of pinpoint mismatches on court opinions.

Why models fail on pinpoint pages: they “believe” the wrong support

The paper’s error analysis is where the story gets really concrete. When models accept corrupted citations (false negatives), they often do it for predictable reasons.

1) They invent support rationales

In about two-thirds of missed pinpoint mismatches, the model rationale claims the cited page “expressly states” or “explicitly says” language that isn’t actually there.

That’s huge for safety because it creates a false sense of validation. A lawyer reading the rationale might not have reason to doubt it—because it sounds confident and doctrinally fluent.

2) They use topical overlap as a substitute for page-level verification

In wrong-pinpoint corruptions, the wrong page is still from the same case and often still within the same doctrinal dispute. So the model treats that overlap as evidence of support, rather than forcing itself to check what the page actually contains.

3) Quoted text is often the giveaway—and models still miss it

They note that many Hard examples include verbatim quotes attributed to the authority. In missed cases, that quoted language appears on the cited page only 92% of the time? (They phrase it as: for GPT-5 false negatives on these examples, the quoted text does not appear on the cited page 92% of the time.) In other words, the quote-matching signal is present, but models still often fail to use it as a verification step.

Can prompting fix it? Yes, partially—at a cost

The paper tests an intervention called a page-grounded prompt. Conceptually, it tries to force the model to do two things:
1. verify that the cited page contains the relevant quoted/claimed language (or a close paraphrase that’s actually present), and
2. judge support for the specific proposition, not just doctrinal similarity.

What changes in results with page-grounded prompting

Across models, page-grounded prompting improves recall on Hard examples by:
- ~9.7 to 27.8 pp on CLERC
- ~6.5 to 35.5 pp on BriefMe

As expected, the gains are concentrated on Hard examples. Easy-example recall is already saturated (often above ~95%), so there’s not much room to improve. They report that on GPT-5, the recall improvement is larger when verbatim quotes are present versus absent (they use an example breakdown showing +14.1 pp on cases with verbatim quotes vs +10.5 pp without).

The catch: false positives go up

This prompt change increases the false positive rate across the board—roughly 1.0 to 24.7 percentage points depending on the model/dataset. The reason is intuitive: when you tell a model to verify page-level support, it becomes more skeptical of all citations, not only the corrupted ones.

So the prompting approach is diagnostic (helps find some errors) rather than a clean “fix” that selectively targets only wrong pinpoints.

So, is a citation “on point” after all? Practical implications for legal AI

Stepping back, the paper supports a concrete operational lesson:

If your AI system only checks citation existence or superficial plausibility, it will miss the misrepresentation problem. Wrong-pinpoint citations remain a major vulnerability.

A sensible workflow implication

If you’re building or using an AI citechecking tool today, you probably want a two-level system:
- Level 1 (easy checks): detect nonexistent cases or obviously mismatched citations.
- Level 2 (hard checks): verify proposition support at the pinpoint location.

This paper’s central point is that current models often succeed at Level 1 and partially at Level 2—but not reliably enough for high-stakes legal drafting without careful supervision.

Where this could be applied immediately

The most immediate use case is in tools that assist attorneys during review:
- highlighting citations that the model judges as potentially not on point,
- prompting a reviewer to open the pinpoint page,
- and flagging specific failure indicators (e.g., “the cited page doesn’t appear to contain the quoted language”).

This isn’t just for the attorney who drafted the brief. The paper notes legal cases where fabricated citations were not caught until after judicial prompting, suggesting that citechecking obligations may matter for opposing-party filings too.

Key Takeaways

• The hardest citation error isn’t fake cases—it’s real citations that don’t support the claim. The paper focuses on misrepresentation at the pinpoint-page level.
• Models are strong on easy corruption but weak on pinpoint mismatches:
  • CLERC Hard recall: ~37–61%
  • BriefMe Hard recall: ~52–83%
• More reasoning and newer models help, but don’t close the gap:
  • Even GPT-5.4 with high reasoning effort still misses about 40% of pinpoint mismatches on court opinions.
• Prompting can improve page-level verification (“page-grounded prompting”), improving Hard recall by roughly ~9.7–27.8 pp (CLERC) and ~6.5–35.5 pp (BriefMe)—but it raises false positives by ~1.0–24.7 pp.
• Failure mode diagnosis matters: models often rely on topical overlap and sometimes generate rationales claiming a page “expressly states” text that isn’t there.
• Practical takeaway for legal teams: citation existence checks aren’t enough. Real safety requires proposition-level, pinpoint-aware support verification—ideally surfaced as reviewer prompts rather than fully trusted judgments.

If you want, I can also turn this into a “how to implement a safer citechecking workflow” checklist based directly on the paper’s failure modes (invented support rationales, topical matching, and missed quote presence).