Legal citation accuracy evaluation: testing drafting AI against verified authorities

This page sits in the evaluation datasets hub and is distinct from clause-level contract review evaluation: contract evals test extraction from a document in hand, while citation evals test claims about authorities outside the document.

Why public legal benchmarks do not measure citation accuracy

Public legal benchmarks mostly test reasoning, not whether a model's citations survive a cite-check. LegalBench, the main collaborative benchmark, covers 162 tasks across six types of legal reasoning, hand-crafted with legal professionals [1]. That makes it useful for issue spotting, rule application and interpretation, but its tasks rarely ask a model to produce a citation to an external authority and defend it.

Citation accuracy needs its own set because the dominant failure is not bad reasoning but bad grounding. A model can apply a rule correctly and still attach it to a case that does not exist, quote language the opinion never used, or cite a real case for a proposition it rejects. Reasoning tasks that supply the relevant law in the prompt cannot detect any of these.

Two design lessons follow. First, retrieval can lower the rate of fabricated authorities without removing misgrounded ones, where a real case is cited for something it does not hold, so existence checks alone can overstate quality. Second, a model's own confidence is not a reliable proxy for correctness, so the eval needs externally verified labels assigned by people who read the authority.

The eval unit: proposition, authority and verification label

The atomic unit of a citation eval is one proposition paired with one cited authority and a lawyer-assigned verification label. A brief paragraph that says "a party waives an argument not raised below" and cites two cases yields two units, because each case can independently support, partially support or fail to support the proposition.

Keep the proposition text, the pinpoint, and the signal ("see," "cf.," "but see") together. Signals matter: under Bluebook conventions, no signal means the authority directly states the proposition, "see" means it clearly supports it by inference, and "cf." means it supports an analogous proposition, and a model that drops or swaps a signal can turn an accurate citation into a misleading one. Record the jurisdiction and court level of both the filing and the authority, since a correct holding from a non-binding court can still be the wrong authority for the argument.

Illustrative example: invented to show structure; it does not describe an available dataset.

{
  "unit_id": "memo-0412-p07-c2",
  "source_doc_type": "internal_research_memo",
  "practice_area": "civil_procedure",
  "forum": {"jurisdiction": "US-federal", "court_level": "district", "circuit": "9th"},
  "proposition": "Arguments raised for the first time in a reply brief are generally deemed waived.",
  "signal": "see",
  "authority": {
    "type": "case",
    "citation_as_written": "[Case name], [vol] F.3d [page], [pin] (9th Cir. [year])",
    "normalized_id": "reporter:F.3d|vol|page",
    "pinpoint": "[pin]",
    "quoted_text": null
  },
  "verification": {
    "exists": true,
    "citation_form_correct": true,
    "pinpoint_supports": true,
    "holding_characterized_correctly": true,
    "supports_proposition": "direct",
    "good_law_as_of": "2026-09-30",
    "subsequent_history": "none_found",
    "verifier_role": "licensed_attorney",
    "second_review": "agree"
  },
  "redaction": {"client_identifiers_removed": true, "method": "recorded"}
}

The good_law_as_of date is load-bearing: a citation that was valid when the memo was written may have been reversed or overruled since, and your gold label must be anchored to a stated date.

Scoring citation hallucinations by failure type

Score each model citation on a ladder of checks, stopping at the first failure, so a single headline number decomposes into actionable error classes. Collapsing everything into "hallucinated or not" hides the difference between a fabricated case (fixable with a citator lookup) and a real case cited for the wrong rule (fixable only with better grounding).

Illustrative example: invented to show structure; it does not describe an available dataset.

Report checks 1 to 3 as automated precision metrics and checks 4 to 7 as rubric-scored metrics with inter-annotator agreement. The rubric design guide for domain experts covers how to write anchors for "partial support" so two reviewers converge.

For premise-sensitivity, add adversarial prompts that assert a false rule and check whether the model corrects it: a model that accepts a false premise and then finds citations for it fails in a way existence checks never catch.

Building the set from real briefs and memos

Real work product is the best raw material because it contains the citation patterns, signals and string cites that drafting models must reproduce. Filed briefs, internal research memos, and bench-style memos each contribute differently: filed briefs are adversarial and dense with string cites; internal memos contain hedged, multi-jurisdiction analysis; and drafts with partner comments show which citations a senior reviewer struck and why.

A practical build sequence:

1. Extract citations from each document with a citation parser and normalize to reporter, volume and page, keeping the original string.
2. Segment propositions so each citation attaches to the smallest sentence or clause it supports.
3. Verify existence and currency against a citator, then route characterization and support to licensed attorneys.
4. Double-review a stratified sample and adjudicate disagreements before freezing labels; test-set label errors of a few percent are enough to reorder model rankings [2].
5. Slice by practice area, jurisdiction, court level, authority type (case, statute, regulation, secondary source) and citation age.
6. Freeze and version the set with its good_law_as_of date.

Gold-label quality checks are covered in accepting a delivered eval set. For retrieval-grounded products, pair this set with question-answer-citation triples for RAG evaluation so you can attribute errors to retrieval or generation.

Contamination and held-out design

Citation evals contaminate easily because filed briefs and published opinions are public and heavily crawled. If a test brief is on a public docket, a model may have memorized its citations, which inflates existence and form scores without improving grounding.

Mitigations include favoring unfiled internal memos, re-pairing real authorities with paraphrased propositions, and keeping a rotating held-out slice. See designing contamination-resistant evaluation sets and private eval sets versus public benchmarks for split strategies.

Privilege, confidentiality and copyright review before licensing

Briefs and memos can only enter an eval set after privilege, confidentiality and copyright questions are cleared by the holder. Internal memos are typically attorney work product and may contain privileged client communications; the duty of confidentiality under ABA Model Rule 1.6 and its state analogues applies to client information even when no privilege attaches. Client names, matter numbers, opposing parties and fact patterns that identify a client need removal or replacement, and the method should be recorded.

Copyright is a separate check. In Thomson Reuters v. ROSS, the Third Circuit (September 29, 2026; status as of October 2026) affirmed a ruling that Westlaw headnotes at issue were sufficiently original to be copyrightable [3]. Eval records should therefore reference authorities by citation and court opinion text, not by copying proprietary headnotes, synopses or key-number annotations from commercial research services.

The legal LLM fine-tuning data guide covers privilege handling in more depth for training use. Note also that a lawyer who files AI-drafted briefs remains responsible for the citations under Federal Rule of Civil Procedure 11(b)(2), which is why buyers' customers care about the failure classes above.

This page is general information, not legal advice. Confirm requirements with counsel for your jurisdiction and use case.

What to specify when sourcing brief and memo eval data

A request for citation eval data should describe documents, verification depth and allowed use, not name firms. Buyers who describe the records they need to SourceX are matched against US businesses that may hold them; each release is approved by the supplying company, and a request does not guarantee a match.

Illustrative example: invented to show structure; it does not describe an available dataset.

For license terms that govern evaluation-only use and result publication, see evaluation-only data license terms and publishing results on licensed eval data. The owner pages for legal briefs and memos and legal AI data describe the broader category.

Sourcing verified briefs and memos for citation evals

SourceX sources operational documents, including legal workflow records, from US companies on request and manages the licensing process through Find, Assess, Agree, Transact and Manage. Every dataset is rights-reviewed, personal details are removed or replaced before delivery with the method recorded, and nothing is contracted until the supplier agrees. Describe your citation eval data needs to SourceX.