Insurance claims AI evaluation: adjudication outcomes as test labels

What a claims evaluation set should test

A claims eval set should test the decisions an adjuster is accountable for, scored separately, because a model can write a fluent coverage letter and still pay the wrong amount. Treat each target as its own metric with its own error cost. The targets below are a working hypothesis for most property and casualty programs; adjust them to the decisions your agent actually makes.

• FNOL triage: routing to the right queue, line of business and complexity tier; flagging injury, total loss or subrogation potential at first notice.
• Coverage decision: covered, partially covered or denied, plus the policy provision cited. A correct outcome with the wrong provision is a partial failure.
• Reserving: initial reserve and direction of later adjustments. Score error bands, not exact match.
• Fraud or SIU referral: whether the file was referred, and whether the referral was substantiated.
• Payment amount: indemnity and expense paid, net of deductible, salvage and subrogation recovery.

If your agent drafts communications, add a policy-following check like the ones used for support agents in policy-following evaluation for customer support agents. Keep it separate from the adjudication score so one cannot mask the other.

Why final adjudication beats first-pass decisions as ground truth

The final adjudication, after reopenings, appeals and litigation, is the better label because first-pass decisions are exactly what claims automation is meant to improve. Production-failure guidance for LLM systems treats escalations and reversals as signals that an output was wrong, and claims files carry the same signal in reopen flags, supplemental payments and suit dates [2]. A claim denied in week two and paid after appeal in month five should be labeled "covered", with the reversal recorded as metadata.

This argues for a maturity window: sample claims closed long enough that reopen rates for that line have flattened. Short-tail auto physical damage settles quickly; bodily injury and workers' compensation can stay open for years, so the eligible population skews older. Record the closure date and window in the spec so buyers can see the tradeoff.

Even final labels contain errors. Audits of widely used benchmark test sets found an average label error rate of at least 3.3%, enough to reorder model rankings [3]. Plan a gold-label audit on a sample, and read accepting a delivered eval set before acceptance.

How claim file documentation supports labels

Claim files are usable as eval data largely because regulators expect them to be reconstructable. The NAIC model regulation on property and casualty claims settlement practices calls for files detailed enough that pertinent events and dates can be reconstructed, and for denials to reference the specific policy provision relied on [4]. Where a state has adopted similar rules, a closed file should contain a dated diary, the coverage position and the reason, which is what an evaluator needs.

In practice the useful fields sit in claim administration systems such as Guidewire ClaimCenter, Duck Creek Claims or in-house platforms: loss date, report date, line and coverage codes, cause of loss, reserve transactions, payment transactions, diary or activity notes, and attachments such as estimates, photos and recorded statement transcripts. Ask suppliers which of these exist as structured fields and which only appear in free-text adjuster notes, since notes need extraction before they can serve as labels.

Slicing by line of business and severity

Report results by slice, because aggregate accuracy on claims is dominated by high-volume, low-severity files that are the easiest to automate. A model that is 95% right on glass claims and wrong on most litigated bodily injury files can look excellent overall. Minimum slices to request:

• line of business (personal auto, homeowners, commercial property, general liability, workers' compensation);
• severity band by incurred amount;
• claim outcome (paid, partially paid, denied, closed without payment, subrogated);
• jurisdiction (state), since coverage law and claims rules vary;
• reopened versus never reopened.

Set a minimum count per cell and oversample rare outcomes such as substantiated fraud and coverage litigation. Historical decisions also encode past adjuster behavior, including bias, so review historical decision bias in operational labels before treating every outcome as correct.

Grading agent actions in the claim system

Claims agents should be graded on the state they leave the system in, not on transcript similarity. Tool-agent benchmarks such as τ-bench compare the database end state after an episode with an annotated goal state and measure consistency across repeated runs [1]. For claims, the goal state is the reserve set, payment issued, coverage code applied and referral created, all reconstructed from the closed file's transaction history.

This needs a sandboxed claim system or a mock API that replays the policy, coverage and prior transactions at the evaluation point. Run each case several times: an agent that issues the right payment in four of five runs is not ready to issue payments. For task-suite design, see agent evaluation task suites.

Illustrative eval record

A record should freeze the file at the decision point and store the later outcome separately, so the model never sees the answer. Deliver as JSON Lines, one UTF-8 record per line [8].

Illustrative example: invented to show structure; it does not describe an available dataset.

{"case_id": "eval-000123",
 "line_of_business": "homeowners", "state": "TX", "severity_band": "25k-100k",
 "evaluation_point": "coverage_decision",
 "inputs": {"fnol_text": "[REDACTED_NAME] reports water damage in kitchen after supply line failure...",
            "policy_form": "HO-3", "endorsements": ["water_backup_excluded"],
            "adjuster_notes_to_date": ["Inspection 2025-03-04: sudden discharge, no long-term seepage observed"],
            "attachments": ["estimate_xactimate.pdf", "photos_01-12.jpg"]},
 "gold": {"coverage_position": "covered", "provision_cited": "Section I - Perils Insured Against",
          "initial_reserve_band": "25k-50k", "siu_referral": false,
          "indemnity_paid_band": "25k-50k", "reopened": true, "final_after_reopen": true},
 "provenance": {"closed_date": "2025-09-30", "label_source": "final_adjudication", "deid_method": "token_replacement"}}

Privacy, regulation and documentation

Claim files hold names, policy numbers, addresses, bank details and often medical information, so de-identification has to preserve the facts graders rely on. Replace identifiers with consistent tokens rather than deleting them, keep dates as relative offsets if needed, and test that coverage outcomes still follow from the redacted file, as covered in de-identifying evaluation data without breaking the test. Health information in bodily injury or workers' compensation files can bring in HIPAA when the source is a covered entity or business associate.

Insurers using AI are also expected to show their testing in states that have adopted the NAIC model bulletin on insurers' use of AI systems. The bulletin expects a written program with testing and controls that extend to third-party data and models [5]; see the NAIC bulletin and state rules on third-party data. In Colorado, as of October 2026, SB26-189 replaces the 2024 AI Act and, from January 1, 2027, requires developers of automated decision tools that materially influence consequential decisions to give deployers technical documentation [6]. A documented, outcome-labeled eval set is part of that evidence.

This page is general information, not legal advice. Confirm requirements with counsel for your jurisdiction and use case.

Keeping the set held out

Private claims eval sets stay useful only while they stay out of training data. Public test items leak into newer models' training corpora and inflate scores, which is why benchmark designers refresh questions over time [7]. Keep claim-derived sets under an evaluation-scoped license, restrict who can export them, and refresh with newly closed claims each cycle; see contamination-resistant evaluation design and the wider evaluation datasets hub.

Sourcing closed claim files from operating companies

SourceX sources operational datasets from US companies on request, including document and workflow records, and manages licensing and ongoing purchases; it does not hold claims data in stock and a request does not guarantee a match. Buyers describe the data they need, such as lines, states, closure windows and fields, and SourceX looks for US businesses that hold it, with every release approved by the supplying company. You can describe your claims evaluation requirement once your spec is drafted with the evaluation dataset specification guide.

Each dataset is rights-reviewed for ownership and consents and delivered under a license that defines records, uses, term and delivery. Names, emails, phone numbers and account numbers are removed or replaced before delivery, the method is recorded and a sample is checked, though no method is perfect; health records require HIPAA de-identification by Safe Harbor or Expert Determination. For category context see insurance claims datasets, evaluation sets built from real business work and insurance brokerages as buyers.

Request closed claim files for evaluation

SourceX finds US businesses that hold the claim records you describe, assesses the data and licensing permissions, and agrees pricing and allowed uses in a license before anything is delivered through a private, access-controlled workflow. Nothing is contracted until a supplier agrees. Start a buyer request.