Step-level labels for process reward models: annotation spec and sourcing

This page is the specification layer. For sourcing options and supplier types, see the owner page on training data for process supervision; for the concept itself, the process supervision glossary entry; and for the wider landscape of post-training data, the fine-tuning and post-training buyer's guide.

Outcome labels versus process labels

Outcome labels say whether the final answer is right; process labels say where the reasoning first went wrong. An outcome reward model learns from final-answer correctness, while a PRM learns from per-step correctness and can therefore localize a failure. When a domain lacks an easy-to-check final answer, judging a whole solution largely reduces to finding its first mistake, which is exactly what step labels record.

That framing matters for buyers. If your target domain has a cheap verifier (unit tests, a numeric answer, a schema check), outcome or verifier-based data may be enough; see RLVR datasets with prompts, reference answers and verifiers. Step labels earn their cost when answers are hard to verify, when correct answers can come from flawed reasoning, or when you want a dense reward for search and reranking.

Step segmentation rules come first

Segmentation is the most common silent defect in PRM data, so fix it in the spec before labeling. If raters decide their own step boundaries, the same reasoning gets split differently and per-step agreement becomes meaningless.

Three workable rules, in order of preference:

• Generator-defined steps. Prompt the policy model to emit one step per line or per delimiter (for example \n\n or a <step> tag) and treat each as atomic. Raters may not merge or split.
• Deterministic post-hoc splitting. Split on newline, numbered list item or sentence boundary with a published script; record the script version in every record.
• Semantic steps for domain work. For a contract analysis or a reconciliation, define a step as one claim plus its support (one clause interpretation, one ledger match). Write examples of what counts as one step.

Whatever you choose, store step_index, character offsets into the full solution, and the segmenter version. A step that bundles two inferences, one right and one wrong, has no correct label; the spec should tell raters to flag it as segmentation_issue rather than guess.

The label set and its tie-breaks

A three-way label per step (correct, incorrect, neutral) is a common choice, and it maps cleanly to a +1, -1, 0 reward. If you plan to benchmark against a public human-labeled PRM dataset, check that dataset's own documentation for its exact rating scheme before adopting it. The spec needs explicit definitions so raters do not drift:

• Correct (+1): the step is mathematically or logically valid given prior steps and moves toward the answer.
• Incorrect (-1): the step contains a false statement, an invalid inference, a calculation error, or relies on an earlier error.
• Neutral (0): the step is valid but adds nothing (restating the problem, a harmless detour), or is ambiguous in a way the rubric cannot resolve.

The tie-breaks are where specs fail. Decide in writing: Is a correct step built on an earlier wrong step incorrect, or correct-given-context? Is an unjustified but true claim correct? Is a step that is right but would mislead a reader neutral or incorrect? Most teams label "correct given prior steps" and rely on first-error truncation to avoid the cascade question entirely.

First-error labeling versus full labeling

Labeling only up to the first incorrect step is cheaper, and it is a widely used convention in published process-supervision work. Everything after the first error is left unlabeled, because once a solution is wrong, later steps are ambiguous to judge and add little training signal for a model that is meant to detect the first failure.

Full labeling of every step costs more rater time and needs the cascade tie-break above, but it supports recovery behavior (a model that notices and corrects its own error). Choose by use case:

Illustrative example: invented to show structure; it does not describe an available dataset.

Human labels versus rollout-estimated labels

Automated step labels are a real alternative to human raters, with a known trade-off. Rollout-based labeling samples N completions from each step and scores the step by how many reach the correct final answer, treating a step's value as its potential to reach the answer. More rollouts give less noisy estimates at higher compute cost, and the labels stay noisy because a wrong step can still be followed by a lucky recovery.

In practice, use rollout labels where a final-answer checker exists and the domain is narrow; use human labels where no checker exists, where right answers can come from wrong reasoning, or for the evaluation split. A common hybrid: rollout-label the training pool, then have experts relabel a stratified sample to measure how far the automatic labels drift from human judgment. For model-written traces generally, see reasoning trace datasets: human-written vs model-generated.

Who is qualified to judge a step

Rater qualification should match the domain of the steps, not the domain of the platform. A generalist can check arithmetic; a step that reads "this indemnity cap survives termination" or "this accrual reverses next period" needs someone who has done that work. Build a qualification test from 20 to 50 pre-labeled solutions that include planted subtle errors, and gate access on first-error accuracy rather than overall agreement.

The same logic applies to preference work; see domain-expert preference data for specialized models. Record the rater ID, qualification version and seniority in each label so you can audit by rater later.

Quality control that fits step labels

Step labels need QA tuned to the first-error decision, because that single index carries most of the signal. Even established benchmark test sets carry an estimated average label error rate of at least 3.3% [1], so assume your PRM set will have errors and plan to measure them.

Practical controls:

• Gold solutions (honeypots). Mix pre-labeled solutions with known first-error positions into the queue; consensus, honeypot and qualification tasks are standard features of annotation platforms [2]. Our guide to gold questions, honeypots and consensus covers set-up.
• Double labeling on a sample. Report agreement on the first-error index (exact match and within-one-step), not just per-step percent agreement, which the many easy correct steps inflate (and which can make per-step kappa misleadingly low).
• Adjudication. Route disagreements on the first-error step to a senior rater with a written rationale; see resolving annotator disagreement.
• Class balance checks. Track the share of solutions with no error; a pool that is 90% all-correct trains a PRM that says yes.

Record schema and documentation

Store each solution as one record with per-step children so you can re-segment or re-truncate later without relabeling. Document the labeling process in a machine-readable datasheet; Croissant-RAI extends dataset documentation to cover data labeling and related responsible-AI metadata [3].

Illustrative example: invented to show structure; it does not describe an available dataset.

{
  "solution_id": "sol_000412",
  "problem_id": "prob_0193",
  "domain": "accounts_payable_reconciliation",
  "generator": {"model": "policy-v3", "temperature": 0.8},
  "segmenter": {"rule": "double_newline", "version": "1.2"},
  "labeling_mode": "first_error",
  "final_answer_correct": false,
  "first_error_step": 3,
  "steps": [
    {"step_index": 1, "start": 0, "end": 142, "label": 1},
    {"step_index": 2, "start": 144, "end": 301, "label": 0},
    {"step_index": 3, "start": 303, "end": 488, "label": -1,
     "error_type": "invalid_inference",
     "rationale": "Matches invoice to PO on amount only; vendor IDs differ."}
  ],
  "rater": {"id": "r_17", "qualification": "ap_v2", "gold": false},
  "adjudicated": false
}

Estimating effort before you commission

Effort scales with steps judged, not solutions, so estimate from your own pilot rather than vendor rules of thumb. Run 100 solutions through the full spec, then measure:

1. Mean steps per solution, and mean steps judged under first-error truncation.
2. Median seconds per judged step for qualified raters, split by step type.
3. Double-label rate and adjudication rate needed to hit your first-error agreement target.
4. Share of records flagged segmentation_issue; above a few percent, fix the segmenter before scaling.

Total rater hours is roughly solutions x judged steps x seconds per step x (1 + double-label rate + adjudication rate) / 3,600. Add spec-revision time: most teams revise the tie-break rules at least once after the pilot.

Sourcing the problems and traces

The problems you label matter as much as the labels. Real tasks from operational work, such as support escalations, finance reconciliations, engineering incident write-ups or contract reviews, produce error distributions that synthetic textbook problems do not. If you need those underlying records, SourceX sources operational datasets from US companies on request, covering support and sales histories, engineering records, documents and finance and legal workflows; categories are not held in stock and a request does not guarantee a match. You can describe the records your PRM needs.

Each dataset is rights-reviewed for ownership and consents and delivered under a license defining records, uses, term and delivery, with personal details such as names, emails and account numbers removed or replaced before delivery; no de-identification method is perfect. Before buying any input data, use the checklist in how to evaluate a fine-tuning dataset before you buy it.

Find source records for step-level reward data

If your PRM needs reasoning grounded in real business work rather than textbook problems, describe the records, not the companies that might hold them. SourceX looks for US businesses that hold the described data, assesses data and licensing permissions, and nothing is contracted until a supplier agrees and approves the release. Start a buyer request.