Pairwise, ranking or rating: choosing a preference data format

This guide sits in our fine-tuning and post-training data hub. It covers format choice only; for measuring label noise once data exists, see preference data quality and noise, and for the general concept see the preference data glossary entry.

How each format turns into training signal

Every common post-training objective consumes pairs in the end, so the real question is how much information is lost or added on the way to a pair. DPO is derived from a Bradley-Terry model of pairwise preference and trains directly on (prompt, chosen, rejected) triples [3]. Hugging Face TRL's DPO and reward trainers expect exactly that "preference" structure, while KTO uses an "unpaired preference" structure of one completion plus a boolean label [4][5].

• Pairwise (A vs B). One annotator decision yields one pair. It maps one-to-one onto DPO and reward-model losses. Llama 2 collected binary comparisons and attached a strength level to each [2].
• k-way ranking. An annotator orders K responses, which expands into K(K-1)/2 implied pairs. InstructGPT had labelers rank between four and nine outputs per prompt and trained its reward model on all the implied comparisons, treating them as one batch element per prompt to avoid overfitting [1]. DPO also has a Plackett-Luce form that consumes full rankings without expanding them [3].
• Absolute rating (Likert or attribute scores). Each response gets a score, often on several axes. HelpSteer2 rates every response on helpfulness, correctness, coherence, complexity and verbosity [6]. Ratings can train a regression reward model directly, or be converted to pairs by comparing scores of responses to the same prompt.

Why pairwise labels are easier to keep consistent

The usual rationale for pairwise collection is that relative judgments do not require each annotator to hold a stable internal scale, so a "4" from one rater and a "4" from another do not have to mean the same thing. That is a widely held working hypothesis rather than a settled result, and it should be tested on your own task. Even pairwise labels are noisy: one reward-modeling study cites inter-annotator agreement typically in the 63-72% range [10].

Absolute ratings fail in specific, recognizable ways. Raters compress onto the middle of a 5- or 7-point scale, drift upward over a long session, and anchor on whichever response they saw first. Per-rater normalization (z-scoring each annotator's scores) helps, but only if every rater labels enough items to estimate their personal mean and spread.

Pairwise labels fail differently. Position bias (preferring whichever response appears on the left) and length bias (preferring the longer answer) leak straight into the reward model. Randomize presentation order, log it in the record, and check the left-win rate during collection rather than after.

When k-way rankings pay off

Rankings make sense when the expensive step is reading a prompt and its context, not comparing responses. If an annotator must absorb a 3,000-word contract or a long support thread before judging, ranking four responses in one sitting amortizes that reading cost across six implied pairs instead of one.

The catch is that implied pairs are not independent. All six pairs from a 4-way ranking share one annotator, one reading of the prompt and one possible mistake at the top of the list. InstructGPT's choice to keep a prompt's comparisons together in one batch element was a direct response to this correlation [1]. If you shuffle implied pairs into a flat DPO file, you overweight high-K prompts and let one mislabeled ranking contaminate many rows.

Rankings also get harder as K grows. In practice, beyond about five responses, annotators tend to sort the clear best and worst carefully and order the middle carelessly. A practical compromise is "best-worst" or partial ranking: record only the top and bottom choices, and treat the middle as unordered.

Converting ratings to pairs within a prompt

Ratings convert to pairs cleanly only when both responses were rated against the same prompt, by the same rubric, ideally by the same rater. UltraFeedback publishes per-aspect ratings for multiple completions per prompt, and downstream "binarized" releases pick a chosen and a rejected response from those scores [8]. The conversion rule is a design decision you should write down, not a default.

Common conversion rules, each with a failure mode:

• Highest vs lowest. Maximizes margin, but trains the model mainly on easy contrasts.
• Highest vs random lower. Adds harder pairs but keeps some near-ties that are mostly noise.
• All pairs above a score gap. Drops pairs whose scores differ by less than a threshold (for example one point on a 5-point scale), which removes ambiguous pairs at the cost of volume.
• Weighted aggregate across attributes. HelpSteer2-style multi-attribute scores need an explicit weighting before you can rank; a verbosity score weighted positively will teach length preference.

HelpSteer2-Preference collected pairwise preferences with strength and written justifications on responses that already had ratings, specifically so Bradley-Terry and regression reward models could be compared on matched data [7]. If you are designing a new collection and can afford it, gathering both on the same responses is the cleanest way to settle the format question for your domain.

Preference strength labels and margins

A strength label turns a binary choice into an ordinal one, and it is the cheapest way to add information to pairwise data. Llama 2 asked annotators whether the chosen response was significantly better, better, slightly better, or negligibly better or unsure, and used those levels to set a margin in the reward-model loss so that clear preferences push scores further apart [2].

Strength labels change your label distribution in ways you should plan for. Annotators tend to overuse the middle levels, and "slightly better" pairs disagree far more often across raters than "significantly better" pairs. Before training, look at agreement broken out by strength level; many teams drop or down-weight the weakest level entirely.

Note that a strength label is still relative. It does not tell you whether either response is acceptable, which is why some pipelines add a separate binary "is this response acceptable" flag per response.

Ties, "both bad" and abstentions

Allowing ties makes labels more honest and more expensive to use, because standard DPO and Bradley-Terry losses have no slot for a tie. Chatbot Arena lets voters pick a winner, declare a tie, or mark both responses as bad, and ranks models with a Bradley-Terry model fit to those votes [9].

Decide before collection what each outcome will become in training:

• Tie (both good): drop from DPO, keep for evaluation, or model explicitly with a tie-aware extension of Bradley-Terry such as the Rao-Kupper or Davidson formulations.
• Both bad: drop from pairwise training, but route the prompt to SFT rewriting or to KTO as two negative examples, since KTO accepts unpaired desirable or undesirable labels [5].
• Cannot judge (needs expertise, missing context): never coerce into a tie. Log it with a reason code; a high rate signals the wrong annotator pool, which is the problem expert preference data addresses.

Forcing a choice (no tie option) inflates apparent volume and hides ambiguity. If you do force a choice, pair it with a strength scale whose lowest level acts as a soft tie.

Decision table: picking a format

The right format depends on the trainer you will use, the cost of reading the prompt, and whether you also need evaluation-grade scores.

Illustrative example: invented to show structure; it does not describe an available dataset.

For side-by-side evaluation rather than training data, see human preference evaluation with side-by-side judgments.

A record schema that keeps every option open

Store the raw judgment, not the converted pair, so you can re-derive training files when your objective changes. A flat chosen/rejected file discards rater identity, presentation order, ties and strength, and none of those can be recovered later.

Illustrative example: invented to show structure; it does not describe an available dataset.

{
  "judgment_id": "j_000184",
  "prompt_id": "p_00921",
  "prompt": [{"role": "user", "content": "Summarize the refund policy in this ticket thread..."}],
  "responses": [
    {"response_id": "r1", "model_id": "policy-v3", "content": "..."},
    {"response_id": "r2", "model_id": "policy-v3", "content": "..."},
    {"response_id": "r3", "model_id": "baseline", "content": "..."}
  ],
  "format": "ranking",
  "presentation_order": ["r2", "r3", "r1"],
  "ranking": ["r2", "r1", "r3"],
  "strength_top_vs_next": "slightly_better",
  "ties": [],
  "both_bad": false,
  "attribute_ratings": {"r2": {"correctness": 4, "verbosity": 2}},
  "acceptable": {"r1": true, "r2": true, "r3": false},
  "annotator_id": "a_017",
  "annotator_expertise": "support_ops_lead",
  "guideline_version": "pref-guide-2.3",
  "rationale": "r2 cites the 30-day window; r3 invents a fee.",
  "seconds_spent": 212
}

From this one record you can emit three DPO pairs, a Plackett-Luce ranking, KTO rows from the acceptability flags, and attribute-level regression targets. Keep the prompt in the same message structure you use for SFT, as described in chat fine-tuning data format, so preference and SFT data share tokenization and masking.

What to ask for when buying or licensing preference data

Format choice also shapes what you should demand from a supplier, because a dataset delivered as bare chosen/rejected pairs has already made the conversion decisions for you. Whether you commission new collection or license existing records, buying RLHF comparison data compares those routes in detail.

Ask suppliers for:

1. The original collection format (pairwise, ranking or rating) and the annotation guideline version.
2. The conversion rule used to produce pairs, including score gaps and tie handling.
3. Strength or margin fields and their distribution.
4. Presentation order and annotator identifiers, pseudonymized if needed.
5. Which models generated the responses, since off-policy pairs behave differently from on-policy ones; see on-policy vs off-policy preference data.
6. Agreement statistics broken out by strength level, computed with a stated metric.

Many enterprise teams also need preference signal grounded in real operational work, such as support resolutions or engineering reviews. If that is your case, you can describe the data you need to SourceX: SourceX sources operational datasets from US companies on request, and every dataset is rights-reviewed and delivered under a license that defines records, uses, term and delivery.

Sourcing preference data for your post-training stack

SourceX sources operational datasets from US companies on request, including support and sales histories, engineering records and new recordings of hands-on work, and manages licensing and ongoing purchases. Nothing is contracted until the supplying company agrees, and personal details are removed or replaced before delivery. Tell us the format, domain and volume you need at sourcex.si/buyers.