Search query and click logs for retriever training

Why public click logs rarely work for a commercial retriever

Public click datasets are mostly research-licensed, so they can benchmark a model but usually cannot train one you ship. ORCAS offers about 18 million clicked query-document connections across roughly 10 million distinct queries, yet as of October 2026 it sits under the MS MARCO terms, which say the datasets are for non-commercial research only [1][5]. Its authors note that click logs are normally withheld because they reveal personal or commercially sensitive information, and that ORCAS needed aggregation and frequency filtering before release [2].

Domain logs are where the value is. The TripClick health search log, about 5.2 million interactions from 2013 to 2020 turned into roughly 700,000 benchmark queries, showed that a single vertical's clicks can be rich enough to train neural rankers [3]. Production teams already do this privately: one domain question-answering system trained its dense retriever on proprietary click logs from help-site search [4]. For a broader map of which open retrieval sets permit commercial training, see which public retrieval datasets allow commercial use.

What a usable query and click log contains

A usable log records what the user typed, what the engine showed in order, and what the user did next, with no direct identity. Logs that keep only query and clicked URL lose the skipped results, which are your cheapest hard negatives, and they make position-bias correction impossible. Ask for the impression list, not just the clicks.

Illustrative example: invented to show structure; it does not describe an available dataset.

The document side matters as much as the log. If the corpus has changed since logging, clicked IDs may point to rewritten or deleted pages, so request a snapshot or version hashes aligned to the log window. Pair this with the guidance on metadata a licensed RAG corpus should ship with.

Correcting position bias before clicks become labels

Clicks are biased toward top-ranked results, so treat them as relevance only after you correct for position. Users click what they see first, so a document at rank 1 collects clicks the rank-8 document would have earned if shown higher. The standard remedy in the learning-to-rank literature is to estimate how likely each position is to be examined, then weight each click by the inverse of that propensity, or to model examination explicitly with a click model.

This dictates procurement. You can only estimate propensities if the log keeps the rank of every shown result, the ranker version that produced it, and ideally periods where the ranker or layout changed, because those natural interventions let you compare the same document at different positions. A log stripped of impressions forces you back to naive click-as-positive training, which mostly teaches the new model to copy the old ranker.

Other failure modes to plan for:

• Bot and internal traffic. Health checks, QA scripts and employee testing inflate head queries; ask how the supplier filtered them.
• Logging-policy lock-in. If one ranker produced every impression, documents it never surfaced have no click evidence, so pair logs with hard negatives mined carefully for false negatives.
• Head-tail skew. A few navigational queries dominate volume; deduplicate or cap per-query pairs before training.
• Evaluation leakage. Split by time or by session, not randomly by row, or reformulations leak across train and test.

For converting cleaned clicks into pointwise, pairwise or listwise groups, see reranker training data.

Real logs versus synthetic queries

Real logs capture vocabulary mismatch, typos, acronyms and intent drift that LLM-generated queries tend to smooth over. Synthetic queries generated from documents are cheap and license-clean, but they are written by a model that has already read the answer, so they overweight lexical overlap. Many teams use real logs for the head of the training mix and evaluation, then synthesize for the long tail; the tradeoffs are covered in synthetic queries vs real user queries for retriever training and the broader licensed vs synthetic vs scraped data guide.

Which systems hold useful logs depends on your target domain. Internal enterprise search over wikis and tickets, help-center and documentation search, ecommerce site search and vertical professional search each produce different query distributions. For catalog search specifically, see product search relevance data.

Privacy and license questions to settle before delivery

Query text is the riskiest field, because users type names, account numbers, order IDs and email addresses into search boxes. Session IDs, IP addresses and fine-grained timestamps can link sessions back to people even after names are removed, which is why the ORCAS release relied on aggregation and filtering [2]. The detailed scrubbing methods are on de-identifying search query logs before licensing.

Illustrative example: invented to show structure; it does not describe an available dataset.

Buyer checklist for a query and click log license

1. Permitted uses name retriever, reranker and embedding model training, not only evaluation or grounding.
2. The license states whether trained weights and derived indexes survive termination; compare with grounding license vs training license.
3. The supplier confirms its privacy notice and terms allow sharing search behavior with a third party for this purpose.
4. Query text has been scanned for emails, phone numbers, account and order numbers, with the method documented.
5. Session IDs rotate per session and are not account IDs; timestamps are coarsened if the volume is low.
6. Rare queries below a frequency threshold are dropped or reviewed, since tail queries carry the most identifying detail.
7. Document text or snapshots are licensed alongside the log, or the license explains how to resolve doc_id.
8. The log window, ranker versions and known logging outages are documented.
9. Delivery happens through an access-controlled channel, never as an email attachment.

How SourceX handles requests for search logs

SourceX sources operational datasets from US companies on request and manages the licensing process; it does not hold logs in stock, and a request does not guarantee a match. You describe the data you need, such as help-center search sessions with impressions and dwell, and SourceX looks for US businesses that hold it, with every release approved by the supplying company. Each dataset is rights-reviewed for ownership and consents, personal details such as names, emails, phone numbers and account numbers are removed or replaced before delivery with the method recorded and a sample checked, and no method is perfect. Delivery runs through private, access-controlled workflows only after an executed agreement. You can describe your retriever training data needs to SourceX or start from the retrieval and RAG data hub or the AI data guides.

License query and click logs for your retriever

SourceX serves AI teams wherever they are based and runs each request through Find, Assess, Agree, Transact and Manage, with nothing contracted until a supplier agrees. Terms, including allowed uses and delivery, are set per deal in a license. Tell SourceX which search logs your retriever needs.