Coverage-gap analysis: using unanswered queries to decide what content to license

Why aggregate RAG metrics hide the gaps worth buying

Aggregate scores hide content gaps because frequent query types dominate the average, so a rare but expensive topic can fail almost every time while recall@10 looks healthy. Practitioners recommend scoring retrieval per query cluster rather than only in aggregate, precisely so missing-content clusters stand out [1]. Coverage is also measurable in its own right: recent benchmark work treats information coverage across tasks and domains as a property you can test, not a vague impression [4].

The practical consequence for a buyer is that a vendor demo or a public benchmark tells you little. The only coverage number that matters is coverage against your own query patterns, which is also the advice vendors of content feeds give when evaluating sources [3]. Build the gap analysis before any sourcing conversation, so you arrive with evidence instead of a category name.

What to log so unanswered queries become evidence

Capture one row per query turn with enough context to replay it later against a candidate corpus. At minimum, record the raw query, a normalized form, session and tenant identifiers, timestamp, the retriever's top-k document IDs and scores, the reranker scores if you run one, the generator's final answer, whether it abstained, and any user signal such as thumbs-down, rephrase within 60 seconds, or handoff to a human agent. Store it in a columnar format such as Parquet so clustering and per-cluster scoring stay cheap.

Failure signals worth flagging include:

• Explicit abstention: the model said it could not find the answer, or your guardrail blocked an unsupported answer.
• Low-confidence retrieval: top-1 similarity or reranker score below a calibrated threshold.
• Implicit dissatisfaction: immediate rephrase, session abandonment, thumbs-down, or a support ticket opened after the chat.
• Ungrounded answers: a faithfulness check finds claims with no supporting passage (see faithfulness evaluation sets).
• Zero-result search: if the assistant fronts a search index, zero-hit and no-click queries are a cheap early signal.

Query logs contain names, account numbers and free-text health or financial details, so scrub them before anyone outside the product team sees them. Tools such as Microsoft Presidio detect and replace common PII entities, but the project itself warns it cannot guarantee it finds everything, so pair it with sampling review [7]. Our guide to de-identifying search query logs covers the details.

Separating retrieval bugs from missing content

Most unanswered queries are not content gaps, and buying content will not fix them. Classify every failing query by type before you cluster; difficulty taxonomies used in RAG benchmarks, which tag test queries by the kind of reasoning or retrieval they demand, are a workable starting model [2]. The goal is a single question per query: does the answer exist anywhere in the corpus we already hold?

Illustrative example: invented to show structure; it does not describe an available dataset.

Confirm "missing content" with an exhaustive check rather than the production retriever: run lexical and dense search at a large k, and have a subject-matter reviewer sample the result. The stale-content row deserves attention because superseded manuals and policies produce confident wrong answers; see evaluating RAG on outdated and conflicting documents and superseded versions and near-duplicates.

Clustering missing-content queries into sourceable topics

Cluster only the queries confirmed as missing content, and label each cluster in terms a supplier would recognize. Embed normalized queries, cluster with HDBSCAN or k-means over a range of k, then have a reviewer name each cluster with three things: the document type that would answer it (service bulletin, claims adjudication note, warranty policy, resolved support ticket), the domain, and the freshness it requires. A cluster labeled "error code E-42 on 2019-2021 compressor models" maps to service documentation; a cluster labeled "billing questions" does not map to anything.

Keep a held-out set of 30 to 100 real queries per priority cluster, with the expected answer type noted. That set becomes your acceptance test when a candidate source arrives, and it can be pinned alongside a corpus snapshot for reproducibility, as described in pinned corpus snapshots for RAG evaluation.

Scoring and ranking gaps for a content-sourcing list

Rank clusters by expected value of closing them, not by query count alone. A useful score multiplies monthly query volume by the cost of a failure (escalation cost, churn risk, compliance exposure) and by the share of the cluster a single source plausibly covers, then divides by acquisition and integration effort. Treat the coverage share as a hypothesis until a pilot measures it.

Illustrative example: invented to show structure; it does not describe an available dataset.

Two cautions apply. High-volume, low-cost clusters are usually cheaper to author internally than to license. And long-tail clusters with tiny volume can still top the list when each failure triggers an escalation or a regulatory risk, which fits the NIST AI RMF approach of mapping context-specific risks and measuring them over time (its MAP and MEASURE functions) [6].

Turning ranked gaps into a content request

A good content request describes the data that would close a gap, not a supplier you hope exists. For each priority cluster, specify document types, example queries (de-identified), required fields and metadata, date range and update cadence, volume range, format, and the intended use: internal assistant, customer-facing answers, or retriever training. The use matters because licensing terms differ sharply between them; see internal vs customer-facing RAG licensing and RAG content license terms.

Illustrative example: invented to show structure; it does not describe an available dataset.

gap_cluster: legacy-hvac-fault-codes
unanswered_queries_sample: ["E-42 after defrost cycle on 2019 unit", "reset sequence for lockout code 7"]
document_types: [service manuals, technical service bulletins, field technician notes]
required_metadata: [model_number, revision_date, document_status, section_id]
date_range: 2015-2023 equipment years
update_cadence: quarterly
format: PDF plus extracted text, or JSONL with section boundaries
intended_use: customer-facing assistant answers with citations
acceptance_test: held-out set of 60 queries; target answer-found rate set before pilot

The SourceX guide on how to write a data request for suppliers covers the general structure. Ask for metadata up front, because missing revision dates and document status make an otherwise relevant corpus hard to trust; metadata a licensed RAG corpus should ship with lists the fields.

Validating lift before you sign

Replay the held-out cluster queries against a candidate sample before committing, and measure answer-found rate and faithfulness per cluster rather than one blended score [1]. Include deliberately unanswerable questions in the replay set: reading-comprehension research shows models tend to guess when the context has no answer, so a source that raises answer rates while also raising unsupported guesses is not a win [5]. Design the pilot, its sample size and its success threshold using pilot-testing a content source for retrieval lift.

After a source goes live, keep the gap log running. Clusters that stay unanswered tell you whether the coverage estimate was wrong, whether the content arrived but retrieval failed, or whether the query mix shifted. That loop is what makes the next sourcing decision cheaper than the first.

Where SourceX fits in RAG gap analysis

SourceX sources operational datasets from US companies on request, including support and sales histories, engineering records, documents, and finance and legal workflows, which are often the material behind long-tail missing-content clusters. Nothing is held in stock, and a request does not guarantee a match; you describe the data your gap clusters need, and every release is approved by the supplying company. Each dataset is rights-reviewed for ownership and consents and delivered under a license that defines records, uses, term and delivery. Buyers can start a request on the SourceX buyers page, and the retrieval and RAG content licensing hub covers the surrounding licensing questions.

Close your RAG knowledge gaps with sourced content

If your gap analysis points to operational documents your corpus lacks, describe the document types, fields and intended use, and SourceX will look for US businesses that hold that data. Personal details are removed or replaced before delivery, and pricing and allowed uses are agreed in a license before anything is delivered. Describe the content your RAG gaps need.