Seed data for synthetic instruction generation

What seed data does in a self-instruct style pipeline

Seed data sets the distribution that the generator samples around, so a narrow seed pool produces a narrow synthetic set no matter how many items you generate. Self-Instruct, the template most teams still adapt, starts from a small pool of human-written tasks, prompts a model with a few of them to write new instructions, inputs and outputs, filters out invalid and near-duplicate generations, and fine-tunes on what remains, as described in the Self-Instruct paper by Wang et al. (ACL 2023). Each round can add accepted generations back into the pool, which means seed flaws compound instead of washing out.

Three seed shapes dominate current pipelines, and they fail differently:

• Task seeds (instruction plus example input and output): best for controlling format and task taxonomy; weak at domain realism if written by your own team in an afternoon.
• Document seeds (a real ticket, contract clause, incident report or spreadsheet that the generator writes questions or tasks about): best for grounding; the generator inherits whatever the document contains, including personal data.
• Persona seeds (short descriptions of a user, such as "a hospital revenue-cycle analyst reconciling denied claims"): best for diversity at scale; Persona Hub showed persona-driven prompting can diversify instructions, math problems, knowledge texts and tool definitions, using personas curated automatically from web data (Ge et al., Tencent AI Lab, 2024).

Persona seeds diversify voice and intent but not facts. If personas are paired with no real documents, the generator fills domain details from its own pretraining, which is how synthetic sets end up fluent, varied and quietly wrong.

Grounding synthetic data in real examples

Grounding means every generated item can point to a real artifact that constrains its content, and it is the main reason to spend money on seeds rather than prompt engineering. Real business records carry things a generator rarely invents: internal product codes, the actual order of steps in a refund workflow, the three ways customers misdescribe the same error, malformed CSV exports, and the escalation paths that only exist because something went wrong once.

For an applied team building SFT data for, say, a support or finance assistant, useful document seeds look like:

• Resolved support tickets with the agent's final reply and the internal macro or knowledge-base article used.
• Engineering records such as incident postmortems, Jira-style issue threads and code review comments.
• Finance and legal workflow artifacts such as invoice exception notes, contract redlines and approval chains.
• Recordings or transcripts of hands-on work where the task depends on procedure, not just text.

The generator's job then shifts from inventing content to transforming it: write the question a customer would ask that this ticket answers, write the extraction task whose gold output is this invoice's fields, write a harder variant that changes one constraint. Our guide to turning business records into instruction-response pairs covers that conversion directly; this page is about choosing and licensing the records that feed it.

How many seed examples you need

Seed counts are driven by the number of distinct task types and edge cases you need covered, not by a target dataset size. Published results point the same way: LIMA reached competitive alignment from 1,000 carefully curated prompt-response pairs, while noting curation was labor-intensive [1], and AlpaGasus found that filtering a self-instruct-style synthetic set down to fewer, higher-quality items improved results [2]. Neither study fixes a seed count for your domain, so size the pool from a coverage plan.

A practical sizing method:

1. List the task types the model must handle (classify, extract, draft, summarize, reason over a table, refuse).
2. For each, list the axes that change difficulty: document length, language, format, missing fields, conflicting instructions.
3. Require several seeds per populated cell of that grid, more for cells where errors are expensive.
4. Generate a pilot batch, embed it, and look for clusters with no seed ancestor and seed cells with no descendants. Both are coverage bugs.

For the downstream question of how many synthetic items to train on, see how much data you need to fine-tune an LLM.

Seed specification you can hand to a data team or supplier

A seed specification states the population, the fields and the exclusions before anyone pulls records, which keeps the seed pool from becoming whatever was easiest to export. Write it the way you would write a request to a supplier, describing the data rather than naming companies; the same spec works as a sourcing brief if you request licensed operational data through SourceX.

Illustrative example: invented to show structure; it does not describe an available dataset.

Store this as a column set in a Parquet or JSONL manifest next to the generated data, not in a spreadsheet that drifts from the corpus. Croissant-RAI gives a machine-readable vocabulary for life cycle and provenance metadata if you need to publish or exchange the documentation [6]; our page on provenance records for synthetic training data goes field by field.

Licensing seed data for synthetic generation

A seed license must expressly permit generating derivative synthetic data, using those outputs to train models, and keeping or deleting them under defined conditions at termination. Generic "internal research" or "model training" grants often leave three gaps that surface late in diligence:

• Derivation. Does the grant cover creating new records that paraphrase, recombine or are conditioned on licensed records, or only training directly on them?
• Survival. If the license ends, must synthetic items derived from seeds be deleted, and must models trained on them be retired? If the agreement is silent, assume you will be arguing about it later.
• Field-level restrictions. Can the generator see free-text fields with third-party content, such as customer messages, or only structured fields?

There is a second license layer that is easy to miss: the generator model's terms. The March 2025 Gemma Terms of Use define Model Derivatives to include models trained to behave like Gemma through its Outputs, explicitly covering distillation and synthetic data [3]. OpenAI's terms restrict using Output to develop competing models [4]. Meta's Llama 3.1 Community License permits using outputs to train other models but requires "Llama" at the start of the name of any such model that is distributed or made available [5]. Your synthetic set inherits obligations from both the seed license and the generator license, and the stricter one wins in practice.

The rights questions on derived datasets are covered in more depth in can you generate synthetic data from licensed data. Disclosure also matters downstream: California AB 2013 required developers of generative AI systems offered to Californians to post training-data documentation by January 1, 2026 [9], so record whether and how synthetic generation was used while you still have the lineage.

This page is general information, not legal advice. Confirm requirements with counsel for your jurisdiction and use case.

Privacy controls before seeds reach the generator

Seeds containing personal data must be de-identified before generation, because the generator can copy seed text verbatim into outputs and the trained model can later reproduce what it saw. Research on extractable memorization showed that adversaries can recover training data from both open and production models by querying them [7], so a phone number that survives in one seed can travel through synthetic items into model weights.

Controls that work in practice:

• Replace identifiers with typed, consistent placeholders ([CUSTOMER_NAME_1], [ACCOUNT_ID_2]) rather than deleting them, so tasks that depend on entity tracking still make sense.
• For health records, require a documented HIPAA de-identification method, either Safe Harbor or Expert Determination [8].
• Run an n-gram or substring overlap check between generated items and seeds; long verbatim spans are a signal that the generator is copying rather than transforming.
• Sample and manually review redacted seeds; no automated method catches everything.

Where seeds are sensitive enough that redaction is not sufficient, differentially private synthetic text is the stronger option, with a utility cost.

Keeping seeds and descendants out of evaluation sets

Seeds and every item generated from them belong on the training side of the split, because a synthetic training item that paraphrases an eval item inflates scores without improving the model. The failure is subtle: teams hold out a clean eval set, then reuse the same document pool as seeds, and the generator writes near-copies of eval questions.

Enforce separation at the seed level, not the item level:

1. Partition source documents into seed and eval pools before any generation runs, by document ID and by customer or account where possible.
2. Propagate seed_id to every generated item and refuse any eval item whose source document appears in the seed pool.
3. Run embedding-similarity and n-gram checks between the generated set and eval set as a backstop.

The broader mechanics are covered in contamination through synthetic data. If you are buying synthetic data rather than generating it, the same questions belong in due diligence for purchased synthetic fine-tuning data.

Seed failure modes to check before scaling generation

Most synthetic-data quality problems trace back to the seed pool, so audit it before generating at volume. The common patterns:

• Template collapse. Seeds written by one person share phrasing, and the generator amplifies it into thousands of items that start the same way.
• Happy-path bias. Exported records skew to resolved, well-formed cases; the messy tickets that define production difficulty are missing.
• Silent domain drift. Persona-only seeds let the generator substitute plausible but wrong policies, units or regulations.
• Rights mismatch. One seed batch came from a source with narrower terms, and its descendants are now mixed into the corpus without a lineage key.
• Leakage. Seeds overlap the eval pool, so offline metrics look better than production.

For how seed sets fit into a wider data strategy, the combining licensed and synthetic data guide covers mix ratios and roles, and the fine-tuning and post-training data hub links the rest of this cluster.

Sourcing licensed seed data for synthetic generation

SourceX sources operational datasets from US companies on request, including support and sales histories, engineering records, documents, finance and legal workflows, and new recordings of hands-on work, and manages the licensing process. Every dataset is rights-reviewed and delivered under a license defining records, uses, term and delivery, with personal details removed or replaced before delivery; nothing is contracted until the supplying company agrees. Describe the seed data you need at SourceX for AI data buyers.