Synthetic vs Real Video Training Data: Simulation, Game Engines and Generated Clips

Three kinds of synthetic video, three different failure modes

The three sources of synthetic video fail differently, so evaluate them separately rather than as one "synthetic" bucket. Physics simulators (robotics stacks, driving simulators) give you exact state, perfect labels and repeatable scenarios. Game-engine pipelines add photoreal rendering, crowds and weather on top of that. Generative video models produce realistic-looking clips from text or image prompts, but they give you no ground-truth state at all.

• Simulation. Open-source driving simulators such as CARLA exist precisely to train and validate urban driving systems under scripted scenarios. You get per-pixel semantic labels, depth, instance IDs and actor trajectories for free. The failure mode is behavioral: scripted agents follow rules that real drivers, pedestrians and workers do not.
• Game-engine rendering. Engine scenes give you control over lighting, materials and camera intrinsics. The failure mode is appearance and motion statistics: shader artifacts, overly clean surfaces, motion-captured or keyframed human motion that lacks the hesitation and fumbling of real work.
• Generated clips. Text-to-video and image-to-video models produce plausible footage fast. The failure modes are temporal (object permanence breaks, hands merge with tools, counts drift across frames), the absence of trustworthy labels, and the generator's own license terms.

For a deeper map of where real video comes from and what to check, start at the video data hub.

Where the sim-to-real gap shows up in video models

The sim-to-real gap in video is a gap in both appearance and behavior, and the behavior half is the harder one to close. Appearance gaps (texture, lighting, lens blur, rolling shutter, compression) can often be narrowed by randomization. Domain randomization deliberately varies textures, lighting and camera pose, including non-realistic textures, on the theory that with enough variation reality looks like one more variant.

Behavior is different, because no amount of texture randomization changes how a simulated person moves. Action recognition depends on timing, pauses, partial actions, occlusions by the actor's own body, and the long tail of how people actually do a task. A synthetic "pick from shelf" clip rarely contains the regrasp, the dropped item or the coworker walking through frame that dominate real warehouse footage.

Common gap symptoms to look for in a pilot:

• Codec and sensor mismatch. Rendered frames are noise-free and losslessly encoded, while deployment footage is H.264 or H.265 at variable bitrate with motion blur, low-light grain and dropped frames.
• Camera placement bias. Synthetic cameras sit at idealized heights and angles; real fixed cameras are mounted wherever an electrician could reach.
• Temporal statistics. Action durations in synthetic data cluster near the scripted value; real durations have heavy tails.
• Class co-occurrence. Simulated scenes rarely reproduce real correlations, such as which tools appear with which steps.

What generated video adds and what it carries with it

Generated video adds visual diversity cheaply, but it inherits the generator's artifacts, its training distribution and its output terms. Three checks matter before generated clips enter a training set.

First, label validity: a prompt says "person tightens a bolt," but the clip may show the wrong tool, a hand passing through the part or a step that never completes. Someone has to verify each clip against its intended label, which erodes the cost advantage. Second, contamination: if the generator was trained on a public benchmark's footage, generated clips can leak benchmark-like content into training, a risk covered in contamination through synthetic data.

Third, the generator's terms. Model licenses can reach into what you train with their outputs. The Gemma Terms of Use (24 March 2025 version), for instance, define Model Derivatives to include models trained to behave like Gemma by transferring patterns from its Outputs, explicitly covering distillation and training on synthetic data Outputs generated by Gemma [1]. Video generators have their own terms, and they differ, so record the generator, version, terms snapshot, prompts and seed data for every batch, as described in provenance records for synthetic training data.

When synthetic video can carry most of the training load

Synthetic video can carry most of the training load when the target is geometric or physical and the deployment camera is controllable. Good candidates include:

• Dense prediction pretraining: optical flow, depth, surface normals and instance segmentation, where real labels are nearly impossible to annotate by hand.
• Rare and unsafe events: near-miss collisions, falls, equipment failures and edge-case weather in driving and industrial settings.
• Viewpoint and lighting coverage for a fixed task, where you can render many camera placements and illuminants.
• Robot policy learning in simulation, followed by fine-tuning on real demonstrations (see human demonstration video for robot learning).

It carries much less when the target is human behavior in context: action recognition on real workflows, procedural step detection, mistake detection, driver monitoring or anything evaluated on footage from specific real sites. For those, licensed real footage with good temporal action segmentation labels is the core, and synthetic clips fill specific gaps.

Real licensed footage as the evaluation anchor

Even when training is mostly synthetic, evaluation should rest on real footage that the model has never seen and that matches deployment conditions. A synthetic test set measures how well the model fits the simulator; it does not measure transfer. The case is laid out in synthetic evaluation data limits and, for the validation workflow, in using a licensed real-data holdout.

Real test sets need their own quality controls. Northcutt and colleagues estimated an average label error rate of at least 3.3% across the test sets of 10 widely used benchmarks [3], so double-annotate the real holdout and adjudicate disagreements before trusting small metric differences. Public action benchmarks are useful references for class design; Kinetics, for example, defines 400 human action classes with at least 400 clips each [2]. But public benchmark clips are rarely your deployment domain, which is why teams license real footage from the environments where the model will run.

Real footage brings its own rights layers. Faces and bodies may trigger biometric law: Illinois BIPA defines biometric identifiers to include scans of face or hand geometry and requires a published retention schedule for biometric data [5]. Check footage ownership, worker or bystander consent, audio and on-screen content before licensing; rights layers in a video clip breaks these down, and dashcam fleet footage covers a common driving case.

This page is general information, not legal advice. Confirm requirements with counsel for your jurisdiction and use case.

Finding the mix ratio with a staged pilot

The right synthetic-to-real ratio is found empirically for each task, not set by rule. Run a controlled sweep against a fixed real holdout, change one variable at a time and keep the compute budget constant so a larger synthetic share is not confused with more training steps.

Illustrative example: invented to show structure; it does not describe an available dataset.

Read results per class and per condition, not only as a top-line metric. A synthetic share that improves rare-event recall while degrading common-action precision on night footage is a different decision from one that helps everywhere. Report per-site results, since a gain on one site's camera layout can hide a loss on another.

Track at least these fields for each arm so the result is auditable later:

arm_id: C
real_sources: [license_id, site_ids, capture_dates, camera_models, codec]
synthetic_sources:
  - kind: simulator | game_engine | generative_model
    tool_and_version: "<name and version>"
    terms_snapshot: "<URL or file hash, date captured>"
    randomization: [textures, lighting, camera_pose, actor_behavior]
    prompts_or_scenarios: "<file reference>"
label_source: simulator_ground_truth | human_verified | prompt_only
holdout_id: real_holdout_v3 (frozen)
metrics: [mAP_per_class, recall_rare_events, per_site_accuracy]

Documenting these choices in a datasheet or data card, covering upstream sources, collection and annotation methods and intended use, keeps the mix decision explainable to reviewers [4].

Specifying the real footage you still need

Once the pilot shows where synthetic falls short, specify real footage by the gap rather than by volume. Useful fields for a request: target actions and the failure modes synthetic missed, camera placement and resolution, codec and frame rate, lighting conditions, number of distinct sites and actors, clip length, annotation scheme and whether faces, voices or screens appear.

This is the kind of operational footage SourceX sources. It sources operational datasets from US companies, including new recordings of hands-on work, and manages the commercial process from licensing through ongoing purchases; see the buyer request page for how to describe the data. Data is sourced on request rather than held in stock, so a request does not guarantee a match. Every dataset is rights-reviewed for ownership and consents and delivered under a license that defines records, uses, term and delivery.

For broader context on combining sources, see combining licensed and synthetic data, the comparison of licensed, synthetic and scraped training data, and the glossary entry on synthetic data. The AI data hub lists related guides for other modalities.

Sourcing real video to anchor a synthetic data mix

If your pilot shows you need real footage for training gaps or a deployment-matched holdout, describe the data and SourceX will look for US businesses that hold it, with every release approved by the supplying company. Personal details are removed or replaced before delivery, the method is recorded and a sample is checked, though no method is perfect. Describe the real video data you need.