Training Data for Omni-Modal Assistants: Spoken Guidance Grounded in a Live Camera or Screen View

What makes a recording omni-modal training data rather than video with audio

A recording becomes omni-modal training signal when speech, the visual stream the speakers refer to, and the timing between the two are all preserved and aligned. A captioned video or a transcript with a thumbnail does not teach a model when to look, when to speak or what "that one, on the left" points at. The signal lives in deictic references ("this valve", "scroll up a bit"), in corrections triggered by what the camera shows, and in the pauses while someone does what they were told.

Real-time speech models make the timing requirement concrete. Full-duplex systems such as Moshi model both speakers' audio streams at once instead of chaining voice activity detection, speech recognition, a text model and text-to-speech, a cascade the authors note adds latency and discards overlap, interruptions and other non-linguistic cues [1]. Training a listen-look-respond assistant on the same principle needs data where overlap, backchannels ("mm-hm", "okay, got it") and silences survive intact. For speech-only dialogue data, see speech-language model training data; this page covers speech grounded in a live visual scene.

Remote-assistance sessions are the closest real-world match

Remote expert assistance produces exactly the instruction-observation-correction loop omni-modal assistants are meant to perform. Research collections show the pattern: HoloAssist pairs a performer wearing a head-mounted camera with an instructor who watches the first-person view and gives spoken guidance, and it was built to study mistake detection and when an assistant should intervene [2]. Ego4D shows the scale and consent work that egocentric collection takes, with 3,670 hours from 931 camera wearers gathered with consenting participants and de-identification where needed [3]. Both are research resources with their own licenses, so check terms before any commercial training use.

Operational equivalents exist inside businesses that already run guided sessions. Typical sources include:

• Field service and equipment support: a senior engineer on a video call guiding an on-site technician through a diagnosis, often tied to a work order (see maintenance work order datasets).
• Software and IT support: screen-share sessions where an agent narrates and the user clicks, which pair naturally with support tickets with screenshots.
• Insurance and claims: video inspections where an adjuster directs a policyholder's phone camera, close to FNOL intake conversations.
• Training and onboarding: recorded sessions of a trainer coaching a new hire through a physical or software procedure.

Multi-party meetings with slides are a different signal, covered under multimodal meeting recordings. In those, the visual stream is a presentation, not the object of joint attention that the speakers act on.

Screen-grounded sessions need the same treatment as camera sessions

Screen-share recordings are omni-modal data when the speech refers to on-screen state and the user's actions change it. They overlap with computer-use agent data: benchmarks such as OSWorld evaluate agents on real desktop and web applications across Ubuntu, Windows and macOS [7], but they do not include the spoken coaching that a helper gives a human. A support session that keeps cursor position, window focus and click events alongside the audio gives you both the narration and the action trace.

Screens carry dense personal data. Email previews, CRM records, account numbers and browser tabs appear in frame for seconds at a time, so plan redaction with the guidance in PII in screen recordings and agent trajectories. If you also need coding-specific sessions, developer session recordings are a related but narrower source.

Specify timing, channels and sync before anything else

The capture format decides whether a recording can train a real-time model, and it cannot be repaired afterward. Many conferencing platforms record a single mixed track by default, and many post-processing tools trim silences or normalize speed; either change destroys the turn-taking signal. Ask for these properties up front:

• Per-speaker audio: separate tracks (multitrack WAV or FLAC, 16 kHz or higher) rather than one mixed MP4 audio stream, so overlap and barge-in can be labeled.
• Original timing: no silence removal, no speed change, no stitched clips; dead air while a technician works is training signal.
• Visual stream at native rate: the camera or screen feed with its original frame timestamps, plus any resolution or frame-rate drops caused by bandwidth, which an assistant will face in deployment.
• A shared clock: a session start time and per-track offsets so audio and frames align within a known tolerance, documented per session.
• Event logs: annotations, pointer or AR markers, screen-share start and stop, and camera switches, timestamped on the same clock.
• Outcome labels: whether the issue was resolved, escalated or required a follow-up visit, ideally joined from the ticket or work order.

Illustrative example: invented to show structure; it does not describe an available dataset.

{
  "session_id": "ra-2026-000417",
  "setting": "remote_field_assistance",
  "duration_s": 1312.4,
  "clock": {"reference": "session_start_utc", "max_av_skew_ms": 40},
  "tracks": [
    {"track": "audio_expert", "format": "flac", "sample_rate_hz": 16000, "offset_ms": 0},
    {"track": "audio_technician", "format": "flac", "sample_rate_hz": 16000, "offset_ms": 12},
    {"track": "video_technician_rear_camera", "codec": "h264", "fps_nominal": 30, "frame_timestamps": "vfr_pts.csv"}
  ],
  "events": [
    {"t_ms": 84120, "type": "ar_marker", "by": "expert", "label": "circle"},
    {"t_ms": 84390, "type": "utterance_ref", "speaker": "expert", "text_span": "that fitting, top left", "target": "marker_1"}
  ],
  "silence_trimmed": false,
  "outcome": {"resolved_remotely": true, "linked_work_order": "WO-REDACTED"},
  "deidentification": {"faces": "blurred", "serials": "masked", "voices": "unaltered_with_consent", "method_log": "deid_v3.json"}
}

Transcripts are useful but secondary. Word-level timestamps per speaker let you derive turn boundaries, but a transcript produced by a cascade ASR system should be labeled as machine-generated, with its model and version noted, so it is not mistaken for ground truth.

Consent, bystanders and voices are the main rights risks

Camera and screen sessions capture people and places the original consent may not cover, so rights review has to run frame by frame and party by party. Recording rules differ by state; California, for example, prohibits recording a confidential communication without the consent of all parties [4], and many remote sessions cross state lines. A recording made with a "this call may be recorded for quality" notice does not by itself show permission to license it for model training; ask what the notice and the service terms actually said.

Biometric law adds a second layer. Illinois BIPA requires written notice and a written release before collecting biometric identifiers such as face geometry or voiceprints, plus a published retention schedule [5]. Whether a given training use involves a biometric identifier is a question for counsel, but the safer specification is clear: blur faces of bystanders and unconsenting customers, and decide explicitly whether speaker voices are kept, transformed or limited to consenting staff.

Frames also leak business information. Customer site interiors, equipment serial numbers, asset tags, whiteboards, license plates and documents on desks show up in field video, and they can identify a customer even after names are removed. Multimodal de-identification has to treat face, voice, screen and metadata together, as covered in de-identifying multimodal records.

This page is general information, not legal advice. Confirm requirements with counsel for your jurisdiction and use case.

Rightsholders in a single session can be several parties

One remote-assistance recording can involve the platform operator, the employer of the expert, the customer whose equipment and premises appear, and the end user on the call. The business that holds the recording may control the file without holding every right a training license needs. Map who owns the recording, who appears in it, and what each party agreed to before negotiating uses; the pattern is set out in licensing multimodal records from several rightsholders.

A buyer checklist for omni-modal session data

A short checklist catches the defects that make grounded-session data unusable after delivery. Use it in your request and again on the sample.

Illustrative example: invented to show structure; it does not describe an available dataset.

For documentation, a machine-readable format such as Croissant with its responsible-AI extension can record collection method, labeling and participant information alongside the files [6]. If you are building a held-out benchmark rather than a training set, keep the eval split from suppliers and sessions that never enter training, as discussed in private multimodal evaluation sets. For drafting the full request, start from the multimodal data specification template, and see the multimodal training data hub for related categories.

Delivery and volume shape the pipeline more than file count

Grounded session data is large and sensitive, so delivery should run through private, access-controlled storage rather than links or attachments. A common market pattern is a scoped cross-account grant on a cloud bucket, where the data owner's policy allows a buyer account specific operations on a delivery prefix [8]. Plan for video-heavy volume, per-session manifests with checksums, and a re-delivery path if de-identification is redone.

How SourceX approaches grounded session data

SourceX sources operational datasets from US companies on request and manages the licensing and ongoing purchase process; it does not hold this category in stock, and a request does not guarantee a match. Buyers describe the data they need, such as remote-assistance sessions with per-speaker audio and the shared camera or screen view, and SourceX looks for US businesses that hold it. The kinds of data SourceX works with include support histories and new recordings of hands-on work, and it does not source scraped web content or generic CCTV. You can describe a request on the SourceX buyers page.

The process runs Find, Assess (the data and its licensing permissions), Agree (pricing and allowed uses in a license), Transact and Manage, and nothing is contracted until a supplier agrees. Each dataset is rights-reviewed for ownership and consents, personal details such as names, emails, phones and account numbers are removed or replaced before delivery with the method recorded and a sample checked, and no method is perfect. Delivery runs through private, access-controlled workflows only after an executed agreement and supplier approval.

Request omni-modal assistant training data

If your team needs real spoken guidance grounded in a camera or screen view, describe the sessions, channels, timing and consent scope you require. SourceX serves AI teams wherever they are based and will look for US businesses that hold matching data, with every release approved by the supplying company. Start a request at https://sourcex.si/buyers.