Domain Captions from Work Records: Using Inspection Notes and Adjuster Comments as Image Text

This guide sits in the image data cluster and focuses on text that originates in work systems, not captions written for a dataset. If you need open or CC-licensed captioned sets instead, start with image and caption datasets you can train on commercially.

Why work-record text beats generic captions for domain VLMs

Expert notes encode the judgment a domain model must learn, which generic captions never contain. A web caption says "a roof with shingles"; a field adjuster writes "hail strikes on south slope, approx. 8 per test square, granule loss, recommend full slope replacement." That sentence carries a location on the object, a measurement protocol, a severity call and an action.

Converting paired images and text into instruction-style examples is now a common route to domain VLMs. Published examples show the pattern with domain labels: a garment set built for VLM fine-tuning pairs images with structured attributes [1], damage-assessment pipelines pair vehicle photos with expert-derived repair cost labels [2], and medical researchers convert specialist reports into visual instruction tuning data [3].

Operational text also arrives with outcomes. A claim note links to an approved estimate, a work order links to parts consumed and a close-out code, and an inspection finding links to a pass, fail or re-inspect status. Those downstream fields let you build targets a caption writer would never think to produce. For the text-only side of this pattern, see report-generation fine-tuning data.

Where image-plus-expert-text pairs live in operational systems

Most usable pairs sit in four record types, each with a different link between photo and text. Knowing which one you are buying tells you how much alignment work remains.

• Insurance claim files. Field and desk adjusters attach photo sets to a claim and write diary notes, while estimates (commonly built in tools such as Xactimate for property or estimating platforms for auto) itemize line items by room, slope or panel. See insurance claims datasets.
• Inspection reports. Home, roof, elevator, bridge and equipment inspections usually embed photos next to numbered findings, which gives the cleanest photo-to-sentence link. See inspection reports for AI training.
• Maintenance and field-service work orders. CMMS and field-service systems store technician notes, failure codes and before/after photos per work order. See maintenance work order datasets and before-and-after image pairs.
• Quality and nonconformance records. Manufacturing NCRs and supplier corrective actions pair defect photos with disposition text (use as is, rework, scrap).

Photo-only holdings are covered on our images and inspection photos page. When the text is itself a scanned form or PDF report, the pairing problem becomes a document-understanding problem; see training data for multimodal document models.

Aligning one note to the right photo

The hard part is that one note usually describes several photos, and some photos are never mentioned. A claim diary entry may summarize 40 images in one paragraph, so naive pairing teaches the model to hallucinate details that are not in the frame it sees.

Ask the supplier how the link was made and grade each pair by alignment tier:

• Tier A, explicit: the photo carries its own caption field in the inspection app, or the report embeds the image directly beside a numbered finding.
• Tier B, referenced: the note cites a photo number, filename or estimate line ID ("see photo 12", "line 4 R&R ridge cap").
• Tier C, inferred: the link comes from capture timestamp windows, GPS proximity or photo order relative to note time.
• Tier D, record-level: text applies to the whole set only; useful for set-level classification or retrieval, weak for grounded captioning.

Keep tier C and D pairs out of your evaluation split. For training, many teams use tier A and B text as targets and treat tier D text as a multi-image context input rather than a per-image caption.

Illustrative example: invented to show structure; it does not describe an available dataset.

{
  "pair_id": "p_000184",
  "record_type": "property_claim",
  "record_key": "rec_7f3a",
  "image_ref": "img/rec_7f3a/IMG_0012.jpg",
  "image_role": "close_up",
  "capture_time_offset_s": 312,
  "alignment_tier": "B",
  "alignment_evidence": "note cites 'photo 12'",
  "source_text": "Hail strikes south slope, approx 8 per test square, granule loss, functional damage.",
  "text_author_role": "field_adjuster",
  "structured_labels": {"component": "roof_shingle", "peril": "hail", "severity": "functional", "action": "replace_slope"},
  "outcome_fields": {"estimate_line_ids": ["L4", "L5"], "decision": "approved"},
  "redaction": {"text_method": "ner_plus_regex_plus_review", "image_method": "face_plate_housenumber_blur", "exif_stripped": true},
  "split_group": "rec_7f3a"
}

The split_group field matters: assign train, validation and test splits by record, not by image, or near-identical photos of the same loss will leak across splits.

Turning notes into training targets and evaluation rubrics

Raw notes are rarely good captions, so plan a conversion step and keep the original text alongside it. Field shorthand ("R&R", "TSq", "NFD"), templated boilerplate and notes written after the repair all need handling before they become supervised fine-tuning targets.

Common target formats built from the same record:

1. Grounded caption: a cleaned sentence describing only what is visible in the aligned photo.
2. Visual question answering: questions derived from structured fields ("What peril caused this damage?", "Is this functional or cosmetic?").
3. Structured extraction: the image maps to a JSON object of component, condition, severity and action, matching the downstream schema.
4. Report drafting: a multi-image input produces the finding paragraph, scored against the expert's note.

Quality beats volume for this kind of fine-tuning; LIMA showed a model tuned on 1,000 carefully curated examples performing competitively [9]. A few thousand tier A pairs reviewed by a domain expert can be worth more than a large pool of record-level text.

Expert notes also make strong evaluation rubrics. Hold out a set of records, extract the checkable facts from each note (component, severity, recommended action, any measurement) and score model outputs fact by fact instead of with caption-overlap metrics. Track disagreement too: two adjusters can call the same photo differently, and that rate is the ceiling your model can reasonably hit. The training data quality guide covers contamination and coverage checks in more depth.

De-identifying both the text and the pixels

Work-record pairs leak personal data through two channels, so redaction has to cover each. Web-scale image-text collections have already been shown to carry personal information at scale [4]; operational records concentrate it, because every claim and work order belongs to a named customer at a real address.

In the text, look for names, street addresses, claim and policy numbers, VINs, license plates, phone numbers, email addresses and account numbers, plus indirect identifiers such as "the corner unit next to the church." Rule-based and model-based redactors both miss things, and LLM-based redaction has its own measurable failure modes [5], so ask for the method used and the recall measured on a reviewed sample.

In the image, the usual targets are faces, plates, house numbers, mail and documents visible in frame, screens, and EXIF or XMP metadata carrying GPS coordinates, device serials and timestamps. Our guide to EXIF metadata in image training data covers what to strip and what to keep.

Legal regimes differ by content. Illinois BIPA treats scans of face geometry as biometric identifiers while excluding photographs themselves from that definition [6], so the risk depends on what your pipeline does with faces. Images and notes that are protected health information held by a HIPAA covered entity or business associate must be de-identified by Safe Harbor or Expert Determination before release, and Safe Harbor lists full-face photographs and comparable images among the identifiers to remove [7]. People, private property and logos may also need releases; see model and property releases for AI training images and the broader de-identified data buyer's guide.

This page is general information, not legal advice. Confirm requirements with counsel for your jurisdiction and use case.

Rights, provenance and memorization risk

Operational photos often have several possible rights holders, so confirm who can license each one before price discussions. A claim file can mix adjuster-taken photos, policyholder uploads, contractor photos and third-party aerial imagery, each under different terms.

Documentation is where public datasets most often fail. An audit of popular dataset hosting sites found license information missing for more than 70% of datasets and license errors above 50% [11], and large captioned image sets are now promoted specifically as commercially usable [12]. Ask for a Data Card style record of upstream sources, collection and annotation methods and intended use [10].

Memorization is a real concern for this data. Image models can regenerate individual training images, including photos of people and trademarked logos [8], so redaction should happen before training, not as an output filter. If you fine-tune a general-purpose model with acquired data, also check whether you take on provider or developer duties; see fine-tuning with acquired data under the EU AI Act and AB 2013.

Buyer checklist for expert-captioned image data

Use this checklist when describing the data you need or reviewing a supplier's sample.

Illustrative example: invented to show structure; it does not describe an available dataset.

How SourceX approaches work-record image data

SourceX sources operational datasets from US companies and manages the commercial process, including licensing agreements and ongoing purchases. The data kinds include engineering records, documents, finance and legal workflows, support and sales histories, and new recordings of hands-on work. Data is sourced on request rather than held in stock, so a request does not guarantee a match, and SourceX does not source scraped web content or generic CCTV or photos.

You describe the data (record type, photo roles, text author, alignment tier, volume range) rather than naming businesses, and SourceX looks for US businesses that hold it. Every dataset is rights-reviewed for ownership and consents, personal details such as names, emails, phones and account numbers are removed or replaced before delivery with the method recorded and a sample checked, and no method is perfect. Health records require HIPAA de-identification. You can describe your VLM data request to SourceX using the checklist above.

Request domain image-text pairs for VLM fine-tuning

The process runs Find, Assess, Agree, Transact and Manage, nothing is contracted until a supplier agrees, and each release is approved by the supplying company and delivered under a license defining records, uses, term and delivery. SourceX serves AI teams wherever they are based, and its own dataset rights, including any training use, are set out in the signed supplier agreement. Start a buyer request at SourceX.