API and Streaming Feeds vs Batch Files for Ongoing Data Purchases

This page is about the delivery channel and cadence. Whether each delivery is a full snapshot or only the changes is a separate decision covered in incremental deliveries vs full refreshes. For the wider delivery topic, start at the dataset delivery hub.

Freshness need decides the channel, not technical preference

Pick the slowest cadence that still meets the model's real freshness requirement, because every step toward real time adds operational and contractual surface. A continual fine-tuning run that retrains monthly gains nothing from a sub-second stream; it needs a clean, versioned increment each cycle. A retrieval-augmented generation (RAG) index over support tickets or product documents may need hours-level freshness, because answers go stale when a policy changes. An evaluation set refreshed to track drift usually needs a frozen, versioned cut, which a stream cannot give you without extra work.

Ask the team that consumes the data to state freshness as a number: "records must be queryable within 24 hours of creation at the supplier" is testable; "real time" is not. If the honest answer is a day or longer, batch wins. For how often buyers typically want refreshes, see how often AI buyers want fresh data.

What batch file delivery gives you

Batch files are the easiest channel to audit because each delivery is a closed, countable object you can hash, sign off and retain or delete as a unit. A typical delivery is a dated prefix in an object store or SFTP drop containing Parquet or JSONL data files, a manifest with row counts and SHA-256 checksums, and a schema version. Parquet files are self-describing: each begins and ends with the PAR1 magic number, and footer metadata records the location of every column chunk, so a truncated upload fails fast [1].

The strengths compound over a long purchase. You can pin a model or eval to delivery_2026-10-01, diff two deliveries, rerun quality checks, and prove to an auditor exactly which records entered a training run. The verification steps are covered in dataset manifests and checksums, and landing patterns in cross-account cloud bucket delivery.

The failure modes are well known: late or skipped drops that nobody notices, a schema change slipped into one file, and "full refreshes" that silently drop rows. All are caught by a manifest check that runs before ingestion. If you are unsure whether a purchase can be structured as discrete drops at all, see can I license data in batches.

What API, webhook and streaming feeds add

Feeds add latency in minutes or seconds, but they move integrity work from the supplier's packaging step onto your consumer code. There are four common feed shapes, and each has its own failure profile.

• Pull API. You poll an endpoint such as GET /records?updated_since= with a cursor. Watch for clock skew on updated_since, pagination that reorders under concurrent writes, and rate limits that make backfills slow.
• Webhooks. The supplier pushes each event to your HTTPS endpoint. Under the widely used Standard Webhooks pattern, each delivery carries a webhook-id that stays constant across retries, so you can use it as an idempotency key, plus a timestamp that receivers check against a short tolerance window. The signature is an HMAC over the ID, timestamp and raw body [9], verified against the raw bytes rather than re-serialized JSON.
• Change data capture (CDC). Tools such as Debezium read a database's transaction log and emit row-level change events describing the before and after state of each row [2], typically published to Kafka topics [3]. CDC is precise, but it exposes the supplier's internal schema, which then needs de-identification before it leaves their environment.
• Kafka or similar logs. A topic gives ordered, replayable events per partition. With log compaction, Kafka keeps only the latest value per key, and a deletion is a tombstone (a null value for the key) that the broker removes after delete.retention.ms; a slow consumer that misses that window never sees the delete.

Source systems often already expose event histories. Zendesk's ticket audit API, for example, records each property change as a Change event with field_name, value and previous_value [8]. That history is useful training signal, but it means a feed of "tickets" is really a feed of edits, and you need to decide which one you are licensing.

Data shares sit between files and feeds

Live data shares give you near-current data without running a stream, and they often fit retrieval refresh better than either extreme. Delta Sharing is an open protocol that serves Delta Lake and Parquet tables from cloud object storage over HTTP/REST [4]. Recipients authenticate with a bearer token from a profile file that may carry an expiration time, and data is organized as share, schema and table [5]. Snowflake Secure Data Sharing gives the consumer read access to provider objects without copying them into the consumer account [6]; across regions, listings are fulfilled by replication on a refresh schedule [7], so "live" is still periodic.

The trade-off is control. With a share, the supplier can change or withdraw the table under you, so you still need to snapshot what you train on. Table formats as a delivery mechanism are covered in Iceberg and Delta Lake tables as a delivery format.

Minimum requirements before accepting a feed

A feed is only acceptable for licensed training or retrieval data if it is idempotent, ordered within a key, replayable, and carries deletions as first-class events. Without these, you cannot reconstruct what you hold, and your copy diverges from the supplier's.

Illustrative example: invented to show structure; it does not describe an available dataset.

Feed acceptance checklist

An illustrative event envelope shows the fields in practice:

Illustrative example: invented to show structure; it does not describe an available dataset.

{
  "event_id": "evt_01J9ZQ8X4K",
  "record_id": "tkt_7f3a91",
  "op": "update",
  "sequence": 48211,
  "schema_version": "2.3",
  "emitted_at": "2026-10-08T14:02:11Z",
  "license_ref": "LIC-EXAMPLE-01",
  "payload": { "status": "resolved", "resolution_code": "refund_partial" }
}

Stable identifiers are covered in depth in stable record IDs and join keys, and delete handling in propagating deletions and corrections.

Streams do not change the license or the audit trail

Data that arrives as a stream is governed by the same license as a file: the records in scope, allowed uses, term, and deletion obligations still apply to every event you persisted. The practical difficulty is that streamed data spreads fast, into topic retention, consumer caches, feature stores, vector indexes and training shards. If the license requires deletion at term end or on a withdrawal event, you need lineage from event_id to every downstream copy, including embeddings.

For that reason, many teams land every feed into an append-only raw table, cut a dated snapshot on a fixed schedule, and train or evaluate only from snapshots. That gives you the freshness of a feed for retrieval and the auditability of batch for training. Post-delivery controls are covered in access controls for licensed training data, and contract structure for refreshes in ongoing data supply agreements.

A decision rule you can apply this quarter

Default to batch, upgrade to a share for hours-level freshness, and accept an event feed only when a documented use needs minutes or seconds and the supplier meets the checklist above. Write the choice into your technical delivery specification: channel, cadence, file or event schema, reconciliation method, and how deletions travel.

Remember the supply side too. Many operational datasets sit in ticketing, CRM, ERP or document systems where the supplier's team exports on a schedule; asking a small company to run a 24/7 Kafka endpoint can be the difference between a deal and no deal. Text-specific cadence questions are covered in recurring text data feeds. If you are scoping an ongoing purchase now, you can describe the refresh you need to SourceX.

Sourcing an ongoing data feed or batch supply

SourceX sources operational datasets, such as support and sales histories, engineering records and documents, from US companies on request and manages the commercial process, including licensing agreements and ongoing purchases. Every release is approved by the supplying company, personal details are removed or replaced before delivery, and delivery runs through private, access-controlled workflows only after an executed agreement. Describe the data, cadence and use you need at sourcex.si/buyers.