CSV Deliveries Done Right: RFC 4180, Encoding and Embedded Text

CSV looks simple because the happy path is simple. Exports from CRM, ticketing, ERP and billing systems are not the happy path: ticket bodies contain quoted email threads, notes fields carry pasted spreadsheets, and every system's export button has its own idea of encoding. This page is the specification a receiving data engineer should hand the supplier before the first file is written. It sits under the delivery formats and schemas hub, and pairs with the format decision covered in Parquet vs JSONL for licensed training data.

What RFC 4180 actually guarantees, and what it leaves open

RFC 4180 gives you a quoting grammar, not a complete contract. It says each record sits on its own line ending in CRLF, that a header line is optional, that fields containing commas, double quotes or line breaks must be enclosed in double quotes, and that a double quote inside a quoted field is escaped by doubling it ("") [1]. It also registers the text/csv media type with optional charset and header parameters [1].

What it does not settle matters more in practice. RFC 4180 is an Informational RFC, not a Standards Track specification, and it was written to document existing behavior rather than to define one [1]. It does not mandate an encoding, says nothing about null versus empty string, does not cover date or number formats, and treats spaces as part of a field. An IETF Internet-Draft revising it (rfc4180-bis) has circulated, but as of October 2026 it remains a draft, so do not cite it as the governing standard in a license or data contract [2].

For machine-readable dialect and column metadata, the W3C CSV on the Web (CSVW) Recommendations (2015) define a tabular data model and a JSON metadata vocabulary that can describe delimiter, quote character, header rows, column datatypes and null tokens next to the file. You do not need a full CSVW toolchain, but borrowing its vocabulary for the sidecar keeps your specification precise.

Why embedded newlines in free text break pipelines

Embedded newlines are legal CSV and the most common cause of silent corruption. A Zendesk or ServiceNow ticket description, a Salesforce case comment or an email body almost always contains line breaks; RFC 4180 handles this by quoting the field [1]. A compliant parser such as Python's csv module, pandas, DuckDB read_csv or Spark with multiLine=true reads that field correctly.

The failures happen in the line-oriented tools between export and parser. wc -l overcounts records, split -l and head cut records in half, grep returns fragments, and Spark with the default multiLine=false splits files on newline boundaries and produces misaligned columns. Any shard boundary chosen by byte offset or line number can land inside a quoted field.

Three further failure modes recur with business-system exports:

• Unescaped quotes. Exporters that wrap fields in quotes but write an inner " as \" (backslash escaping) produce files RFC 4180 parsers misread. Specify doubling, not backslashes.
• Mixed line terminators. CRLF between records and bare LF inside text is fine when quoted, but a bare CR from older Mac-origin text or a pasted spreadsheet can be read as a record break by permissive readers.
• Formula prefixes. Cells beginning with =, +, - or @ are interpreted as formulas when a reviewer opens the file in a spreadsheet application. Tell suppliers not to "sanitize" these with a leading apostrophe, which alters training text; instead, require reviewers to open samples as text.

The practical rule: count records with a CSV parser, never with wc -l, and shard only on record boundaries.

Encoding, BOM and Unicode normalization for text fields

Require UTF-8, state the BOM policy explicitly, and normalize text to NFC. UTF-8 should be the only accepted encoding; reject Windows-1252, Latin-1 and UTF-16 exports at intake rather than guessing, because a mis-decoded é or curly quote becomes permanent mojibake in training text.

BOM policy is a choice, not a default. Many spreadsheet tools write a UTF-8 BOM (EF BB BF) to signal encoding, and some readers then treat it as part of the first header name, giving you a column called ticket_id. Pick one rule (we suggest "no BOM", matching the JSON Lines convention that forbids it [4]) and make your intake strip-and-log any BOM it finds.

Normalization matters for deduplication, tokenization and PII matching. Unicode Standard Annex #15 (UAX #15) defines four normalization forms; NFC applies canonical decomposition followed by canonical composition, so canonically equivalent strings such as a precomposed é and e plus a combining accent get one binary representation. Avoid NFKC for training text unless you intend it: it also folds compatibility characters such as full-width letters and ligatures, which changes content. Apply the same form before running de-identification, so a name in decomposed form is not missed by a matcher built on composed strings; see de-identifying tabular data and free-text columns.

Control characters need an explicit rule too. Allow tab, LF and CR inside quoted text fields; reject or replace NUL (0x00) and other C0 controls, which crash some loaders and truncate strings in C-based tools.

Delimiters, nulls, numbers and dates

Declare every value convention in writing, because CSV has no types. Comma is fine as a delimiter when quoting is correct; switching to tab or pipe does not remove the need for quoting, since free text can contain those too. If a supplier's tool cannot quote reliably, that is a signal to change format, not delimiter.

Nulls are where most semantic errors start. An empty unquoted field, an empty quoted field (""), the string NULL, \N (the MySQL export default and the PostgreSQL COPY text-format default) and N/A can all mean different things. Require one null token, state that "" means empty string, and document it in the data dictionary for the delivery.

For numbers, require a dot decimal separator with no thousands separators, and send currency amounts as decimal strings with a separate ISO 4217 currency column rather than locale-formatted values like 1.234,56 €. For dates and times, require ISO 8601 with an explicit offset or UTC (2026-03-14T09:26:53Z), and never 03/04/2026, which is ambiguous between US and European exports. Booleans should be true/false, not Y, 1 or X, which ERP exports mix freely.

The CSV delivery specification to hand your supplier

The specification below is the artifact to attach to the technical schedule of a deal; adapt values to your stack. It slots into the broader technical delivery specification template.

Illustrative example: invented to show structure; it does not describe an available dataset.

A matching manifest entry might look like this:

Illustrative example: invented to show structure; it does not describe an available dataset.

{
  "file": "support_tickets_part-0003.csv.gz",
  "sha256": "9f2c...e41a",
  "bytes": 184220193,
  "records": 412806,
  "columns": 14,
  "encoding": "utf-8",
  "bom": false,
  "normalization": "NFC",
  "null_token": "",
  "dialect": {"delimiter": ",", "quote": "\"", "escape": "doubled", "line_terminator": "\r\n"}
}

The records value must come from a CSV parser on the supplier side, not a line count. Checksums and byte sizes are covered in more depth in dataset manifests and checksum verification.

Intake checks that catch broken CSV before training

Run a parser-based intake gate on every file before anything lands in your feature store or training mix. The minimum checks:

1. Verify SHA-256 and byte size against the manifest.
2. Decode strictly as UTF-8 (errors="strict"); fail on any invalid byte sequence. Check for and log a BOM.
3. Parse with a strict RFC 4180 reader and assert the parsed record count equals the manifest records value. A shortfall usually means an unescaped quote swallowed many rows into one field.
4. Assert every record has the header's column count.
5. Check that text fields are already NFC (unicodedata.is_normalized("NFC", s) in Python 3.8+) and contain no disallowed control characters.
6. Profile field lengths: a single field that is orders of magnitude longer than its column's typical value is the classic symptom of a runaway quote.
7. Type-check dates, decimals, booleans and the null token against the data dictionary.

Steps 4, 5 and 7 overlap with the general checks in validation checks for structured dataset deliveries; the CSV-specific part is steps 2, 3 and 6. Quarantine failing files and return the exact record offset to the supplier instead of patching them yourself, so the fix lands in the export job and does not recur in the next recurring delivery.

When to require Parquet or JSON Lines instead of CSV

Require another format when the data stops being flat, short and typed-by-convention. CSV remains reasonable for narrow, mostly numeric or categorical tables such as invoice headers, order lines or opportunity stage histories. It becomes a liability under any of these conditions:

• Long free text dominates. Ticket threads, call notes and email bodies are where quoting breaks. JSON Lines encodes newlines and quotes as JSON escapes on one physical line per record, and requires UTF-8 without a BOM [4].
• Nested or repeated fields. Ticket comments, line items, tags and attachment lists flattened into delimited strings inside a cell (tag1;tag2) lose structure. Use JSON Lines for document-shaped records, or Parquet with nested types.
• Types matter. Parquet stores a schema in file metadata alongside columnar data, so decimals, timestamps and nulls are typed rather than conventional [3].
• Scale. Once a single table runs to tens of gigabytes or hundreds of millions of rows, columnar compression and column pruning in Parquet usually reduce both storage and scan cost; treat this as a rule of thumb to benchmark, not a threshold.
• Repeated quoting failures. If two deliveries in a row fail step 3 above, change format rather than iterating on the exporter.

A common compromise: structured columns in Parquet, free-text bodies in JSON Lines keyed by a stable record ID. For a short overview of what buyers generally accept, see what file formats AI buyers accept.

How SourceX handles tabular exports from business systems

SourceX sources operational datasets from US companies, including support and sales histories, finance workflows and engineering records, and manages the commercial process through licensing and ongoing purchases. Every dataset is rights-reviewed and delivered under a license that defines the records, uses, term and delivery, and personal details such as names, emails, phones and account numbers are removed or replaced before delivery, with the method recorded and a sample checked; no method is perfect. Delivery runs through private, access-controlled workflows, never email attachments, and only after an executed agreement and supplier approval. Data is sourced on request rather than held in stock, so a request does not guarantee a match. If you already have a format specification like the one above, bring it when you describe the data you need to SourceX.

Request tabular business-system data under a defined delivery spec

SourceX sources operational datasets such as support and sales histories and finance workflows from US companies on request, and every release is approved by the supplying company. Each dataset is rights-reviewed and delivered under a license that defines records, uses, term and delivery. Describe the tabular data you need on the SourceX buyers page.