Differential privacy for LLM fine-tuning on sensitive licensed data: DP-SGD in practice

What DP-SGD actually changes in a fine-tuning loop

DP-SGD changes two steps of ordinary training and adds an accountant. For each minibatch, it computes per-example gradients, clips each to a maximum L2 norm C, sums them, adds Gaussian noise scaled to C times a noise multiplier sigma, then averages and steps [1]. A privacy accountant composes the per-step loss over all steps, given the sampling rate (batch size divided by dataset size), and reports epsilon at a fixed delta [1].

The knobs that matter in practice are:

• Clipping norm C. Too high means the noise must be large in absolute terms; too low biases gradients toward the clip boundary. Sweep it on a public proxy task first.
• Noise multiplier sigma and steps. These two, plus the sampling rate, set epsilon. More epochs cost budget.
• Batch size. Large logical batches (built by gradient accumulation) improve the signal-to-noise ratio, because the noise is added once per batch while the summed signal grows with batch size.
• Delta. Set it below 1/N, where N is the number of privacy units, so the guarantee is not trivially weak.

The cost is real. Per-example gradients raise memory and compute, which practitioner write-ups flag as a practical barrier and address with the private-transformers library for Hugging Face models [1]. That library's memory-saving clipping avoids materializing full per-example gradients for linear layers. Opacus is the other common PyTorch path.

Why you start from pretrained weights and train few parameters

Start from a strong public pretrained checkpoint and apply DP only during fine-tuning. Pretraining on public data carries no privacy cost for your licensed records, and a model that already knows language needs fewer noisy updates to learn your task [1]. Training from scratch under DP on a few thousand licensed documents rarely produces anything useful, while fine-tuning a capable base model often does.

Parameter-efficient methods help further. Noise is added per trainable parameter, so training a low-rank adapter or similar small module concentrates the signal. Fewer trainable parameters also means smaller per-example gradients to store, which eases the memory cost. Published gaps between private and non-private fine-tuning mostly come from academic benchmarks (classification and short text-generation tasks); do not assume they carry over to your domain or to long-form instruction tuning.

Choosing epsilon and measuring utility loss

There is no regulator-blessed epsilon for fine-tuning; pick it empirically against a non-private baseline. A sound protocol sweeps several budgets, for example epsilon 1, 3 and 8 alongside a non-private run, and compares task metrics and memorization on the same held-out set [2]. Report epsilon, delta, the accountant used, the privacy unit and the number of training steps together, because an epsilon without those is not comparable.

Expect utility loss to grow as epsilon shrinks, and to compound with other constraints. Distilling or compressing a DP model stacks losses: one paper notes DP and compression together degrade utility more than either alone, and proposes DP synthetic text as the distillation channel instead [3]. If you plan to quantize or distill for deployment, budget for that in the sweep. If you need a shareable artifact rather than a private model, see differentially private synthetic text.

Typical failure modes in DP fine-tuning runs:

• Accountant mismatch: training used shuffled batches, but epsilon was computed assuming Poisson sampling.
• Hyperparameter search on the private data without counting it toward the budget.
• Evaluation on records that were also in the training split, which hides memorization.
• Reporting record-level epsilon when one customer appears in thousands of records.

Record-level versus user-level privacy for conversation data

For operational text, the privacy unit is usually a person or account, not a row. Support tickets, chat transcripts and email threads put the same customer or employee in many records. Record-level DP bounds the influence of one message; a person with 400 messages gets a far weaker effective guarantee (our reasoning from group privacy, not a cited benchmark).

User-level DP fixes this by sampling and clipping at the user level: group all records for a stable pseudonymous key, cap contributions per user, and clip the user's aggregated gradient. That requires the licensed data to carry a consistent pseudonymous identifier after de-identification. Ask for one in the data request, and confirm it cannot be reversed. The same issue applies to indirect identifiers discussed in quasi-identifiers in business text.

How DP fits with de-identification of licensed data

DP-SGD and de-identification solve different problems, and sensitive licensed text needs both. De-identification removes direct identifiers before anyone touches the data, which protects the corpus in storage, in annotation queues and in logs. DP bounds what the trained weights reveal about any training unit. NIST SP 800-188 describes formal privacy methods such as differential privacy alongside traditional de-identification, and cautions about the limits of traditional methods alone [5].

The practical ordering is redact first, then train privately. Redaction misses things; DP limits the damage a missed phone number can do through the model. DP does not, however, cover outputs from retrieval at inference time, nor the license limits on how you may use the data. For the redaction side, see PII redaction for LLM training data and the de-identification evidence package.

Verifying the guarantee with canaries

Insert canaries and measure memorization before and after; a correct DP run should show low canary exposure where a non-private run does not [2]. A canary is a random secret sequence (a fake card number, for example) inserted into training data; you then rank its model likelihood against random candidates of the same format. Carlini et al. extracted verbatim training sequences from a large language model, which is why buyers of sensitive data should test rather than assume [4].

A canary test catches implementation bugs (wrong accountant, clipping disabled on some layers) that an epsilon number cannot. Run the same canaries on the non-private baseline as a positive control [2]. The full audit method, including membership inference, is on auditing a fine-tuned model for leakage, and the threat model is covered in training-data extraction and memorization risk.

A DP fine-tuning run record to keep with the model

Write down every parameter that defines the guarantee so a reviewer can reproduce it. The record below is the kind of artifact privacy reviewers and counsel ask for when a model was trained on licensed sensitive text.

Illustrative example: invented to show structure; it does not describe an available dataset.

run_id: dpft-support-tickets-r3
base_model: "public 7B checkpoint, revision pinned"
trainable_params: "LoRA rank 16 on attention projections"
dataset:
  license_ref: "license schedule A (records, permitted uses, term)"
  deidentification: "names, emails, phones, account numbers replaced; method documented"
  privacy_unit: user            # pseudonymous customer_key
  n_units: 48210
  max_records_per_unit: 50
dp:
  library: "private-transformers"  # or Opacus
  sampling: poisson
  sampling_rate: 0.004
  noise_multiplier: 0.9
  clipping_norm: 0.1
  steps: 3000
  accountant: "RDP"
  epsilon: 3.0
  delta: 1.0e-6                 # below 1/n_units
  tuning_budget_counted: true
evaluation:
  baseline_nonprivate_metric: 0.81
  dp_metric: 0.76
  canaries_inserted: 20
  max_exposure_dp: "near chance"
  max_exposure_nonprivate: "high"

Decision table: when DP-SGD fine-tuning is worth it

DP-SGD is worth its cost when the model will be shared beyond the data's original trust boundary or the data carries residual identification risk.

Illustrative example: invented to show structure; it does not describe an available dataset.

Getting sensitive text you can fine-tune on privately

SourceX sources operational datasets, such as support and sales histories, engineering records and documents, from US companies on request, and manages licensing and ongoing purchases. Every dataset is rights-reviewed and delivered under a license that defines records, uses, term and delivery. Personal details are removed or replaced before delivery, with the method recorded, though no method is perfect. When you describe the data you need to SourceX, include the privacy unit and pseudonymous key your DP plan requires.

For broader context, start at the privacy and de-identification hub, the fine-tuning data guide, the differential privacy glossary entry and training data for domain-specific fine-tuning.

Request licensed data for differentially private fine-tuning

SourceX finds US companies that hold the data you describe and handles the license; nothing is contracted until the supplier agrees. Data is sourced on request rather than held in stock, so a request does not guarantee a match. Describe your DP fine-tuning data needs.