How to test whether a de-identified dataset can be re-identified

Why is removing names not enough?#

Removing names is not enough because ordinary details combine into a fingerprint. Latanya Sweeney estimated from 1990 U.S. Census data that 87% of the population was likely unique on five-digit ZIP code, gender and date of birth alone. In the Netflix Prize research, Narayanan and Shmatikov reported that eight movie ratings, two of which may be wrong, plus dates with up to a 14-day error were enough to uniquely identify 99% of records.

Business records carry their own fingerprints: a rare job title, a named product integration, an outage on a known date, a project in a small town. None of these is a name, and any of them can point to one customer or one employee for someone who knows the context.

That is why de-identification is judged by what an attacker could do with the released file, not by which columns were deleted. A test turns that judgment from an assumption into evidence.

What do the main standards expect a test to show?#

The main standards expect evidence that identification is not reasonably likely, using methods someone could repeat. They differ in wording and in where they apply, so the test file should speak to whichever ones counsel says are relevant to the deal.

The red-team procedure, step by step#

The red-team procedure is a planned attempt to re-identify records from the prepared dataset, run by people who did not prepare it. Outside reviewers or colleagues from another team work well; the preparers know too much about what they removed to attack it fairly.

• 1Define the release: which files, which fields, and how the buyer will receive and store them.
• 2Name the attackers: an outsider with public sources and skill, often called a motivated intruder, and an insider such as a former employee or customer who knows the events.
• 3Draw a sample that includes rare cases on purpose: small customers, unusual products, senior roles, short time windows.
• 4List the quasi-identifiers left in each record: dates, locations, titles, product names, amounts, sequence of events.
• 5Attempt linkage against public sources such as company websites, press releases, professional profiles, review sites and court filings, and against what each insider would know.
• 6Score every sampled record and note which fields made identification possible.
• 7Fix those fields by generalizing, suppressing or removing them, then repeat the attack on a fresh sample.
• 8Document the attackers, sources, sample, scores, fixes and the final decision.

How should results be scored?#

Results should be scored per record on a short scale that separates a confident name from a narrowed guess. A single number for the whole file hides the rare records that matter most.

For structured fields, the classic measure is k-anonymity: a release is k-anonymous when each record cannot be distinguished from at least k-1 others. Later work on l-diversity and t-closeness addresses a gap in that measure, where everyone in a group shares the same sensitive value. Mainstream tooling now computes these: Google's Sensitive Data Protection API offers k-anonymity, l-diversity, k-map and delta-presence analysis.

Why free text needs a different attack#

Free text needs a reading attack because its identifying details do not sit in columns that a uniqueness count can measure. Support tickets, email, chat and project notes describe events, and events are what insiders remember.

Testers should read sampled records looking for nicknames, signatures that survived cleaning, unusual job titles, project code names, quoted contract terms, local landmarks and references to well-known incidents. They should also look across records, since several harmless tickets from one coded customer can add up to an obvious identity.

Record the clue types found, not just the counts. A list such as release names tied to a single customer or city names inside product configuration tells the preparation team exactly which rule to add.

Illustrative: testing a support ticket release#

Illustrative: a fictional vertical software company prepares several years of Zendesk tickets for license, with names, emails and company names replaced by codes. Its general counsel asks two people from the finance team and an outside reviewer to attack a sample.

The outside reviewer, using only the company's public changelog and customer case studies, links a group of tickets to a named customer: a custom integration mentioned in the tickets appears in a published case study, and an outage date matches a public status page. A finance analyst, acting as an insider, names a customer from a renewal dispute described in detail.

The team generalizes custom integration names to their category, coarsens dates to the month, and removes commercial negotiation threads from scope. A fresh sample is attacked again and no record is named. The test, the fixes and the retest are filed with the release documentation.

How SourceX documents re-identification testing#

SourceX places re-identification testing at the end of the Preparation step in the SourceX five-step transaction, before the supplier's Approval. The sample, the attackers considered, the scores, the fixes and the residual-risk decision are recorded in the privacy record of the SourceX Evidence Packet.

The license itself should carry the other half of the protection: a prohibition on re-identification attempts and on combining the data to single people out. California's definition of deidentified data expects recipients to be bound contractually, so the test and the contract terms work together.