De-identifying company data for AI training: the full playbook

Which data anonymization techniques fit which kinds of fields?#

Match the technique to the field: remove or mask direct identifiers, coarsen the fields that identify people in combination, and save heavier methods such as noise or synthetic data for releases where record-level detail is not needed. Direct identifiers are fields like a name or email that point to one person on their own. Quasi-identifiers are fields like ZIP code, job title and dates that are harmless alone but can single someone out together.

NIST's glossary describes k-anonymity as releasing person-specific data so that linking it to other information through quasi-identifiers is limited, achieved through suppression and perturbation, and notes that the protection can be reversed in some cases. NIST SP 800-188 covers both traditional de-identification and formal privacy methods such as differential privacy, and lists publishing synthetic data based on identified data as one of the ways to share. The table is editorial guidance on how the common techniques trade privacy against usefulness for AI training data.

Is de-identified data the same as anonymized data?#

Not always: whether data counts as de-identified or anonymized depends on the law that applies, and the tests differ. Under the HIPAA Privacy Rule, HHS recognizes two methods for health information: Expert Determination, where a qualified expert concludes the risk is very small that an anticipated recipient could identify someone, and Safe Harbor, where listed identifiers are removed, such as names and geographic units smaller than a state, with ZIP codes cut to their first three digits only where that area holds more than 20,000 people. HHS also says there is no single numerical risk level that counts as acceptable everywhere.

In the EU, the European Data Protection Board says pseudonymised data that can be attributed to a person using additional information remains personal data under the GDPR. So replacing names with tokens while keeping the key table is not anonymization there; it lowers risk but leaves the data in scope. Which standard applies to a given dataset, and whether a release still needs a legal basis, is a question for counsel reviewing the specific facts; this is general information, not legal advice.

How do I anonymize data step by step?#

Start with an inventory of identifying fields, pick the standard you are meeting, then test the output for re-identification before anyone outside the company sees it. NIST SP 800-188 suggests that agencies can create a disclosure review board, adopt a de-identification standard with measurable performance levels and perform re-identification studies; the same discipline can be applied to a single company dataset:

• 1List every field and free-text column, and tag each as direct identifier, quasi-identifier, sensitive value or safe content.
• 2Name the standard you are meeting, such as HIPAA Safe Harbor or Expert Determination for health records, or a written internal threshold for everything else.
• 3Choose one technique per tag from the table above and write it down as a field-by-field rule.
• 4Run the rules on a copy, never on the system of record, and keep any pseudonym key table separate from the output.
• 5Try to re-identify a sample: search for rare combinations, unusual job titles, small locations and names hidden in free text or attachments.
• 6Record what was removed, what was kept and why, plus who signed off, so the next release follows the same rules.