A team testing a billing change needs test data with accounts with overdue invoices, a customer on a legacy plan and an order that crosses a month boundary. Copying production gives realistic cases but can bring personal data into test environments; writing fixtures by hand is safe but covers only the cases someone thought to write.
Teams can combine several sources, depending on the test. NIST’s guidance on de-identifying government datasets (SP 800-188, September 2023), written for releasing agency data rather than for testing, lists publishing synthetic data based on identified data as one data-sharing model, alongside publishing de-identified data, providing a query interface and sharing data in protected enclaves. The guidance asks agencies to evaluate their goals and risks before using de-identification.
Common approaches:
- masked or pseudonymized copies of production data, with identifiers replaced
- subsets of production data, extracted with their related records so tests stay consistent
- synthetic data generated to match the structure and distributions of real data, or to add cases production rarely contains
- hand-written fixtures for specific edge cases, kept under version control with the tests
How is test data management different from data masking?
- Data masking is one technique: it replaces sensitive values in a copy of real data.
- Test data management is the wider practice of deciding which data each test needs, where that data comes from, how it is protected, and how it is refreshed and versioned.
- A TDM process may use data masking, subsetting, synthetic generation and fixtures together, depending on the test.
Where does test data management break in AI work?
Test data management breaks when the data behind a test cannot be traced to what it stands for. AI systems are tested on evaluation records and task cases as well as on application data, and a synthetic or masked case that looks plausible may not reflect the situation the test is meant to check.
Some values can be generated and some cannot. Synthetic records can stand in for realistic structure, rare combinations and protected fields, but a test that checks a specific fact, such as the amount a customer was actually billed or the permissions an account actually holds, needs that fact from a trusted source. Recording which test data was generated, which was copied and which was checked against the source, together with its version, tells a team which results to recheck when data or rules change.