Duplicates enter data in ordinary ways: the same customer created twice in two systems, a transaction logged by both a gateway and a ledger, a document scraped from two mirrors. Exact copies are easy to find. Copies are also common at scale: a study presented at ACL 2022 found that widely used language-model training sets contain many near-duplicate examples and long repeated passages, and that deduplicating them reduced how often models copied training text verbatim. Near-duplicates, with a changed spelling, a reformatted date or a different ID, need matching rules, which is the work described under entity resolution.
For AI work, the split matters as much as the count. In CUBIG’s UCI Shill Bidding experiment (October 2026), most test records share an auction or bidder with the training records, so the write-up states that performance on unseen auctions has not been established. These are related records, not duplicates, but the effect is similar: related records on both sides of a split can make a test easier than the task. A study of duplication in code datasets (Allamanis, 2019) found that duplicates shared between training and test sets can inflate reported performance metrics by up to 100%.
Common approaches:
- exact matching on a key or a hash of the full record
- fuzzy matching on names, addresses or text, with a similarity threshold
- rule-based matching that combines several fields, such as email and date of birth
- deciding which version survives when duplicates disagree, and keeping a record of the merge
How is data deduplication different from entity resolution?
- Data deduplication removes or merges repeated records within a dataset.
- Entity resolution decides which records, across one or more sources, refer to the same real-world entity, whether or not they are duplicates.
- Deduplication often uses entity resolution to find near-duplicates, then applies a rule for which record to keep.
Where does data deduplication break in AI work?
Deduplication done after the train-test split, or not at all, lets copies of the same record land on both sides. The model is then tested partly on records it has already seen, a form of data leakage that can inflate the score. Deduplicating before splitting, and grouping related records so they stay on one side, keeps the test honest.
Over-aggressive deduplication breaks things too. Two records that look alike may be two real events, such as a customer who really did place the same order twice. Merging them removes signal the task may need, so the matching rule should be checked with the people who own the process, and the original records kept so the merge can be reviewed.