To detect duplicates in a CSV file, first compare strictly identical lines. Then identify a business key, normalize the values and define the line to keep. Do not delete anything until you have saved the original file and checked the meaning of each duplicate.
A duplicate is not always a mistake
Two identical lines may be an import defect, but also two actual events recorded at the same time. A contact list, an order export and a technical log do not follow the same rules.
Good control therefore answers two questions: which lines are similar, and which lines actually represent the same business object?
Distinguish three types of duplicates
| Type | Example | Method |
|---|---|---|
| Exact line | All cells are identical | Compare full line |
| Identical key | Same e-mail or order number | Compare one or more columns |
| Near duplicate | Differences in whitespace, case or formatting | Normalize and validate |
Start with the exact duplicates: they are easy to explain and verify. Then go to business keys and near duplicates only if the need warrants.
1. Keep a copy intact
Always work on a copy. The deletion of duplicates is a data transformation and must remain reversible. Keep the source file, transaction date and rule applied.
2. Choose the right business key
A key must identify the object that each line represents. An internal identifier is preferable where it exists. Otherwise, combine several stable columns.
- contacts: normalized email address, possibly associated with the name;
- orders: order number, never the only amount;
- products: reference or SKU rather than wording;
- events: identifier and timestamp at the required level of precision.
A column is not a good key simply because it contains few repetitions. It must be unique by business definition.
3. Normalize before comparing
Values client@example.fr, Client@Example.fr and client@example.fr may designate the same address despite their visual differences. Depending on the context, normalize:
- spaces placed before or after values;
- letter case when it has no business significance;
- date and number formats;
- separators and invisible characters;
- empty values represented in several ways.
However, keep the original value in the reference file. Normalization is used for comparison, not to erase information without control.
4. Set the line to keep
Once the duplicate groups have been identified, choose a reproducible rule: keep the most recent, most complete line, the one with a validated status or the one from the reference source.
If no reliable rule exists, mark the lines for manual review instead of deleting them.
5. Check the result
Compare the number of lines before and after processing. Review a sample of removed groups, reopen the processed file and check that the columns, encoding and separator are still correct.
To run the check directly, open the CSV duplicate finder. For wider preparation before import, see the guide Clean a CSV file before import. To also control the structure, use CSV validator.
Detect duplicate rows without changing the file
DataCheck analyses the CSV statically, reports strictly duplicate rows and presents the other anomalies to be checked.
Analyze my CSV file Create my Protect accountFrequent errors
- delete all repetitions without understanding the file;
- choose a non-unique column as the only key;
- directly modify the original file;
- confuse a repeated cell with a duplicate row;
- normalize data without retaining the original values;
- forget to check the output file before importing it.
Frequently Asked Questions
How to detect duplicates in a CSV file?
First identify strictly identical lines, then compare a business key such as an ID, email or order number.
Can all identical lines be deleted automatically?
No. Two identical lines may represent separate events. A business rule is required before deletion.
Which column should you use to find duplicates?
Choose a truly unique column or combine several stable columns when the file does not have a reliable identifier.
What is the difference between exact and near duplicates?
An exact duplicate repeats the same values. A near duplicate contains variations that require normalization and validation.
Detect first, then decide
Technical detection produces candidates. The decision to merge or delete depends on the meaning of the data. A safe method keeps the source, documents the key and checks the result before any import.
Published on 20 August 2026.