Remove duplicates from a CSV

Steps

1. Open the CSV in Rowlode.
2. Tick Remove duplicate rows.
3. Leave the column list empty to compare whole rows, or select key columns (hold Ctrl / ⌘ to select several), for example email.
4. Choose whether to ignore case and spaces at the start/end, then press Apply. The result line tells you how many duplicates were removed.
5. Export the result, or check it first in the table.

Which row is kept?

The first occurrence, in file order (with several files: in the order the files are listed). Later copies are dropped. Combine with a filter to, for example, keep one row per customer from a single country.

How it works – and its limit

Rowlode keeps a 64-bit fingerprint of each distinct row (or key) instead of the row itself, so tens of millions of rows fit in memory. Two different rows would need the same 64-bit fingerprint to be confused – for 100 million rows the chance is about 1 in 3,700. Trailing empty cells don’t make rows different.