How to use it
- Paste your list into the input box, one item per line.
- Choose whether letter case and surrounding spaces should matter, and whether empty lines should be removed.
- Choose whether to keep the first or the last occurrence of each repeated line.
- Copy the cleaned list from the output box. The summary shows how many lines were removed.
How it works
Two lines count as duplicates when their text is identical after the options you choose are applied. With “Ignore case” on, “Apple” and “apple” are the same line; case folding follows the page language, so in Turkish “İzmir” and “izmir” match. With “Ignore spaces at the start and end of lines” on, “ cherry” and “cherry” match. These options only change how lines are compared: the line that is kept appears exactly as it was in your input.
The order of your list is never changed. With “First occurrence”, each line stays where it first appeared and later copies are deleted. With “Last occurrence”, earlier copies are deleted and the line stays at its last position, which is useful when later entries are more up to date. To sort the list as well, use the Sort Lines tool afterwards.
Empty lines are compared like any other line, so several blank lines are reduced to the first one. Turn on “Remove empty lines” to delete them all, including lines that contain only spaces. Windows (CRLF), old Mac (CR) and Unix (LF) line endings are all recognised; the output uses LF line endings.
Examples
Cleaning a mailing list
A list exported from two sources contains “anna@example.com” and “Anna@Example.com ”. With “Ignore case” and “Ignore spaces” on, only the first one is kept, written as it appeared.
Keeping the latest entry
A log of status changes lists “server-2” on lines 3 and 40. With “Last occurrence”, the copy on line 3 is removed and server-2 stays at its later position.
Unique keywords
Paste keyword lists collected from several tools into one box to get each keyword once before importing them into a spreadsheet.
Frequently asked questions
Does it change the order of my lines?
No. Lines stay in their original order; only repeats are removed. Sort the result separately if you need it in alphabetical order.
Why are two lines that look the same not removed?
They differ in something you cannot see, such as a trailing space, a tab, a non-breaking space or different letter case. Turn on “Ignore case” and “Ignore spaces”, or clean the text first with the Text Cleaner.
Can it remove duplicate words inside a line?
No, it compares whole lines. To deduplicate words, put one word per line first, for example with the split option of the Line Tools.
How long can the list be?
There is no fixed limit. Lists with hundreds of thousands of lines work, though older phones may take a moment.
Privacy
Your list is processed only in your browser. It is not uploaded, stored or included in analytics.