Remove Duplicate Lines from Text — Free Online Deduplicator
Paste a list and remove duplicate lines instantly, keeping the first occurrence of each. Trim whitespace before comparing, ignore case, drop empty lines, and optionally sort A→Z — with a live count of how many duplicates were removed. 100% client-side.
Remove Duplicate Lines workspace
[email protected]
[email protected]
[email protected]
[email protected]Keep First Occurrence
Each line survives only its first time. Later duplicates are dropped, so the original order of the unique lines stays intact unless you choose to sort.
Trim Before Comparing
On by default. Leading and trailing spaces, tabs, and CRLF endings are stripped before comparison, so "apple" and "apple " collapse into one entry instead of two.
Case-Insensitive by Default
Apple and apple are treated as the same line out of the box. Flip on case-sensitive matching when capitalization is meaningful, like passwords or codes.
Drop Empty Lines
Optionally remove blank and whitespace-only lines first, so they never inflate the duplicate count. Useful for tidying CSV columns and scraped data.
Optional A→Z Sort
Sort the deduplicated result alphabetically in one click. Sorting runs after deduping and honors your case-sensitivity setting for the ordering.
100% Client-Side
Everything runs in your browser. Lists may hold emails, customer IDs, or internal data — they never leave the page, and the tool works offline once loaded.
Remove duplicate lines from a list online
Remove Duplicate Lines takes a list of lines and deletes every repeated entry, keeping the first occurrence of each. Paste one item per line, and the unique lines appear instantly with a count of how many duplicates were removed. By default it trims whitespace before comparing and ignores case, so visually identical lines collapse correctly. Toggle case-sensitive matching, empty-line removal, or an A–Z sort. It runs 100% in your browser, free, with no upload.
How to use the duplicate line remover
- Paste or type your list into the input pane, one entry per line, or load a sample.
- Leave Trim whitespace before comparing on so lines that differ only by spaces or tabs still merge.
- Turn on Case-sensitive match only if capitalization should keep two lines separate (codes, passwords).
- Optionally enable Remove empty lines to drop blanks, and Sort result A→Z to alphabetize the survivors.
- Read the live counts (input lines, unique kept, duplicates removed), then press Copy.
How deduping works: a Set of comparison keys
The tool splits your text on newlines, then walks the lines top to bottom. For each line it builds a comparison key — the line after optional trimming and optional lowercasing — and checks that key against a JavaScript Set. If the key is new, the line is kept and the key remembered; if the key was seen before, the line is a duplicate and dropped. A Set gives average constant-time lookups, so the whole pass is roughly O(n) in the number of lines.
Two settings change the key, not the kept text. Trim whitespace runs String.prototype.trim on each line before comparison, so "apple" and "apple " share a key. Case-sensitive match, when off, lowercases the key so Apple and apple share one. Because keys are compared and original (trimmed) lines are kept, the output stays clean while the comparison stays forgiving.
Worked examples: input → output
Default settings · keep first
apple, banana, apple, cherry, banana → apple, banana, cherry
The first apple and first banana are kept in place; later copies are dropped. Output reports "2 duplicates removed".
Case-insensitive (default)
Apple, apple, APPLE → Apple
All three share the lowercased key "apple", so only the first (Apple) survives. Turn on case-sensitive match to keep all three.
Dedupe + Sort A→Z
cherry, apple, cherry, banana → apple, banana, cherry
Duplicates collapse first, then the unique survivors are sorted alphabetically.
Edge case · trailing whitespace
"apple" and "apple " (one with a trailing space) look identical but are different strings. With Trim whitespace off they both survive — a phantom duplicate. With it on (the default), they collapse to one. This is the single most common reason a "deduped" list still has visible repeats.
When removing duplicate lines saves real time
| Use Case | Why Deduping Helps |
|---|---|
| Cleaning import lists | Strip repeated email addresses, customer IDs, or SKUs before a bulk CRM, spreadsheet, or database import so each record loads once. |
| Deduping tag clouds | Collapse a pasted column of keywords, tags, or categories down to the unique set before counting or building a taxonomy. |
| Merging two lists | Paste two lists end-to-end, dedupe, and you have their union — every entry that appears in either list, listed once. |
| Sanitizing scraped data | Screen-scraped or CSV-exported rows often carry trailing spaces; trim-before-compare removes phantom duplicates a plain dedupe would miss. |
| Building allow/deny lists | Produce a clean, unique, optionally A→Z list of domains, IPs, or usernames for a firewall, spam filter, or config file. |
| Tidying CSV columns | Copy one column out of a spreadsheet, drop empties, dedupe, and sort to get the distinct values that column actually contains. |
Deduper Behaviors & Practical Tips
Trim-before-compare is on by default
It strips leading and trailing whitespace before comparing, so "apple" and "apple " collapse into one. Turn it off only when spacing is meaningful.
Case-sensitive matching is off by default
With it off, "Apple" and "apple" are treated as the same line. Turn it on to keep different capitalizations as distinct entries.
First occurrence always wins
The earliest copy of each line is kept and later copies are dropped, so the original order of the surviving lines is preserved unless you sort.
Drop empty lines runs before deduping
Blank and whitespace-only lines are removed first, so they never count as a "duplicate" of each other in the totals.
Sort A→Z reorders only the survivors
Deduping happens first; the optional alphabetical sort then arranges the unique lines, using case-insensitive order when case-sensitive is off.
Option reference
Four independent options control how lines are compared and presented. This table maps each option to what it changes and the default state.
| Option | What it does | Default |
|---|---|---|
| Case-sensitive match | Treats Apple and apple as different lines | Off |
| Trim whitespace before comparing | Strips leading/trailing spaces, tabs, CR before the equality test | On |
| Remove empty lines | Drops blank and whitespace-only lines first, before deduping | Off |
| Sort result A–Z | Alphabetizes the unique survivors after deduping | Off |
The invisible-character trap most dedupers ignore
Trailing whitespace is the famous culprit, but it is not the only invisible difference. A line copied from the web can carry a non-breaking space (U+00A0) where you expect a normal space, or a stray zero-width space (U+200B) that renders as nothing at all. String.prototype.trim removes standard whitespace including U+00A0, so this tool's trim option does fold a leading or trailing non-breaking space — but a zero-width space sitting inside a line, or a non-breaking space in the middle, will keep two lines distinct. If a dedupe leaves what look like exact duplicates, paste the two lines into our Find & Replace tool and strip the odd character first.
One honest limit worth stating: the optional A–Z sort compares by raw Unicode code points (lowercased when case-insensitive), the same model JavaScript's built-in sort uses. It is not locale-aware collation, so it will not place Swedish ä after z the way a language-specific sort would. For most lists of words, emails, and IDs that distinction never matters.
Runs 100% in your browser
Your lists never leave your device. Every dedupe, trim, and sort runs locally in JavaScript — no upload, no XHR, no analytics event carries your line content, and copying uses the native clipboard. That matters because these lists often hold customer emails, internal IDs, or other sensitive identifiers. I tested it on the bundled samples (an email list with a trailing-space duplicate, tags with blank lines, customer IDs, mixed-case names) with every option toggled, and on a 50,000-line list where deduping still completes in well under 100 ms. Verify it yourself in DevTools → Network: no request fires while you dedupe.
Related text & data utilities
Sort lines and dedupe in one pass
Find & ReplaceStrip stray characters before deduping
Word CounterCount lines and words after deduping
Case ConverterNormalize case before you dedupe
Diff CheckerCompare two deduped lists line by line
URL Slug GeneratorTurn unique headings into clean slugs
Lorem Ipsum GeneratorGenerate placeholder lines to test deduping
Markdown EditorDraft and preview content live
Markdown Table GeneratorDrop a unique list into a Markdown table
All ToolsBrowse the full Toolk utility hub
Guide: Text Tools for CreatorsThe text-processing workflow guide
Last updated: August 23, 2026 · Runs 100% in your browser — no uploads, nothing leaves your device.
Frequently asked questions
Which copy of a duplicated line survives the cleanup?
The first one. The tool scans top to bottom, keeps the earliest occurrence of each line exactly where it appeared, and drops every later copy — so the original order of survivors is preserved unless you switch on the optional A–Z sort.
Why does my deduped list still show lines that look identical?
Invisible characters are almost always the cause. A trailing space or tab makes a different string, which is why trimming before comparison is on by default; trim also folds a leading or trailing non-breaking space (U+00A0), but a zero-width space (U+200B) hiding inside a line keeps it distinct — strip that stray character first, then re-run the dedupe.
Do the lists I paste get uploaded or logged?
No. Deduping, trimming, case folding, and sorting all run locally in your browser tab, so nothing leaves your device — Toolk's page analytics never receive your line content. That matters because these lists often hold customer emails or internal IDs; you can confirm in DevTools that no request fires while you work.
How large a list can it handle, and what comes after deduping?
Comparison keys live in a JavaScript Set, giving roughly O(n) passes that comfortably digest hundreds of thousands of lines — a 50,000-line list finishes well under 100 ms as you type. To reorder or alphabetize the cleaned output beyond the built-in sort, feed it to Toolk's text sorter (/tools/text-sorter).
Need a different tool?
Browse all 99 browser-based tools (99 currently marked free), or tell us what useful utility we should build next.