텍스트 · 정리
중복 줄 제거 도구
목록, 이메일 주소, 로그 데이터에서 중복된 행을 즉시 제거합니다. 원래 순서를 그대로 유지하면서 중복 항목 수를 실시간으로 계산해 제공합니다.
How to use
Paste your list, URLs, or lines into the input area. Choose your deduplication settings (Exact match or Ignore case, Trim whitespace, Remove empty lines). Deduplication runs instantly in real time. Click "Copy" or "Download" to save your clean list.
Example
Deduplicating a mixed list with case-insensitivity enabled:
Banana
apple
orange
banana
Banana
orange
Notice that original order is preserved, and "Banana" / "banana" are merged when "Ignore case" is active.
How it works
Layerlee splits incoming text by line break characters (supporting LF, CRLF, and CR). It streams through each line sequentially, keeping track of unique items using a hash set. Only the first occurrence of each unique line is emitted into the output stream, which preserves the original sequence perfectly while eliminating duplicates in linear O(N) time.
Real-world use cases
- Email & marketing lists: Purging duplicate contacts before running newsletter campaigns or imports.
- SEO & keyword research: Removing repeated keywords copied from search engines or keyword planners.
- Developer logs & database IDs: Deduplicating IDs, IP addresses, log traces, or SQL insert values.
- Spreadsheet exports: Cleaning CSV and Excel columns with duplicate values.
- Web scraping & URL lists: Filtering out duplicate scraped links, sitemaps, or domains.
- Inventory & SKU management: Removing repeated SKU codes or product catalog entries.
Important notes & limitations
Order preservation: Unlike SQL DISTINCT or Unix sort -u, Layerlee never re-sorts your lines alphabetically. The relative order of first appearance is strictly maintained.
Client-side security: No lines are sent across any network. Sensitive lists (passwords, emails, customer records) remain private on your computer.
Frequently asked questions
What does "Ignore case" do?
When enabled, lines like "Hello" and "hello" are treated as the same duplicate line. The casing of the first encountered line is kept in the output.
Does this tool sort lines alphabetically?
No. Layerlee preserves the original order of your lines. If you need sorting, you can sort before or after deduplicating.
How many lines can I deduplicate at once?
Because the algorithm runs in your local browser engine in linear time, you can easily process 50,000+ lines in seconds.
Are empty lines removed?
Yes, by default the "Remove empty lines" option is checked. You can uncheck it if you want to keep blank lines in your text.