What Levenshtein Edit Distance Actually Measures
Edit distance counts the minimum number of single-character insertions, deletions, or substitutions needed to turn one string into another. "cat" to "bat" is 1 substitution; "cat" to "cats" is 1 insertion. This tool turns that raw count into a percentage by dividing it by the length of the longer of the two strings and subtracting from 1, so two identical strings score 100% and two completely unrelated strings of similar length score close to 0%.
It's a purely character-level measurement with no understanding of meaning - "Spider-Man" and "Spiderman" score very high (they differ by just a hyphen and a space) precisely BECAUSE the algorithm only looks at characters, not because it "understands" they refer to the same thing.
Why the Right Threshold Depends on Your Data
A high threshold (95%+) only catches near-identical variants - differences of a character or two, like typos or inconsistent punctuation - and is the safest starting point for data where false-positive groupings (merging genuinely different entries) would be costly. A lower threshold (70-80%) catches looser variations but risks grouping entries that merely share a lot of characters despite being conceptually different, like "The Dark Knight" and "The Dark Knight Rises."
There's no universally correct threshold - it's a direct tradeoff between catching more real duplicates and avoiding more false groupings, and the right value depends entirely on how messy your specific dataset is and how costly each type of mistake would be for your use case.
Frequently Asked Questions
No - it's one of several well-known approaches (others include Jaccard similarity on word sets, or phonetic algorithms like Soundex for "sounds like" matching). Levenshtein is particularly good at catching typos and minor formatting differences specifically because it operates at the character level.
The comparison is pairwise - every line has to be checked against every other still-ungrouped line - so the work grows roughly with the square of the list size, not linearly. That's why this tool caps processing at 1,500 non-empty lines rather than letting an extremely large paste silently take a very long time.
For most real-world messy data (inconsistent capitalization, stray punctuation from copy-pasting), yes - they're on by default for that reason. Turn them off specifically when case or punctuation differences are meaningful in your data and shouldn't be treated as noise.
Yes - two entries that are conceptually the same but worded very differently (like "NYC" and "New York City") will score low on pure character-edit-distance even though a human would recognize them as duplicates, since the algorithm has no concept of synonyms or abbreviations.