Text pasted from a document, a browser or a spreadsheet regularly arrives carrying characters that render as nothing at all. They are not corruption — every one of them is doing a job it was designed for — but they break comparisons, splits and parsers in ways that are genuinely hard to see.
The debugging experience is always the same: two strings look identical and are not equal.
Which invisible characters turn up most?
A short list accounts for nearly all of it.
| Character | Where it comes from | What it breaks |
|---|---|---|
| Non-breaking space | Word documents, in HTML | Splitting on spaces, lookups |
| Byte order mark | Files saved as UTF-8 with BOM | The first CSV header, JSON parsing |
| Zero-width space | Copied web text, line-break hints | Search, exact matching |
| Soft hyphen | Word documents, justified text | Search, and it appears when wrapped |
| Smart quotes | Autocorrect anywhere | Code, CSV, matching |
The non-breaking space is the most common by a wide margin. It exists to stop a line wrapping between two words — "10 kg" should not split across lines — and it is a different character from an ordinary space, so any code splitting on spaces walks straight past it.
Why does the first column header have junk in it?
A byte order mark. It is a single character placed at the start of a file to signal the encoding, it is invisible when the file is read correctly, and it becomes part of the first field when it is not.
The symptom is a CSV whose first column will not match by name — the header is not "id" but something-then-id — and a JSON file that fails to parse at position zero with no visible cause. Reading the file as Latin-1 makes it visible as three stray characters, which is the usual way people discover it exists.
It is unnecessary in UTF-8 and the specification does not recommend it, but several widely used tools write it by default. Stripping it is safe.
What is a zero-width joiner doing in my text?
Building an emoji, usually. The zero-width joiner glues separate emoji into one — a family sequence or a profession is several characters joined by invisible ones, which is why deleting one emoji sometimes takes two backspaces and why the character count is higher than the number of pictures.
Its siblings do quieter work. A zero-width space marks a legal break point in a long string without adding visible width; a zero-width non-joiner prevents two characters forming a ligature, which matters in Persian and Arabic typography.
None of them is harmful in isolation. They become a problem when text is compared, indexed or used as a key, because a zero-width character makes two visibly identical strings unequal.
Which ones once broke JavaScript?
The line separator and paragraph separator. Both were valid inside a JSON string and both were treated as line terminators by JavaScript, so JSON embedded directly into a script could produce a syntax error with nothing wrong on screen.
It was fixed by changing the language rather than the data — a 2019 revision made both legal inside string literals — but the story is the useful part. An invisible character that means "line break" to one specification and "ordinary character" to another will find the seam between them.
Soft hyphens work the same way in a smaller register. A soft hyphen is invisible until the line wraps at exactly that point, at which moment a hyphen appears in the middle of a word that never had one.
Direction marks deserve a mention because they are the hardest to spot. The right-to-left and left-to-right override characters change how the text after them is displayed, so a filename carrying one can appear to have a different extension than it really has. Anything arriving from an untrusted source is worth stripping them from.
What should cleaning actually do?
Less than it can. Collapsing runs of whitespace, converting non-breaking spaces to ordinary ones, stripping zero-width characters and normalising quotes covers nearly every real case without touching meaning.
Accent removal is a step beyond that and is only right sometimes. Turning "café" into "cafe" is correct for a URL slug and wrong for a name, since the accented form is how the word is spelled. The same caution applies to stripping emoji — fine for a database key, destructive for a message.
Stripping HTML tags belongs in the same category. It recovers the readable text from a copied fragment, and it discards the structure permanently, so it is a conversion rather than a cleanup.
Questions people ask
How do I tell which character it is? Count the characters and compare against what you see. A count higher than the visible characters means something invisible is present.
Will cleaning break my formatting? Collapsing whitespace flattens deliberate indentation. Clean prose freely; clean code carefully.
Do smart quotes matter outside code? In CSV and search, yes — a curly apostrophe does not match a straight one. In prose they are the correct character.
Is a tab the same as spaces? No, and a pasted table often mixes both. Converting one to the other is a decision worth making explicitly.
Clean what is there, keep what means something. The text cleaner does the common pass in one go, whitespace remover, remove spaces and remove empty lines handle the visible half, remove accents, remove emojis, remove punctuation, remove numbers and remove special characters are the deliberate destructive ones, strip HTML tags recovers text from markup, and find and replace and bulk find and replace handle anything with a pattern you can name.