Guides

Why Text Turns Garbled: Encodings and Mojibake

Garbled text such as café or 안녕 looks like damage, but it is usually something simpler: the right bytes read with the wrong rulebook. Knowing which rulebook went wrong tells you whether the text can be repaired.

Text is bytes, and an encoding is the rulebook

A computer stores text as bytes. An encoding says which byte or bytes stand for which character. UTF-8, the encoding most of the web uses, writes the letter é as the two bytes C3 and A9. The encoding windows-1252 uses one byte for each character, and it says C3 is à and A9 is ©. If a program reads UTF-8 bytes as windows-1252, it shows two characters, é, where one was meant. This kind of garbling is called mojibake, a Japanese word.

Three examples, all from the same mistake

What was written The bytes in UTF-8 Read as windows-1252
é in café C3 A9 (2 bytes) café
안녕 6 bytes 안녕 (6 characters)
the emoji 😀 F0 9F 98 80 (4 bytes) 😀

Plain English does not change, because the letters A to Z and the digits are one byte with the same value in both encodings. That is why garbling shows up first in names, accents and non-English text.

Two common causes in Korean

Garbled Korean usually comes from one of two mistakes. UTF-8 Korean read as windows-1252 produces letters such as ì, ë and í together with symbols such as • and …, as in 안녕. Korean written in EUC-KR (also called CP949) and read as windows-1252 or Latin-1 produces letters such as ¾, È, ³ and ç. Programs such as Excel and Notepad often save Korean in EUC-KR, while many other programs expect UTF-8, which is why a CSV file can open garbled on another computer.

When the text can be repaired

Garbling can be reversed when no information was lost. The garbled characters can be turned back into the same bytes by writing them in the wrong encoding, and then those bytes can be read in the right one. A text that was garbled twice or more can be repaired layer by layer in the same way. A repair is trustworthy when garbling the repaired text in the same way gives back exactly the garbled text you started with.

When it cannot

If the garbled text contains the replacement character � or a question mark where a letter should be, the bytes are already gone: a program wrote those marks because it met bytes it could not read. No tool can bring them back, and a tool that claims to is guessing. This happens most when UTF-8 Korean is read as EUC-KR, or EUC-KR as UTF-8. The only fix is to go back to the original file or source and open it with the right encoding.

What to do

  1. Look at the garbled text. Letters like ì and ë point to UTF-8 read as windows-1252, and letters like ¾ and È point to EUC-KR.
  2. Paste it into the garbled text fixer and, if you know the language, choose it.
  3. Check the proof: the repaired text should read like real writing, and the tool marks it as correct only when garbling it again gives back what you pasted.
  4. If you see � or ?, return to the original source instead.

The corrupted text tool does the opposite on purpose: it turns text into the garbled form so you can see what a wrong encoding does.

Tools for this

Sources

Last updated: