The first time I encountered a character encoding problem was on a client site where every apostrophe had been replaced by "’" and every em dash showed as "â€"". The content looked fine in the CMS editor but turned into garbled nonsense on the live site. It took me two hours to figure out what had happened: the database was set to Latin-1 encoding but the HTML page declared UTF-8. The characters were stored in one encoding system and read in another, producing the mess.
Character encoding is one of those topics that most people never think about until something breaks. Then it becomes urgent and confusing. This guide explains what encoding is, why it matters, and how to fix the most common problems.
What Character Encoding Actually Does
Computers store everything as numbers. When you type the letter "A", your computer stores the number 65. When it needs to display that character, it looks up number 65 in a table and draws the letter "A" on screen. Character encoding is the table that maps numbers to characters. Different encoding systems use different tables, which is where problems start.
ASCII was the original encoding standard, developed in the 1960s. It maps 128 characters using 7 bits: the English alphabet in upper and lowercase, digits 0 through 9, punctuation, and control characters like tab and newline. ASCII works perfectly for plain English text but cannot represent accented characters, non-Latin scripts, or symbols beyond basic punctuation.
When computers spread globally, different regions created their own extended encodings. Latin-1 (ISO 8859-1) added Western European characters like e with an acute accent and u with an umlaut. Windows-1252 extended Latin-1 further with curly quotes, em dashes, and other typographic characters. Shift_JIS handled Japanese. EUC-KR handled Korean. Each system worked within its own context but could not represent characters from other systems.
UTF-8 Solved the Problem
Unicode was designed as a single system that assigns a unique number to every character in every writing system. It currently includes over 149,000 characters covering 161 scripts, plus emoji, mathematical symbols, musical notation, and historical scripts. UTF-8 is the most common encoding format for Unicode, and it is now used by over 98 percent of all websites according to W3Techs.
UTF-8 is clever in its design. Characters that exist in ASCII use exactly the same byte values as ASCII, so any ASCII text is automatically valid UTF-8. Characters beyond ASCII use two, three, or four bytes. This means English text takes no extra space compared to ASCII, while the encoding can still represent any character from any language.
Every HTML page on this site includes the declaration <meta charset="UTF-8"> as one of the first elements in the head. This tells the browser which encoding table to use when interpreting the bytes of the page. If this declaration is missing or wrong, the browser guesses, and it often guesses incorrectly for pages with non-ASCII characters.
Common Encoding Problems
The garbled text I described at the start, where characters like curly quotes turn into multi-character strings, is called mojibake. It happens when text encoded in UTF-8 is read as if it were Latin-1 or Windows-1252. The multi-byte UTF-8 sequences get split into individual bytes, each interpreted as a separate Latin-1 character. The result is recognisable once you know the pattern: "’" is always a UTF-8 right single quote read as Latin-1.
The reverse also happens. Text encoded in Latin-1 or Windows-1252 that gets read as UTF-8 produces replacement characters, usually displayed as question marks in diamonds or blank squares. This happens less often now because most modern software defaults to UTF-8, but legacy databases and old exports still contain Latin-1 encoded text.
Copy-paste problems are another common source of encoding issues. When you copy text from a PDF, Word document, or email into a web form or code editor, the clipboard may carry characters in a different encoding than the destination expects. Smart quotes, em dashes, and ellipsis characters from word processors are the most frequent culprits. Using the find and replace tool to swap these characters for their ASCII equivalents fixes most paste-related encoding issues.
Fixing Encoding Problems
If you are seeing mojibake on a website, the fix is almost always ensuring that the encoding declaration in the HTML matches the actual encoding of the content. Add or correct the charset meta tag to UTF-8 and make sure your server is sending the correct Content-Type header. For Apache servers, adding AddDefaultCharset UTF-8 to your .htaccess file handles this.
For database problems, you need to check three things: the database character set, the table character set, and the connection character set. All three need to be UTF-8 (specifically utf8mb4 in MySQL, which supports the full Unicode range including emoji). If your database is Latin-1 but your application sends UTF-8, the data gets double-encoded and every fix you attempt makes it worse.
For text files with wrong encoding, most code editors can detect and convert encoding. In VS Code, the encoding appears in the bottom status bar. Click it to reopen the file with different encoding or save it with a new encoding. When working with CSV files, explicitly specifying UTF-8 encoding during export and import prevents most problems.
Practical Rules
After dealing with encoding problems on dozens of projects, I follow three rules that prevent almost all issues. First, use UTF-8 everywhere. Every file, every database, every form, every API. There is no practical reason to use any other encoding for new projects. Second, declare the encoding explicitly rather than relying on defaults. Browsers, databases, and text editors all have defaults that may not match your assumption. Third, test with non-ASCII characters early. If your system handles an e with an acute accent and a curly quote correctly, it will handle almost everything else correctly too. If it does not, you want to find out before your database has a million rows of incorrectly encoded data.
The tools on this site handle UTF-8 natively because they run in your browser and modern browsers use UTF-8 by default. You can paste text containing accented characters, emoji, CJK characters, or any other Unicode content and the tools will process it correctly. This is a small thing but it matters when you are cleaning data that includes names from multiple languages or content with typographic punctuation.