Character Encoding

---
name: character-encoding
description: Handle Unicode end to end so text survives storage, transport, and display without mojibake or truncation mid-character. Use when text arrives corrupted, lengths behave oddly, or emoji break a field.
---

# Character encoding

Encoding bugs look like corruption but are almost always a mismatch: one
layer wrote in one encoding and another read in a different one. Getting
this right is mostly about being explicit at every boundary and using
UTF-8 everywhere you can.

## Method

1. **Use UTF-8 throughout and declare it everywhere.** Database, tables,
   columns, connection, HTTP headers, and file reads. A single layer
   defaulting to something else produces the classic garbled output.
2. **Know that a character is not a byte or a code unit.** Length limits
   measured in bytes cut multi-byte characters in half, and a validator
   counting units mis-measures anything outside the basic range.
3. **Count what the user perceives when it matters.** Emoji, combining
   accents, and flags are several code points rendered as one character,
   so a character limit needs grapheme awareness to behave sanely.
4. **Normalise on input.** Deciding one normalisation form at the
   boundary makes comparison, deduplication, and search consistent (see
   locale-aware-sorting).
5. **Check the database column type actually supports full Unicode.**
   Some engines have a legacy utf8 that stops short of the full range and
   silently rejects or truncates emoji.
6. **Validate rather than strip.** Removing characters you did not
   expect mangles legitimate names; rejecting with a clear message is
   more honest than silent mutilation.

## Boundaries

- Encoding correctness does not imply rendering correctness; a font
  still needs the glyphs (see text-expansion-layout).
- Legacy systems may genuinely require another encoding, in which case
  convert explicitly at the boundary and document it.
- Normalisation can alter strings that some systems treat as distinct,
  so apply it consistently rather than selectively.