Unicode normalization explained
Because Unicode can represent the same text multiple ways (precomposed vs combining marks), you often need to “normalize” it to a canonical form before comparing, searching or storing. There are four forms.
The four forms
| Form | Does | Use when |
|---|---|---|
| NFC | Composes to shortest form | General text storage & comparison (most common) |
| NFD | Decomposes into base + marks | Some sorting/searching; stripping accents |
| NFKC | Composes + compatibility folding | Aggressive matching (fi→fi, ①→1) |
| NFKD | Decomposes + compatibility folding | Search indexing, loose matching |
Canonical vs compatibility
Canonical forms (NFC/NFD) preserve meaning exactly. Compatibility forms (NFKC/NFKD) also fold “compatibility” characters — turning fi into “fi” or ① into “1” — which is great for search but loses formatting nuance. Don't use NFKC/NFKD where exact characters matter.
Practical rule
Normalize user input to NFC before storing or comparing, so “é” typed two different ways is treated as the same string.
input.normalize('NFC')Frequently asked questions
Which normalization form should I use?
NFC for general storage and comparison. Use NFKC/NFKD only for loose search matching where folding is desired.
What does normalization fix?
It makes equivalent strings (e.g. precomposed vs combining-mark “é”) compare equal and sort predictably.