Character Encoding

Characters, Graphemes, and Glyphs

A character is a unit of information in textual data. A character has a name. Examples:

• PLUS SIGN
• CYRILLIC SMALL LETTER TSE
• CHEROKEE LETTER TLO
• BLACK CHESS KNIGHT
• PITCHFORK
• MUSICAL SYMBOL FERMATA BELOW

A grapheme is a minimally distinctive unit of writing in some writing system. It is what a person usually thinks of as a character. However, it may take more than one character to make up a grapheme. For example, the grapheme:

R̊

is made up of two characters (1) LATIN CAPITAL LETTER R and (2) COMBINING RING ABOVE. The grapheme:

நி

is made up of two characters (1) TAMIL LETTER NA and (2) TAMIL VOWEL SIGN I. This grapheme:

🚴🏾

is made up of two characters (1) BICYCLIST and (2) EMOJI MODIFIER FITZPATRICK TYPE-5. This grapheme:

🏄🏻‍♀

is made up of four characters (1) SURFER, (2) EMOJI MODIFIER FITZPATRICK TYPE-1-2, (3) ZERO-WIDTH JOINER, (4) FEMALE SIGN. And this grapheme:

🇨🇻

requires two characters: (1) REGIONAL INDICATOR SYMBOL LETTER C and (2) REGIONAL INDICATOR SYMBOL LETTER V. It’s the flag for Cape Verde (CV).

Finally, a glyph is a picture of a character (or grapheme). Two or more characters can share the same glyph (e.g. LATIN CAPITAL LETTER A and GREEK CAPITAL LETTER ALPHA), and one character can have many glyphs (think fonts, e.g., A A A $\mathcal{A}$ $\mathscr{A}$).

Character Sets

A character set has two parts: (1) a repertoire, which is a set of characters, and (2) a code position mapping, which is a function mapping non-negative integers to characters in the repertoire. When an integer $i$ maps to a character $c$ we say $i$ is the code point of $c$.

Unicode

An example character set is Unicode. Here is part of its code point mapping (note that code points are traditionally written in hex):

   25 PERCENT SIGN
   2C COMMA
   54 LATIN CAPITAL LETTER T
   5D RIGHT SQUARE BRACKET
   B0 DEGREE SIGN
   C9 LATIN CAPITAL LETTER E WITH ACUTE
  2AD LATIN LETTER BIDENTAL PERCUSSIVE
  39B GREEK CAPITAL LETTER LAMDA
  446 CYRILLIC SMALL LETTER TSE
  543 ARMENIAN CAPITAL LETTER CHEH
  5E6 HEBREW LETTER TSADI
  635 ARABIC LETTER SAD
  71D SYRIAC LETTER YUDH
  784 THAANA LETTER BAA
  94A DEVANAGARI VOWEL SIGN SHORT O
  9D7 BENGALI AU LENGTH MARK
  BEF TAMIL DIGIT NINE
  D93 SINHALA LETTER AIYANNA
  F0A TIBETAN MARK BKA- SHOG YIG MGO
 11C7 HANGUL JONGSEONG NIEUN-SIOS
 1293 ETHIOPIC SYLLABLE NAA
 13CB CHEROKEE LETTER QUV
 2023 TRIANGULAR BULLET
 20A4 LIRA SIGN
 20B4 HRYVNIA SIGN
 2105 CARE OF
 213A ROTATED CAPITAL Q
 21B7 CLOCKWISE TOP SEMICIRCLE ARROW
 2226 NOT PARALLEL TO
 2234 THEREFORE
 2248 ALMOST EQUAL TO
 265E BLACK CHESS KNIGHT
 30FE KATAKANA VOICED ITERATION MARK
 4A9D HAN CHARACTER LEATHER THONG WOUND AROUND THE HANDLE OF A SWORD
 7734 HAN CHARACTER DAZZLED
 99ED HAN CHARACTER TERRIFY, FRIGHTEN, SCARE, SHOCK
 AAB9 TAI VIET VOWEL UEA
1201F CUNEIFORM SIGN AK TIMES SHITA PLUS GISH
1D111 MUSICAL SYMBOL FERMATA BELOW
1D122 MUSICAL SYMBOL F CLEF
1F08E DOMINO TILE VERTICAL-06-01
1F001 SQUID
1F0CE PLAYING CARD KING OF DIAMONDS
1F382 BIRTHDAY CAKE
1F353 STRAWBERRY
1F4A9 PILE OF POO

Because characters can have multiple glyphs, Unicode lets you represent characters unambiguously with U+ followed by four to six hex digits (e.g. U+00C9, U+1D122).

How many characters are there?

How many characters are possible with Unicode? The powers that be have declared that the highest code point they will ever map is 10FFFF, so it seems like 1,114,112 code points are possible. However, they also declared they would never map characters to code points D800..DFFF, FDD0..FDEF, {0..F}FFFE, {0..F}FFFF, 10FFFE, 10FFFF (2114 code points). So the maximum number of possible characters is 1,111,998.

How many characters have been assigned (so far)? See this cool table at Wikipedia to see how many characters have actually been mapped in each version of Unicode. (BTW, if you don’t peek, Unicode Version 10.0 has 136,755 characters mapped, so there is a lot of room to grow.)

Where can I find all of them?

Please see the complete and up-to-date code charts, with example glyphs of every character. If you would like a much easier way to browse the characters (and why wouldn’t you?), check out the beautiful charbase.com and codepoints.net. (Charbase and Codepoints are run by volunteers, so they may not be up to date.)

Blocks

The code points are not assigned haphazardly. They are allocated into blocks. There are around 300 blocks. Make sure to see the official blocks file at Unicode.org. If you need a little preview, here is a little slice of that file:

0000..007F     Basic Latin
0080..00FF     Latin-1 Supplement
0250..02AF     IPA Extensions
0400..04FF     Cyrillic
0F00..0FFF     Tibetan
2200..22FF     Mathematical Operators
2700..27BF     Dingbats
3100..312F     Bopomofo
10860..1087F   Palmyrene
12000..123FF   Cuneiform
1D100..1D1FF   Musical Symbols
1F000..1F02F   Mahjong Tiles
F0000..FFFFF   Supplementary Private Use Area-A
100000..10FFFF Supplementary Private Use Area-B

Categories

Every Unicode code point is assigned various properties, such as name and category. Here are the categories:

Combining Characters, Modifiers, and Variation Selectors

It’s common to use multiple characters to make up graphemes. This is often done to put marks on letters, modify skin tones, modify genders, and create more complex graphemes. Examples:

See all the emojis at Unicode.org.

Where can I get a list of all the fun stuff?

Awesome Codepoints is super cool.

Normalization

Here are two character sequences:

• U+0065 LATIN SMALL LETTER E, U+0300 COMBINING GRAVE ACCENT
• U+00E8 LATIN SMALL LETTER E WITH GRAVE

In some sense they should represent the same text, right? In fact, they only differ in that the second sequence has a pre-composed character. Since the character sequences are different, we can make get them to compare equal only if we normalize them. Unicode defines four Normalization Forms.

WHAT? Okay, take it slow. Canonical decomposition and composition does the right thing. Compatibility is lossy (one-way) but useful for search: It turns things like ligatures, roman numerals, subscripts, and superscripts into simpler forms. See why it’s great for search?

Example:

Other Character Sets

Unicode is really the only character set you should be working with. However, other character sets exist, and you should probably know something about them.

ISO8859-1

ISO8859-1 is a character set that is exactly equivalent to the first 256 mappings of Unicode. Obviously it doesn’t have enough characters.

ISO8859-2 through ISO8859-16

These 15 charsets also have 256-character repertoires. They all share the same characters in the first 128 positions, but differ in the next 128. Details at http://www.unicode.org/Public/MAPPINGS/ISO8859/.

Windows-1252

This character set, with a repertoire of 256 characters, also known as CP1252, can be found at http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1252.TXT. It is very close to ISO8859-1. Be careful with this set! Users of Windows systems often unknowingly produce documents with this character set, then forget to specify it when making these documents available on the web or transporting them via other protocols with tend to default to Unicode. Then the end result is annoying. It’s best to avoid this.

ASCII

ASCII is a character set that is exactly equivalent to the first 128 mappings of Unicode. Obviously it doesn’t have enough characters. However it is commonly used, because many Internet protocols require it! It is a common subset of many character sets and something most people can agree on.

Character Encodings

A character encoding specifies how a character (or character string) is encoded in a bit string. Character sets with 256 characters or less have only a single encoding: they encode each character in a single byte (so in some sense the character set and the character encode are about the same). But there are, naturally, many encodings of Unicode. The most important are UTF-32, UTF-16 and UTF-8.

UTF-32

This is the simplest. Just encode each character in 32 bits. The encoding of a character is simply its code point! Couldn’t be more straightforward. Of course, you try to convince people to actually use four bytes per character.

There are actually two kinds: UTF-32BE (Big Endian) and UTF-32LE (Little Endian). Examples:

Note that every character sequence can be encoded into a byte sequence, but not every byte sequence can be decoded into a character sequence. For example, the byte sequence CC CC CC CC does not decode to any character because there is not code point CCCCCCCC in Unicode.

UTF-32
The Good: It’s fixed width! Constant-time to find the nth character in a string (provided you care).
The Bad: It’s pretty bloated.

UTF-16

In UTF-16 some characters are encoded in 16 bits and some in 32 bits.

Note that all characters requiring 32-bits have their first 16 bits in the range D800..D8FF. Pretty slick, right? Those code points are never assigned to any character in Unicode.... How perfect, dontcha think?

There are actually two kinds: UTF-16BE and UTF-16LE. Examples:

UTF-16
The Good: Nothing is good about this encoding.
The Bad: Variable width, almost always uses more space than UTF-8, even for East Asian scripts, people.
The Ugly: It’s what JavaScript thinks characters are. Facepalm.

UTF-8

Here’s another variable length encoding.

Examples:

And yeah, you never have to care about big-endian or little-endian with UTF-8.

All character sequences can be encoded as UTF-8 byte sequences, but many byte sequences cannot be decoded as UTF-8. Given a 32-bit unsigned integer, you can use this pseudocode to get its (1–4) bytes:

void to_utf_8(uint32 c) {
    if (c > 0x10FFFF) {
        throw new Error('Code point too large');
    } else if (c <= 0x7F) {
        emit_byte(c);
    } else if (c <= 0x7FF) {
        emit_byte(0xC0 | c>>6);
        emit_byte(0x80 | c & 0x3F);
    } else if (c <= 0xFFFF) {
        emit_byte(0xE0 | c>>12);
        emit_byte(0x80 | c>>6 & 0x3F);
        emit_byte(0x80 | c & 0x3F);
    } else {
        emit_byte(0xF0 | c>>18);
        emit_byte(0x80 | c>>12 & 0x3F);
        emit_byte(0x80 | c>>6 & 0x3F);
        emit_byte(0x80 | c & 0x3F);
    }
}

Here’s Tom Scott’s overview of UTF-8:

You should read the UTF-8 Everywhere Manifesto.

How Can You Determine The Encoding?

A huge and annoying mistake people make is giving people text and not telling others what encoding it is in. For example, if you type in the text:

“Olé”™

into a text editor that encodes your text in UTF-8, then you have stored the data:

E2 80 9C 4F 6C C3 A9 E2 80 9D E2 84 A2

Now if you didn’t know the encoding and you just assumed the data was encoded in Windows-1252, you would decode the bytes as follows:

“Olé�™

Actually that was cheating a little bit: that � isn’t really correct; it’s a placeholder because there is no character in Windows-1252 for the code point 9D. Oh, well. Anyway, conversely, suppose you started with “Olé”™ and encoded this in Windows-1252. This gives the byte sequence:

93 4F 6C E9 94 99

If you tried to decode this as a UTF-8 character sequence you would get an error. Decoding as a UTF-16BE sequence you would get:

鍏泩钙

IMPORTANT

When transmitting textual data, always specify the encoding.

But how?

• Mutual agreement. You can just agree, implicitly, upon an encoding. Usually people agree on UTF-8. In fact, there is a manifesto that says everyone should just use UTF-8 all the time, everywhere.
• In metadata, separate from the text. Many protocols separate metadata, or headers, from a payload. The metadata can contain the encoding to use for the payload. (But yeah, in this case you have to agree beforehand on how the metadata is encoded....)
• In the data itself (woah). The character U+FEFF ZERO WIDTH NO-BREAK SPACE, alternately named BYTE ORDER MARK, or BOM, is allowed to appear at the start of a text stream. If it does, it can be used by the stream consumer to deduce the Unicode encoding. For example, if a stream starts with
  • 00 00 FE FF then the stream is most likely UTF-32BE.
  • FF FE 00 00 then the stream is most likely UTF-32LE.
  • FE FF then the stream is most likely UTF-16BE.
  • FF FE then the stream is most likely UTF-16LE.
  • EF BB BF then the stream is most likely UTF-8. Note: The Unicode Standard permits the BOM, but does not require it, nor does it recommend it.

Other Encodings

I won’t describe any others here, but UTF-7 is worth mentioning. If you like the stuff on this page see the IANA Charsets Page. You may also want to check out the UTF page at czybrra.com, which is very complete and well-written (and from which I borrowed the list of UTF-8 advantages).

Programming with Strings

How hard can this be? Do you know the answers to these two simple questions?

1. What is the length of a string? Oy. Maybe it is the number of graphemes? Or the number of characters? Or the number of bytes when encoded as UTF-8? Or the number of code units in a UTF-16 encoding? Does the BOM count at all? Oh and wait: do we count pre-composed characters as one character or multiple characters or.... AAAAAAAAHHHHHHHHHHHHH.
2. When are two strings equal? Yes there is the whole nastiness of whether we have reference equality (same string object) vs. value equality (same value), but what the heck does it mean to compare strings by value? Should we normalize first? If so, which normalization format should be use?

Well, it’s hard. But if you’ve read this far, you at least are now better able to debug problems.

But wait, think about this a bit. Maybe the fact that these questions are hard means these are the wrong questions. Really. Consider:

1. Why do you care how long a string is? You should really only care about (1) how many bytes of storage are needed for it, or (2) how many pixels wide your the rendered string is going to take up. The number of characters is usually irrelevant!
2. Why are you comparing strings for equality anyway? Are you checking passwords? YOU BETTER BE HASHING THOSE, so then you will be comparing byte sequences! Are you looking up dictionary keys, or doing a search? In these cases, printable, normalized characters will be just fine. (Use normalization when building the search index and when parsing the query string.)

Aren’t these problems with strings widely known? Well, Edaqa Motoray has written about this. He says programming languages dont’t need a string type at all. And in fact, if your language does have one, it is probably badly broken. It helps, if you want to get a handle on all this, that strings and text are not the same.

Further Reading

Unicode is a huge topic. The standard is over 1,000 pages. There is a ridiculous amount to learn. These notes have barely scratched the surface. Maybe you would like to go deeper. Maybe you would like to know everything there is to know.

Start with these articles, and follow links.

• Introduction to Unicode by Mathias Gaunard
• A Programmer’s Introduction to Unicode by Nathan Reed
• Dark Corners of Unicode by Eevee

Got weeks of free time? You can read the entire one-thousand page standard.