UTF-8 and Unicode Code Points, Demystified

Unicode gives every character a number called a code point, and UTF-8 is a specific rule for turning those numbers into one to four bytes. The two ideas are constantly confused, but keeping them separate clears up almost every text bug you will meet. Unicode is a giant lookup table: it says the letter A is U+0041 and a particular emoji is U+1F600, and it stops there. UTF-8 answers a different question, namely how to write those numbers as actual bytes on disk or on the wire. Once you see that one layer names characters and the other encodes them, mojibake, wrong string lengths, and broken emoji stop being mysterious.

What is a Unicode code point?

A code point is the unique integer Unicode assigns to a character, conventionally written as U+ followed by hexadecimal digits. The letter A is U+0041, the euro sign is U+20AC, and a grinning face emoji is U+1F600. The number identifies the character in the abstract. It says nothing about which font draws it, how many bytes it occupies, or how it is stored; it is purely an identity.

Unicode’s code space runs from U+0000 up to U+10FFFF, which is a little over a million possible code points. Not all are assigned, and some ranges are reserved, but the space is deliberately vast so that every script, symbol, and emoji can have a permanent, unambiguous number. Crucially, the first 128 code points, U+0000 to U+007F, are exactly the old ASCII characters in their original order. That deliberate alignment is what makes the rest of the story work.

How does UTF-8 turn a code point into bytes?

UTF-8 encodes a code point into one to four bytes, using a self-describing pattern where the first byte announces how many bytes the character uses. Small code points get one byte; larger ones get more. The scheme is designed so a decoder can always tell, from any byte, whether it starts a character or continues one.

The x positions hold the actual bits of the code point. A leading byte starting with 0 is a lone ASCII character. A leading byte starting with 110, 1110, or 11110 announces a two-, three-, or four-byte character, and every following byte of that character starts with 10. Because continuation bytes are unmistakable, a decoder that lands in the middle of a stream can resynchronise by scanning forward to the next non-10 byte.

Can we walk through a real encoding?

Yes. Take the euro sign, U+20AC. Its code point is in the range U+0800 to U+FFFF, so UTF-8 uses the three-byte template 1110xxxx 10xxxxxx 10xxxxxx, which has sixteen x slots to fill.

Code point:  U+20AC  =  0x20AC  =  0010 0000 1010 1100  (16 bits)

Template:    1110xxxx 10xxxxxx 10xxxxxx
Fill bits:   1110 0010  10 000010  10 101100
                 ^^^^      ^^^^^^      ^^^^^^
             0010      000010      101100   <- the 16 code-point bits

Result bytes: 0xE2 0x82 0xAC

Split the sixteen bits of 0x20AC into groups of 4, 6, and 6, drop them into the x slots left to right, and you get the bytes E2 82 AC. Decoding reverses the process: see a byte starting 1110, read two more continuation bytes, strip the fixed prefixes, concatenate the payload bits, and recover U+20AC. The same procedure scales to one, two, or four bytes; only the template changes.

Why is UTF-8 backward compatible with ASCII?

UTF-8 is backward compatible because it was engineered so ASCII text encodes to identical bytes. Every ASCII character is a code point from U+0000 to U+007F, and UTF-8 encodes exactly that range as a single byte whose top bit is 0, which is the same value ASCII already used. So a file containing only ASCII is simultaneously valid ASCII and valid UTF-8, byte for byte.

This property was decisive for adoption. Decades of files, protocols, and programs assumed ASCII, and a new encoding that broke them all would have been dead on arrival. Because UTF-8 leaves ASCII untouched and uses only high-bit-set bytes for everything else, existing ASCII data kept working while the door opened to every other character. It is also why English-heavy text in UTF-8 is compact: common characters still cost one byte each.

How do UTF-8, UTF-16, and code points relate?

They are three answers to the same question, and mixing them up causes the classic string-length surprise. Code points are the abstract characters. UTF-8 encodes each code point as one to four bytes. UTF-16 encodes each code point as one or two 16-bit units, and code points above U+FFFF require a surrogate pair of two units.

The practical consequence appears with characters like emoji, which sit above U+FFFF. In a language whose strings are UTF-16 under the hood, a single grinning-face emoji has a length of two, because it occupies two 16-bit units, even though it is one code point and one character a human sees. In UTF-8 the same emoji is four bytes, and if you measure length in bytes you get four, not one. None of these are wrong; they measure different layers. When a length looks off, ask which layer you are counting: characters, code points, UTF-16 units, or bytes.

What causes mojibake, and how do you avoid it?

Mojibake is what you get when bytes encoded one way are decoded another way, so the characters come out as garbage. The classic case is UTF-8 bytes read as if they were a single-byte legacy encoding: a two-byte accented letter shows up as two unrelated symbols, because the decoder treated each byte as its own character instead of recognising the multi-byte sequence. The bytes were never wrong; only the interpretation was.

Avoiding it comes down to agreeing on the encoding at every hop. Declare UTF-8 explicitly in your files, HTTP headers, and database connections, and make sure your editor saves in UTF-8 rather than a platform default. When text arrives from an external source, treat its declared encoding as authoritative and decode with that, rather than guessing. The moment two parts of a system disagree about the encoding, mojibake appears, so the fix is always to make the agreement explicit rather than implicit.

A related wrinkle is the byte order mark, or BOM, a code point (U+FEFF) that some tools place at the very start of a file to signal the encoding. In UTF-8 a BOM is optional and usually unnecessary, and a stray leading BOM can cause subtle bugs, such as an unexpected invisible character before the first real one or a configuration file that fails to parse. If a file misbehaves at exactly its first character, an unexpected BOM is a prime suspect. Inspecting the raw bytes reveals it immediately, since it appears as the sequence EF BB BF at the start.

How do you inspect this yourself?

The fastest way to build intuition is to convert a character back and forth between its literal form, its code point, and its bytes and watch the pieces line up. You can paste text into our Unicode converter to see each character’s code point and encoded bytes, which makes the one-to-four-byte pattern concrete and exposes hidden characters that print as nothing. It runs entirely in your browser, so text you are debugging, which may be private, never leaves your machine.

Keep the two-layer model in mind and the whole subject settles. Unicode names characters with code points; UTF-8 is one faithful way to write those code points as bytes, chosen because it preserves ASCII and stays compact for common text. When something looks broken, mojibake usually means bytes were decoded with the wrong encoding, and a surprising length usually means you counted a different layer than you thought. Separate the naming from the encoding and the confusion evaporates.