Character Encoding
Character sets, ASCII and Unicode, why the codes for letters and digits are in order, and why the character '7' is not the number 7.
Character sets
A computer stores only numbers, so text has to be turned into numbers first. A character set is an agreed table giving every character a unique binary code.
Two machines can only exchange text if they use the same character set. Send text encoded in one set and display it using another and you get the wrong characters, which is exactly what the box symbols and question marks in badly handled text are.
There is nothing about the number 65 that makes it an A. It is an A because everybody agreed it would be.
ASCII
ASCII is a 7 bit character set, so it has 2 to the power 7, which is 128 codes, numbered 0 to 127. That covers the upper and lower case English alphabet, the digits, punctuation and a set of control characters such as carriage return.
Extended ASCII uses 8 bits and so has 256 codes, adding accented letters and box drawing characters.
The codes are in order, and that is useful
'A'is 65, so'B'is 66 and'Z'is 90.'a'is 97, so lower case is always the upper case code plus 32.'0'is 48, so the numeric value of a digit character is its code minus 48.
Those relationships get examined, because they let you do things with arithmetic instead of a lookup table.
code ← CHAR_TO_CODE(letter)
IF code ≥ 65 AND code ≤ 90 THEN
letter ← CODE_TO_CHAR(code + 32)
ENDIFThe character '7' has the ASCII code 55. The integer 7 is stored as the value seven. They are different things, which is why input has to be cast before it can be used in arithmetic.
Step 1 of 4 Every character has a number. In ASCII, capital H is 72 and lower case i is 105.
- Each character has a code number, and the computer stores that number in binary.
- The codes run in order, so you can work out C from A. The character 7 is not the number 7.
- More bits per character means more characters: 128, then 256, then over a million.
Unicode
ASCII was designed for English. It has no code for Greek, Arabic, Chinese, Hindi, or for the symbols used in mathematics and music.
Unicode solves this by using far more bits per character, which gives it room for well over a million codes, enough for every writing system in current use and a great many that are not.
| ASCII | Unicode | |
|---|---|---|
| Bits per character | 7, or 8 for extended | Typically 8 to 32, variable |
| Number of characters | 128, or 256 extended | Over a million available |
| Languages covered | English and a little more | Effectively all of them |
| Storage per character | Small | Larger for characters outside the basic set |
The trade off is size. A document in Unicode can take more storage than the same document in ASCII, because some characters need more bytes. The first 128 Unicode codes are deliberately identical to ASCII, so plain English text costs the same in both and older files still open correctly.
Unicode represents far more characters, so it supports languages ASCII cannot, at the cost of using more bits per character and therefore more storage. Both halves are needed for full marks.
Key terms you must be able to define
- Character set
- An agreed table giving every character a unique binary code.
- ASCII
- A 7 bit character set with 128 codes, covering English letters, digits, punctuation and control characters.
- Extended ASCII
- An 8 bit version of ASCII with 256 codes.
- Unicode
- A character set using more bits per character, able to represent over a million characters from every writing system.
- Character code
- The number a character set assigns to a particular character.
- Control character
- A character code that causes an action, such as a new line, rather than printing a symbol.
Flashcards
Say the definition out loud before you flip the card. Retrieving it from memory is what builds recall, reading it again is not.
Reach the top grade on this topic
The examinable subtlety here is the difference between a character and the value it looks like, and the strongest answers make it explicit.
They say what is actually stored. Not "the letter A is stored" but "the code 65 is stored as 01000001, and software uses the same character set to draw an A when it is displayed".
They quantify Unicode's cost. A grade 9 comparison names the benefit and the price in the same sentence: more characters, therefore more languages, therefore more bits per character and a larger file.
They exploit the ordering. When a question asks how a program could convert case or find the numeric value of a digit character, the answer is arithmetic on the code, and saying so shows you understand what a character set is.
Mistakes that cost marks here
- Saying ASCII has 256 characters. Standard ASCII is 7 bit and has 128; extended ASCII has 256.
- Treating the character '5' as the number 5.
- Saying Unicode replaced ASCII entirely. The first 128 Unicode codes are the same as ASCII by design.
- Claiming Unicode always uses more storage. For plain English text in a variable width encoding it uses the same.
- Forgetting that both machines must use the same character set for text to display correctly.
The ASCII code for a capital letter and its lower case partner always differ by exactly 32, which is one bit. That was deliberate: it let 1960s hardware change case by flipping a single bit rather than by looking anything up.
Knowledge check
Exam-style questions
Five written questions in the style of the real paper. Write a full answer, then let the marker check it against the mark scheme.
- LAn agreed set of codes giving every character a unique binary value
A character set is an agreed table that gives every character a unique binary code, so that text can be stored and exchanged as numbers.
- LShows that the number of combinations is 2 to the power of the number of bits
- LGives 256
Each bit can be 0 or 1, so eight bits give 2 multiplied by itself eight times, which is 2 to the power 8. That is 256, so the character set can represent 256 different characters, numbered 0 to 255.
- LUnicode can represent far more characters
- LSo it supports languages and symbols ASCII has no codes for
- LUnicode uses more bits per character for characters outside the basic set
- LSo files can take more storage and take longer to transmit
The advantage is coverage. Unicode has room for well over a million codes, so it can represent Chinese, Arabic, Greek, mathematical symbols and emoji, none of which ASCII has any code for at all. That makes it possible to store and exchange text in any language rather than only in English. The disadvantage is size. Characters outside the basic Latin set need more than one byte, so a document containing them takes more storage and more bandwidth to send than the equivalent ASCII file would, if ASCII could represent it at all.
- LThe keyboard returns the character '9', not the number 9
- LThe character is stored as its character code, for example 57
- LAdding 1 either produces the next character or joins values rather than adding them
- LThe character must be converted to an integer first, for example with STRING_TO_INT
What arrives from the keyboard is the character '9', which is stored as its character set code rather than as the value nine. In ASCII that code is 57. Adding one to it therefore gives 58, which is the code for a colon, or in a language that treats the value as a string it joins the two values rather than adding them. Either way the result is not ten. The correction is to convert the character to an integer before doing arithmetic, for example value ← STRING_TO_INT(input), after which adding one gives ten as intended.