Representing Characters in computer science

In computer systems, characters are represented using various encoding schemes that map each character to a unique binary number. These representations are essential for processing text, as computers inherently work with binary data. This article explores the fundamental concepts behind character representation, including popular encoding schemes such as ASCII, Unicode, and UTF-8.

Understanding Character Encoding

Character Encoding is the process of converting characters (letters, numbers, symbols) into binary form, which computers can process and store. Each character is assigned a unique binary code based on the encoding scheme used.

Key Concepts:

• Binary Representation: Computers use binary (base-2) numbers, consisting of 0s and 1s, to represent data. Character encoding schemes map characters to specific binary sequences.
• Bit: The smallest unit of data in computing, representing a 0 or 1.
• Byte: A group of 8 bits, often used to represent a single character in many encoding schemes.

ASCII: The Basic Character Set

ASCII (American Standard Code for Information Interchange) is one of the earliest and most widely used character encoding schemes. It was developed in the 1960s and is based on a 7-bit binary code, allowing for 128 unique characters.

Key Features of ASCII:

• 7-bit Encoding: ASCII uses 7 bits to represent each character, which means it can represent $2^7 = 128$ unique characters.
• Character Set: ASCII includes:
  • Control Characters: Such as null (NUL), start of text (STX), and carriage return (CR), used for control purposes in data communication.
  • Printable Characters: Includes the uppercase and lowercase English alphabet (A-Z, a-z), digits (0-9), punctuation marks, and special symbols.

Example:

• The letter ‘A’ in ASCII is represented by the binary code $01000001_2$, which is $65_{10}$ in decimal.
• The letter ‘a’ is represented by $01100001_2$, which is $97_{10}$ in decimal.

ASCII Table

Here’s a table showing some common characters and their ASCII representations:

Unicode: A Universal Character Set

Unicode was developed to address the limitations of ASCII by providing a comprehensive encoding system that can represent characters from all languages and scripts. It uses a much larger set of binary codes to represent characters.

Key Features of Unicode:

• 16-bit Encoding (UTF-16): Unicode originally used a 16-bit encoding scheme, allowing for $2^{16} = 65,536$ characters. However, Unicode has since been expanded to include over a million possible characters through supplementary planes.
• Character Set: Unicode includes characters from virtually all writing systems, as well as symbols, emojis, and other special characters.

Example:

• The letter ‘A’ in Unicode (UTF-16) is represented by the same binary code as in ASCII: $00000000 01000001_2$, which is $0041_{16}$ in hexadecimal.
• The character ‘€’ (Euro sign) is represented by $00100000 10111100_2$, which is $20AC_{16}$ in hexadecimal.

Unicode Table

Here’s a table showing a few characters and their Unicode representations in UTF-16:

UTF-8: A Variable-Length Encoding Scheme

UTF-8 (Unicode Transformation Format – 8-bit) is a variable-length encoding scheme that encodes Unicode characters into one to four bytes. It is the most widely used encoding on the internet due to its efficiency and compatibility with ASCII.

Key Features of UTF-8:

• Variable-Length Encoding: UTF-8 uses one byte (8 bits) for the first 128 characters (which correspond to ASCII) and up to four bytes for other characters.
• Backward Compatibility: The first 128 characters in UTF-8 are identical to ASCII, making it compatible with systems that were originally designed for ASCII.
• Efficient Storage: UTF-8 is space-efficient, especially for texts that primarily use ASCII characters, as these are stored in a single byte.

Example:

• The letter ‘A’ in UTF-8 is represented by a single byte $01000001_2$ (same as in ASCII).
• The character ‘€’ is represented in UTF-8 by three bytes: $11100010 10000010 10101100_2$, which corresponds to the hexadecimal code $E2 82 AC_{16}$.

UTF-8 Table

Here’s a table showing the UTF-8 representation for some characters:

Mathematical Representation of Characters

Mathematically, characters can be seen as elements of a set, where each character is mapped to a unique binary number (code point). For example, let $C$ be the set of characters and $B$ the set of binary numbers. The encoding function $f$ maps each character $c \in C$ to a binary number $b \in B$, i.e.,

$f: C \rightarrow B$

In ASCII, the function maps the 7-bit binary number to each character, while in Unicode, the function maps a larger binary number to accommodate more characters.

Example of Encoding Function in ASCII:

Let $f$ be the encoding function for ASCII:

• $f(A) = 01000001_2$
• $f(B) = 01000010_2$
• $f(a) = 01100001_2$

For Unicode, the function would map characters to larger binary numbers, reflecting the broader range of characters available.

Importance of Character Encoding in Computing

Character encoding is crucial in computing because it ensures that text data is consistently represented and understood across different systems and platforms. Without standardized encoding, text data could become corrupted or misinterpreted when transferred between systems with different encoding schemes.

Conclusion

Character representation through encoding schemes like ASCII, Unicode, and UTF-8 is fundamental to computing. These schemes translate human-readable characters into binary form, enabling computers to process, store, and transmit text data. Understanding these encoding methods is essential for anyone working in computer science, data processing, or digital communication.

References

1. The Unicode Standard by The Unicode Consortium: This book provides a comprehensive overview of the Unicode character encoding standard, including its history, implementation, and applications.
2. Digital Design and Computer Architecture by David Harris and Sarah Harris: This textbook includes a detailed discussion of character encoding schemes, including ASCII, Unicode, and UTF-8, within the context of digital systems.
3. Computer Organization and Design: The Hardware/Software Interface by David A. Patterson and John L. Hennessy: This book covers character representation and encoding as part of its broader discussion on computer architecture and organization.
4. Unicode and Character Sets – W3C: An online resource by the World Wide Web Consortium (W3C) that explains Unicode, character sets, and encoding schemes. Available at: [W3C Unicode Overview](https://www.w3.org/International/articles/unicode/