Character Sets

Character sets are standards that map a specific set of real world characters (code elements) to specific numbers (code points) and then encode those code points as bytes (character encoding or encoding schemes). Older character sets corresponded to the characters commonly used in a specific language. Unicode is the modern character set which tries to encompass most languages. A collation is a set of rules for comparing (esp. sorting) characters in a character set or database. EG:

• The Yen code element is ¥.
• The Yen's code point in different code point systems:
  • ASCII: out of range.
  • Window ANSI 1252: 165 = 0xA5.
  • ISO Latin-1 (iso-8859-1): 165 = 0xA5.
  • Unicode/UCS-2: 165 = 0xA5 = U+00A5.
  • Unicode/UCS-4: 165 = 0xA5 = U+000000A5
• The Yen is character encoded this way in different encoding schemes:
  • ASCII: out of range.
  • Windows ANSI 1252: 0xA5 = 10100101.
  • ISO Latin-1 (iso-8859-1): 0xA5 = 10100101.
  • UTF-8: 11000010 10100101.
  • UTF-16: 00000000 10100101.
  • UTF-32: 00000000 00000000 00000000 10100101

A document consists of binary data. A document can explicitly state the character set for itself or parts of itself (see Web Character Sets), but a viewer, like a browser, can also "override" and decode the digital data using its own choice of character encoding. EG: These two bytes (xCE xB1) will be displayed differently in different by browsers depending upon the "Character Encoding chosen for the page by the viewer. In Firefox, go to the menu for View > Character Encoding. In Internet Explorer, go to the menu for View > Encoding.

• Digital data:
  • 1100111010110001. In binary.
  • 206 177. In decimal.
  • CE B1. In hexadecimal.
• Displayed as:
  • α. The raw hexadecimal attempted as HTML NCR (Numeric Character Reference): α.
  • = capital I with circumflex and plus-minus sign in Western European (ISO; iso-8859-1; Windows; windows-1252;).
  • = alpha in Multilingual (utf-8).
  • = qia4 in Traditional Chinese (big5).
  • = wei3 in Simplified Chinese (bg).
  • = ryuu in Japanese (euc-jp).
  • = gwan in Korean (euc-kr).

Once a set of bits have been mapped to a code point, then that code point can be mapped to its code element character, and then software can present that character and apply different fonts, styles, and sizes. EG:

In the ASCII character map the code element J has a code point of 74 (x4A). This code point is encoded as the binary number 1001010 (which is the x4A). The binary code can then be interpreted by programs such as browsers and presented to users:

You may also want to see my articles on Typography.

Choosing a character set is important especially if you have to deal with international code, multiple platforms, or databases. EG:

When SQL Server is installed, a Sort Order ID must be set that is based on a character code. If you try to restore a database to that installation, then it must have the same Sort Order ID. Otherwise you may have to rebuild the database, using something like rebuildm.exe.

Here are some of the major character sets:

• ASCII (American Standard Code for Information Interchange, aka Standard ASCII; plain text, ISO-646). A SBCS (Single Byte Character Set) that uses 7 bits of a byte (x0 to x7F = 0 to 127) to make 128 code points (since 2⁷). The 8th bit was used for parity checking or for anything an application wanted to do with it. Used by Linux/Unix.
• High ASCII are SBCSs that use the 8th bit to encode additional characters by utilizing the bit above the ASCII range.  Here are SBCSs that use all 8 bits of a byte (0-255; x0-FF) to add 128 more code points (since 2⁸ = 256). There are different character sets (aka code pages) for different uses, languages or language groups. Many of the character sets are super sets of ASCII.
  • OEM character sets (Original Equipment Manufacturer). The 8th bit often contained characters for line drawings as a carry over from pre-GUI (Graphics User Interface) days. Used by DOS, OS/2, floppy disks, and the FAT system (File Allocation System).
  • ISO character sets (International Organization for Standardization). There are several ISO character sets but ISO Latin 1 (iso-8859-1) is probably the most prevalent.
  • ANSI character sets (American National Standards Institute). Used by the Windows OS.
• Unicode, aka UCS (Universal Character Set). A MBCS (Multi-Byte Character Set) that uses 2-4 bytes worth of possible code points. The code points may be encoded using a 1-6 bytes per character. Unicode is the sensible international and cross-platform character set. Used by Windows NT/2000 and Linux/Unix.
  • Windows Glyph List 4 (WGL4) [W]. Aka the Pan-European character set. The WGL4 are 652 Unicode characters that are guaranteed to display on all major platforms of Microsoft Windows. It consists primarily of code pages 1252, 1250, 1251, 1253, 1254, 1257, and 437.

Here is a summary table of the major character sets:

Here are some languages and the ISO character sheets they use. (And all of them can use Unicode.)

Different companies refer to a character set by an identifier, often called a code page number. Here are some of the available code page identifiers used by Windows.

Note that an alias in bold is the preferred charset ID for the HTML <meta> tag:

<meta http-equiv="Content-Type" content="text/html; charset=characterSet">

<!-- Used to explicitly state the character set used.
	Examples of characterSet include windows-1252, iso-8859-1, and utf-8. / -->

Systems save EOLs (End Of Lines) and EOFs (End Of Files) differently.

• Systems like Linux, Unix-like, Multics, BeOS, Amiga. The "correct" way.
  • EOL: LF. Aka Line Feed; ^j; NL; New Line; \n; 0x0A; Chr(10), U+000A.
  • EOF: No particular character.
• Systems like Mac OS (up to 9), Commodore, TRS-80, Apple II, OS-9.
  • EOL: CR. Aka Carriage Return; ^m; \r; 0x0D; Chr(13), U+000D.
  • EOF: No particular character.
• Systems like Windows, MS-DOS, OS/2. A carry over from ye olde days when computer terminals were more physical than virtual and the extra characters were used to buy time for the machine to physical relocate to the next line.
  • EOL: CR and LF.
  • EOF: SUB. Aka Substitute; ^z; Chr(26).
• Other line terminator-like characters:
  • FF: Form Feed, U+000C, \f, New Page, NP, 0x0C, ^l.
  • NEL: Next Line, U+0085.
  • LS: Lind Separator, U+2028.
  • PS: Paragraph Separator, U+2029.

Page Modified: (Hand noted: 2006-06-07 12:29:31Z) (Auto noted: 2010-04-02 15:03:28Z)