Character Sets

Character sets are standards that map specific real world characters (code elements) to specific numbers (code points) and then encode those code points as bytes (character encoding or encoding schemes). EG:

• The Yen code element is ¥.
• The Yen's code point in different code point systems:
  • ASCII: out of range.
  • Window ANSI 1252: 165 = 0xA5.
  • ISO Latin-1 (iso-8859-1): 165 = 0xA5.
  • Unicode/UCS-2: 165 = 0xA5 = U+00A5.
  • Unicode/UCS-4: 165 = 0xA5 = U+000000A5
• The Yen is character encoded this way in different encoding schemes:
  • ASCII: out of range.
  • Windows ANSI 1252: 0xA5 = 10100101.
  • ISO Latin-1 (iso-8859-1): 0xA5 = 10100101.
  • UTF-8: 11000010 10100101.
  • UTF-16: 00000000 10100101.
  • UTF-32: 00000000 00000000 00000000 10100101

A document consists of binary data. A document can explicitly state the character set for itself or parts of itself (see Web Character Sets), but a viewer, like a browser, can also "override" and decode the digital data using its own choice of character encoding. EG: These two bytes (xCE xB1) will be displayed differently in different by browsers depending upon the "Character Encoding chosen for the page by the viewer. In Firefox, go to the menu for View > Character Encoding. In Internet Explorer, go to the menu for View > Encoding.

• Digital data:
  • 1100111010110001. In binary.
  • 206 177. In decimal.
  • CE B1. In hexadecimal.
• Displayed as:
  • α. The raw hexadecimal attempted as HTML NCR (Numeric Character Reference): α.
  • = capital I with circumflex and plus-minus sign in Western European (ISO; iso-8859-1; Windows; windows-1252;).
  • = alpha in Multilingual (utf-8).
  • = qia4 in Traditional Chinese (big5).
  • = wei3 in Simplified Chinese (bg).
  • = ryuu in Japanese (euc-jp).
  • = gwan in Korean (euc-kr).

Once a set of bits have been mapped to a code point, then that code point can be mapped to its code element character, and then software can present that character and apply different fonts, styles, and sizes. EG:

In the ASCII character map the code element J has a code point of 74 (x4A). This code point is encoded as the binary number 1001010 (which is the x4A). The binary code can then be interpreted by programs such as browsers and presented to users:

You may also want to see my articles on Typography.

Choosing a character set is important especially if you have to deal with international code, multiple platforms, or databases. EG:

When SQL Server is installed, a Sort Order ID must be set that is based on a character code. If you try to restore a database to that installation, then it must have the same Sort Order ID. Otherwise you may have to rebuild the database, using something like rebuildm.exe.

Here are some of the major character sets:

• ASCII (American Standard Code for Information Interchange, aka Standard ASCII; plain text, ISO-646). A SBCS (Single Byte Character Set) that uses 7 bits of a byte (x0 to x7F = 0 to 127) to make 128 code points (since 2⁷). The 8th bit was used for parity checking or for anything an application wanted to do with it. Used by Linux/Unix.
• High ASCII are SBCSs that use the 8th bit to encode additional characters by utilizing the bit above the ASCII range.  Here are SBCSs that use all 8 bits of a byte (0-255; x0-FF) to add 128 more code points (since 2⁸ = 256). There are different character sets (aka code pages) for different uses, languages or language groups. Many of the character sets are super sets of ASCII.
  • OEM character sets (Original Equipment Manufacturer). The 8th bit often contained characters for line drawings as a carry over from pre-GUI (Graphics User Interface) days. Used by DOS, OS/2, floppy disks, and the FAT system (File Allocation System).
  • ISO character sets (International Organization for Standardization). There are several ISO character sets but ISO Latin 1 (iso-8859-1) is probably the most prevalent.
  • ANSI character sets (American National Standards Institute). Used by the Windows OS.
• Unicode, aka UCS (Universal Character Set). A MBCS (Multi-Byte Character Set) that uses 2-4 bytes worth of possible code points. The code points may be encoded using a 1-6 bytes per character. Unicode is the sensible international and cross-platform character set. Used by Windows NT/2000 and Linux/Unix.

Here is a summary table of the major character sets:

Here are some languages and the ISO character sheets they use. (And all of them can use Unicode.)

Here are some of the available code page identifiers used by Windows.

Note that aliases in bold is the preferred charset ID for the HTML <meta> tag:

<meta http-equiv="Content-Type" content="text/html; charset=characterSet">
    <!-- Used to explicitly state the character set used.
         Examples of characterSet include windows-1252, iso-8859-1, and utf-8. / -->

Systems save EOLs (End Of Lines) and EOFs (End Of Files) differently.

• Windows systems:
  • EOL: CR and LF. The former is aka Carriage Return; ^m; \r; 0x0D; Chr(13). The latter is aka Line Feed; New Line; ^j; \n; 0x0A;  Chr(10).
  • EOF: SUB. Aka Substitute; ^z; Chr(26).
• Unix/Linux systems:
  • EOL: LF.
  • EOF: No particular character.
• Macintosh systems:
  • EOL: CR.
  • EOF: No particular character.
• SGML systems:
  • Start of each line: LF.
  • EOL: CR.
  • EOF: No particular character.

2006-06-07 12:29:31Z