Computers use binary bit patterns to represent, not only numbers, but also characters. A text file contains binary bit patterns that map to printable characters according to some mapping table. While the binary bit pattern all-zeroes, 00000000, usually represents an integer zero, a printable zero character, i.e. '0', usually is not encoded in a text file using the zero bit pattern. (For example, ASCII uses the 7-bit pattern 0x30 to encode a printable zero digit. EBCDIC uses the 8-bit pattern 0xF0. These are not the same bit patterns used to represent the integer value zero.)
Originally, here in North America, the bit patterns being used for characters only handled English. English only needed a 7-bit or 8-bit character size to include all the letters, digits, and common punctuation, so these character bit patterns all fit nicely into one computer "byte".
The mapping of bit pattern to printable character has gotten complex over the past decades due to the introduction of more and more different mappings to include more and more of the world's languages, current and past. (Not all the world speaks English!) For many years, the field was reluctant to break the rule "one-character, one-byte", so many mutually incompatible 8-bit character mappings were developed to handle different languages in different parts of the world. The same 8-bit pattern might map to one printable character in Norway, and a different character in France or Greece. Creating a file containing both French and Greek characters was impossible.
In 1991, a Universal Character set Unicode was introduced, using a 16-bit (two-byte) character format, with later updates to permit extensions to 32-bit characters as needed. This 16-bit character set broke the "one-character, one-byte" rule. It was incompatible with all one-byte character systems used to date, and thus rendered much text manipulation software (sorting, indexing, etc.) useless.
In 1993, some Americans introduced an UTF-8, an 8-bit version of Unicode that was backwards-compatible with ASCII, and thus with English. If you didn't use any non-English characters in your file, the file format was plain 7-bit ASCII. Only when you needed a foreign Unicode character did you have to resort to some 8-bit encoding sequences. Most software that expected ASCII could handle UTF-8 equally well. UTF-8 has become very popular in North America, since it treats ASCII as ASCII with no complications.
Character encodings:
Students should know from memory the basic layout of the letters and digits in the 7-bit ASCII character encoding table. What region of the table contains unprintable control characters? What is the ASCII value of a space? the letters "a" and "A"? What is the lowest standard-ASCII (7-bit) character's name and bit pattern? What is the highest standard-ASCII (7-bit) character's name and bit pattern?
The American Standard Code for Information Interchange (ASCII) coding scheme was developed as a 7-bit code. A 7-bit code provides enough different bit patterns (128) to permit a coding scheme for all the upper- and lower-case characters found on a standard English language keyboard, plus punctuation and some unprintable device control characters (e.g. Newline, Carriage Return, Bell, etc.).
Seven-bit ASCII encoding is normally used in 8-bit bytes with the top (leftmost) bit set to zero. Some extended encodings based on ASCII use the top bit set to include an additional 128 characters, e.g. ISO-8859-1 (Latin-1) is an 8-bit standard that includes the accented letters needed for Western European languages (including French).
Before the development of standards for extended-ASCII encodings, each manufacturer of computer equipment used different incompatible choices for what the extended characters represented. Files written on one machine don't display properly on another.
A byte is the collection of bits used by a particular computer for the most common character encoding scheme used by that computer. The most common byte size is 8 bits, but 6, 7, 9, and 12-bit bytes are used by some (different) computer systems.
Most ASCII encoding/decoding can be performed without tables by knowing a few base codes: the blank, the letter "A", the digit "0", the carriage-return, and the line-feed. The rest of the letters and digits can be figured out from these base codes.
| Low\Hi
Nybbles |
0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| 0 | NUL | DLE | SPACE | 0 | @ | P | ` | p |
| 1 | SOH | DC1 | ! | 1 | A | Q | a | q |
| 2 | STX | DC2 | " | 2 | B | R | b | r |
| 3 | ETX | DC3 | # | 3 | C | S | c | s |
| 4 | EOT | DC4 | $ | 4 | D | T | d | t |
| 5 | ENQ | NAK | % | 5 | E | U | e | u |
| 6 | ACK | SYN | & | 6 | F | V | f | v |
| 7 | BEL | ETB | ' | 7 | G | W | g | w |
| 8 | BS | CAN | ( | 8 | H | X | h | x |
| 9 | HT | EM | ) | 9 | I | Y | i | y |
| A | LF | SUB | * | : | J | Z | j | z |
| B | VT | ESC | + | ; | K | [ | k | { |
| C | FF | FS | , | < | L | \ | l | | |
| D | CR | GS | - | = | M | ] | m | } |
| E | SO | RS | . | > | N | ^ | n | ~ |
| F | SI | US | / | ? | O | _ | o | DEL |
Why it matters - If you use a package like FTP to download a web-based "Text" (i.e. ASCII) file (where Unix is the assumed operating system) to an MS-DOS based system, the FTP process will automatically scan the file and replace each "line feed" character with a "carriage return"-"line feed" pair. If the file originated from an MS-DOS based system (and was uploaded to the web), then it already has a "carriage return" for each line. As a result, the downloaded file would now have two "carriage return" codes and a "line feed" code; for many editors this will cause the file to appear to be double spaced. (One way around this is to transfer your files as "Binary" instead of "Text" so FTP does not attempt any character expansion).
ASCII-encoded files are usually composed of variable length lines of characters. Each line is terminated with one or more unprintable characters. The exact character or characters used at the end of each line depends on what computer system you are using.
When using a file-transfer program to move text (not binary) files between machines, you must know the consequences of incompatible line-end terminators.
41 20 73 69 6D 70 6C 65 0D 0A 33 20 6C 69 6E 65 0D 0A 66 69 6C 65 0D 0A
A simple 3 line file
Character encoded data on IBM mainframe computers is normally based on a scheme called EBCDIC. The EBCDIC character encoding preceded the ASCII encoding. EBCDIC was developed from a basis that involved the computer punched card and has features that, to be properly understood, require a knowledge of that historical medium.
EBCDIC encoded files normally contain fixed-length records.
For a side-by-side comparison, see: http://www.natural-innovations.com/computing/asciiebcdic.html
Q: If you examine an EBCDIC text file copied byte-for-byte onto an ASCII system such as Unix/Linux or DOS/Windows/Macintosh, what will you see on your ASCII screen? (Hints: [1] Do the EBCDIC letters and numbers match any printable 7-bit ASCII characters? [2] Do EBCDIC sentence punctuation and space characters match any printable ASCII characters?)
From: ib1 at teksavvy dot com To: idallen@idallen.ca Subject: 120_CharacterEncoding Date: 2010-10-16T15:21:27Z I happened (goofing off, y'know) to wander into teaching.idallen.com/dat2343/10f/notes/120_CharacterEncoding.html and I saw some errors and oddities on that page. In the "Character encodings:" overview table, EBCDIC is described as "English only". EBCDIC was and is routinely used with more-than-ASCII character sets, using the extra code points for non-English letters, APL, line-drawing, etc. (IBM majorly messed up by not imposing cross-company standards for assigning characters beyond the basic (ASCII-equivalent) code points, so customers had trouble when porting documents between various World Trade divisions, and in/out of Domestic.) In the paragraph starting "Before the development of standards", you say that "each manufacturer of computer equipment used different incompatible choices". The situation was actually worse than you say, since there were situations where one manufacturer had multiple conventions for mapping characters to code points, even within one device. The discussion of text file transfer via FTP conflicts with my observations and perception. If transfers of text files are always done in "text" mode, then any combination of FTP programs should do the right thing, yielding an instance of the text file with line terminators that are correct for the local system. Trouble happens when someone transfers a file in "binary" mode between incompatible systems. (I'm not denying the possibility of a server brokenly serving badly-formed data--just saying that "use binary" is ill-advised in general, since that's the cause of incompatibility problems.) I don't have statistics on the distribution of the several record formats. However, most of the system files (except assembler and JCL source libraries) are undefined (or variable), not fixed. I don't think the speculation about avoiding the 0-1 punch to protect against tearing the card is well founded. 0-1 is the slash character, which was used early, long before EBCDIC was defined. In the EBCDIC era, punches in adjacent rows were common--for example, object decks can be represented directly on cards; they are heavy with 0x00 characters, and 0x00 was punched 12-0-9-8-1. The heading "Standard EBCDIC Files" and the following paragraph repeat the misleading "normally contain fixed-length records" assertion--what standard do you refer to? The "must be split" paragraph is odd. One would ordinarily define a record length suitable to contain all the data fields of a record. One would not define a too-small record length that would then require splitting a logical record's fields across several physical records. The paragraphs under heading "Record Length Information" use "EBCDIC file" to mean "fixed-length-record file", which is consistent with the preceding paragraphs, but now even more misleading, since the non-fixed-length formats *do* have block structure / record length information embedded in the file. As long as "it is the programmer's responsibility" is there to imply complication, you might go on to say that, if the programmer says nothing in his program and in his JCL about record characteristics, the system will merge in that information from the DSCB (file label), and the access methods will handle blocking transparently to the user's program. So straightforward cases are simple. The programmer would have to go out of his way to create the "read 60 character records" error. Sorry to run on so...it was the non-comma in the ASCII chart that got me started. :) -- ib1 at teksavvy dot com RSA/2048/476766B1 (E5A329D8 DC15385D 79B174E2 9BAB4638)