Character Encoding / Line Ends

Computers use binary bit patterns to represent, not only numbers, but also characters. A text file contains binary bit patterns that map to printable characters according to some mapping table. While the binary bit pattern all-zeroes, 00000000, usually represents an integer zero, a printable zero character, i.e. '0', usually is not encoded in a text file using the zero bit pattern. (For example, ASCII uses the 7-bit pattern 0x30 to encode a printable zero digit. EBCDIC uses the 8-bit pattern 0xF0. These are not the same bit patterns used to represent the integer value zero.)

Originally, here in North America, the bit patterns being used for characters only handled English. English only needed a 7-bit or 8-bit character size to include all the letters, digits, and common punctuation, so these character bit patterns all fit nicely into one computer "byte".

The mapping of bit pattern to printable character has gotten complex over the past decades due to the introduction of more and more different mappings to include more and more of the world's languages, current and past. (Not all the world speaks English!) For many years, the field was reluctant to break the rule "one-character, one-byte", so many mutually incompatible 8-bit character mappings were developed to handle different languages in different parts of the world. The same 8-bit pattern might map to one printable character in Norway, and a different character in France or Greece. Creating a file containing both French and Greek characters was impossible.

In 1991, a Universal Character set Unicode was introduced, using a 16-bit (two-byte) character format, with later updates to permit extensions to 32-bit characters as needed. This 16-bit character set broke the "one-character, one-byte" rule. It was incompatible with all one-byte character systems used to date, and thus rendered much text manipulation software (sorting, indexing, etc.) useless.

In 1993, some Americans introduced an UTF-8, an 8-bit version of Unicode that was backwards-compatible with ASCII, and thus with English. If you didn't use any non-English characters in your file, the file format was plain 7-bit ASCII. Only when you needed a foreign Unicode character did you have to resort to some 8-bit encoding sequences. Most software that expected ASCII could handle UTF-8 equally well. UTF-8 has become very popular in North America, since it treats ASCII as ASCII with no complications.

Character encodings:

Character Encoding : ASCII

Students should know from memory the basic layout of the letters and digits in the 7-bit ASCII character encoding table. What region of the table contains unprintable control characters? What is the ASCII value of a space? the letters "a" and "A"? What is the lowest standard-ASCII (7-bit) character's name and bit pattern? What is the highest standard-ASCII (7-bit) character's name and bit pattern?

The American Standard Code for Information Interchange (ASCII) coding scheme was developed as a 7-bit code. A 7-bit code provides enough different bit patterns (128) to permit a coding scheme for all the upper- and lower-case characters found on a standard English language keyboard, plus punctuation and some unprintable device control characters (e.g. Newline, Carriage Return, Bell, etc.).

Seven-bit ASCII encoding is normally used in 8-bit bytes with the top (leftmost) bit set to zero. Some extended encodings based on ASCII use the top bit set to include an additional 128 characters, e.g. ISO-8859-1 (Latin-1) is an 8-bit standard that includes the accented letters needed for Western European languages (including French).

Before the development of standards for extended-ASCII encodings, each manufacturer of computer equipment used different incompatible choices for what the extended characters represented. Files written on one machine don't display properly on another.

Minimal Sizes for Codes Representing Characters

The Major ASCII Codes and Rules

The Full ASCII Table

Most ASCII encoding/decoding can be performed without tables by knowing a few base codes: the blank, the letter "A", the digit "0", the carriage-return, and the line-feed. The rest of the letters and digits can be figured out from these base codes.



0 1 2 3 4 5 6 7
0 NUL DLE SPACE 0 @ P ` p
1 SOH DC1 ! 1 A Q a q
2 STX DC2 " 2 B R b r
3 ETX DC3 # 3 C S c s
4 EOT DC4 $ 4 D T d t
5 ENQ NAK % 5 E U e u
6 ACK SYN & 6 F V f v
7 BEL ETB ' 7 G W g w
8 BS CAN ( 8 H X h x
9 HT EM ) 9 I Y i y
A LF SUB * : J Z j z
B VT ESC + ; K [ k {
C FF FS , < L \ l |
D CR GS - = M ] m }
E SO RS . > N ^ n ~
F SI US / ? O _ o DEL


Summary of line terminators

ASCII-encoded files are usually composed of variable length lines of characters. Each line is terminated with one or more unprintable characters. The exact character or characters used at the end of each line depends on what computer system you are using.

a single carriage-return character:
Used to end lines on Apple Macintosh systems.
a single line-feed character:
Used to end lines on Unix/Linux systems.
a carriage-return followed by a line-feed:
Used to end lines on Microsoft MS-DOS and Windows systems.

When using a file-transfer program to move text (not binary) files between machines, you must know the consequences of incompatible line-end terminators.

Example of ASCII File Decoding

Basic Character Encoding : EBCDIC

EBCDIC material does not need to be memorized

You may need to decode/encode EBCDIC in an assignment

Character encoded data on IBM mainframe computers is normally based on a scheme called EBCDIC. The EBCDIC character encoding preceded the ASCII encoding. EBCDIC was developed from a basis that involved the computer punched card and has features that, to be properly understood, require a knowledge of that historical medium.

EBCDIC encoded files normally contain fixed-length records.

The Punched Card and Hollerith Codes

EBCDIC Codes (Basic Codes)

Standard EBCDIC Files

EBCDIC vs. ASCII Character Sequences

For a side-by-side comparison, see:

Q: If you examine an EBCDIC text file copied byte-for-byte onto an ASCII system such as Unix/Linux or DOS/Windows/Macintosh, what will you see on your ASCII screen? (Hints: [1] Do the EBCDIC letters and numbers match any printable 7-bit ASCII characters? [2] Do EBCDIC sentence punctuation and space characters match any printable ASCII characters?)

Comments on this Document

From: ib1 at teksavvy dot com
Subject: 120_CharacterEncoding
Date: 2010-10-16T15:21:27Z

I happened (goofing off, y'know) to wander into and
   I saw some errors and oddities on that page.
In the "Character encodings:" overview table, EBCDIC is described as
   "English only".  EBCDIC was and is routinely used with
   more-than-ASCII character sets, using the extra code points for
   non-English letters, APL, line-drawing, etc.  (IBM majorly messed up
   by not imposing cross-company standards for assigning characters
   beyond the basic (ASCII-equivalent) code points, so customers had
   trouble when porting documents between various World Trade divisions,
   and in/out of Domestic.)
In the paragraph starting "Before the development of standards", you say
   that "each manufacturer of computer equipment used different
   incompatible choices".  The situation was actually worse than you
   say, since there were situations where one manufacturer had multiple
   conventions for mapping characters to code points, even within one
The discussion of text file transfer via FTP conflicts with my
   observations and perception.  If transfers of text files are always
   done in "text" mode, then any combination of FTP programs should do
   the right thing, yielding an instance of the text file with line
   terminators that are correct for the local system.  Trouble happens
   when someone transfers a file in "binary" mode between incompatible
   systems.  (I'm not denying the possibility of a server brokenly
   serving badly-formed data--just saying that "use binary" is
   ill-advised in general, since that's the cause of incompatibility
I don't have statistics on the distribution of the several record
   formats.  However, most of the system files (except assembler and JCL
   source libraries) are undefined (or variable), not fixed.
I don't think the speculation about avoiding the 0-1 punch to protect
   against tearing the card is well founded.  0-1 is the slash
   character, which was used early, long before EBCDIC was defined.  In
   the EBCDIC era, punches in adjacent rows were common--for example,
   object decks can be represented directly on cards; they are heavy
   with 0x00 characters, and 0x00 was punched 12-0-9-8-1.
The heading "Standard EBCDIC Files" and the following paragraph repeat
   the misleading "normally contain fixed-length records"
   assertion--what standard do you refer to?
The "must be split" paragraph is odd.  One would ordinarily define a
   record length suitable to contain all the data fields of a record.
   One would not define a too-small record length that would then
   require splitting a logical record's fields across several physical
The paragraphs under heading "Record Length Information" use "EBCDIC
   file" to mean "fixed-length-record file", which is consistent with
   the preceding paragraphs, but now even more misleading, since the
   non-fixed-length formats *do* have block structure / record length
   information embedded in the file.
As long as "it is the programmer's responsibility" is there to imply
   complication, you might go on to say that, if the programmer says
   nothing in his program and in his JCL about record characteristics,
   the system will merge in that information from the DSCB (file label),
   and the access methods will handle blocking transparently to the
   user's program.  So straightforward cases are simple.  The programmer
   would have to go out of his way to create the "read 60 character
   records" error.
Sorry to run on was the non-comma in the ASCII chart that got me
   started.  :)

ib1 at teksavvy dot com  RSA/2048/476766B1 (E5A329D8 DC15385D 79B174E2 9BAB4638)