========================== Unix Shell I/O Redirection ========================== -IAN! idallen@ncf.ca ------------------ Output Redirection ------------------ Output redirection diverts (redirects) output that would normally appear on the screen to some other place, either into the input of another command (a pipe) or into a file. This normal output on your screen is called the "standard output" ("stdout") of the command. Output redirection of stdout into files: $ echo hello - stdout goes to terminal $ echo hello >file - erase file; send stdout to file $ echo hello >>file - append stdout to end of file Shells don't care where on or in the command line you do the redirection. All these command lines do exactly the same thing to stdout: $ echo hi there mom >file $ echo hi there >file mom $ echo hi >file there mom $ echo >file hi there mom $ >file echo hi there mom Shells handle redirection and wildcarding (called "globbing" on Unix), before they go looking for the command name to run. The command actually being run doesn't see any part of the redirection syntax. The redirection is done by the shell, then the redirection information is removed from the command line before the command is called. Redirection is never counted as arguments to a command. Examples: $ echo hello there - shell calls "echo" with two arguments ==> echo(hello,there) - "echo" echoes two arguments - output appears in default location (standard output is your screen) $ echo hello there >file - shell creates "file" and diverts standard output into it - shell calls "echo" with two arguments ==> echo(hello,there) (note NO CHANGE in arguments to "echo" from the previous example) - "echo" echoes two arguments - standard output is captured in output "file", NOT on your screen $ >file echo hello there - this is identical to the above example - standard output is captured in output "file", NOT on your screen - you can put the redirection anywhere in the command line! Unix Big Redirection Mistake #1: Do not do this kind of redirection (cat is used as the example program here - anything that reads files and produces output is at risk): $ cat * >z - shell creates "z" and redirects all future standard output into it - shell expands wildcards; wildcard "*" includes file "z" that was just created by the shell (Note: Bourne shells will do the wildcard before the file creation; C Shells do the file creation first.) - shell finds and calls cat command with all file names as arguments ==> e.g. cat(a,b,c,d,e,file1,file2,...etc...,z) - cat command processes each argument, opening each file and sending the output into file "z" - when cat opens file "z", it ends up reading from the top of file "z" and writing to the bottom of file "z" at the same time! - Result: an infinite loop that fills up the disk drive as "z" gets bigger and bigger Fix #1: Use a hidden file name $ cat * >.z - uses a hidden file name not matched by the shell "*" wildcard - the cat command is not given ".z" as an argument, so no loop occurs Fix #2 (two ways): Use a file in some other directory $ cat * >../z $ cat * >/tmp/z - redirect output into a file that is not in the current directory so that it is not read by the cat command and no loop occurs Unix Big Redirection Mistake #2 Do not do this kind of redirection (cat is used as the example program here - anything that reads files and produces output is at risk): $ cat a b >a - shell truncates file "a" and redirects command output into it - original contents of "a" are lost - truncated - GONE! - shell finds and calls cat command with two file name arguments ==> i.e. cat(a,b) - cat command processes contents of file "a" (now an empty file) - cat command processes contents of file "b" - output has been redirected by the shell to appear in file "a" - Result: file "a" gets a copy of "b"; original contents of "a" are lost Fix #1: Append to a $ cat b >>a - double-redirect syntax appends file "b" safely to the end of "a" Fix #2: Use a Temporary Third File $ cat a b >c # mv c a - the third file safely receives the output of "a" and "b" ------------------ Input Redirection: ------------------ Most Unix commands read input from files, if file names are given on the command line, and from standard input ("stdin") if no file names are given. (Not *all* commands read from standard input. Examples of common commands that never read from standard input: cp, mv, date, who, etc.) If a command reads from standard input, you can tell the shell to use input redirection to change from where the command reads: $ cat food - reads from file "food" $ cat - reads from stdin (keyboard) $ cat a >b >c >d >e - the "date" output goes into file "e"; the other files are created by the shell but are empty because only the final redirection wins bash$ date >out | wc 0 0 0 - the "date" output goes into file "out"; nothing goes into the pipe Some shells (including the "C" shells) will try to warn you about silly shell redirection mistakes: csh% date a >b >c Ambiguous output redirect. csh% date >a | wc Ambiguous output redirect. The C shells tell you that you can't redirect stdin or stdout to/from more than one place at the same time. Bourne shells do not tell you - they simply ignore the "extra" redirections and do only one of each. -------------------- Redirection Examples -------------------- A command line to convert lower-case to upper-case from the "who" command: $ who | tr 'a-z' 'A-Z' Shell question: Are the single quotes required around the two arguments? (Are there any special characters in the arguments that need protection?) Using redirection, you can use a similar command to convert a lower-case file of text into upper-case. EXPERIMENT: Why doesn't this convert the file "myfile" to upper-case? $ tr 'a-z' 'A-Z' myfile Why is the file "myfile" empty after this command is run? What about the following command lines - what is in "myfile" when the command finishes? $ cat myfile $ sort myfile $ head myfile Given the above, why is "myfile" not empty in the following case? $ wc myfile The following command line doesn't work because the programmer doesn't understand the "tr" command syntax: $ tr 'a-z' 'A-Z' myfile >new Why does this generate an error message from "tr"? (The "tr" command is unusual in its handling of command line pathnames. RTFM) The following command line redirection is faulty; however, it sometimes works for small files: $ cat foo bar | tr 'a' 'b' | grep "lala" | sort | head >foo There is a critical race between the first "cat" command trying to read the data out of "foo" before the shell truncates it to zero when launching the "head" command at the end of the pipeline. Depending on the system load and the size of the file, "cat" may or may not get out all the data before the "foo" file is truncated or altered by the shell in the redirection at the end of the pipeline. Don't depend on long pipelines saving you from bad redirection! Never redirect output into a file that is being used as input in the same command or anywhere in the command pipeline.