acol
utility script; decoding a puzzle fileUpdated: 2017-01-20 00:52 EST
Do not print this assignment on paper!
- On paper, you will miss updates, corrections, and hints added to the online version.
- On paper, you cannot follow any of the hyperlink URLs that lead you to hints and course notes relevant to answering a question.
- On paper, scrolling text boxes will be cut off and not print properly.
23:59 (11:59pm) Friday December 9, 2016 (end of Week 14)
WARNING: Some inattentive students upload Assignment #11 into the Assignment #10 upload area. Don’t make that mistake! Be exact.
This is a BONUS assignment for extra credit. It is optional.
This assignment is based on your weekly Class Notes and covers these topics:
tar
files.For full marks, follow these directions exactly:
These tasks must be done in your account via Remote Login to the Course Linux Server.
Do the tasks in order, from top to bottom. Do not skip steps. Most tasks are independent, but some depend on successful completion of a previous task.
READ ALL THE WORDS in each task before you begin the task, especially all the Hints and links.
Verify your own work before running the Checking Program. You won’t have a Checking Program at your job interview and the Checking Program is not guaranteed to check everything.
Run the Checking Program at the end of the task to grade your work and help you find some of your errors. A perfect mark from the Checking Program does not mean your answers are correct.
When you are done with this Assignment, submit the output of the Checking Program to Blackboard before the due date, following the directions given at the end of this Assignment.
This is a BONUS assignment for extra credit. It is optional.
You can use the Checking Program to check your work after you have completed each task.
Most task sections below require you to finish the whole task section before running the Checking Program. You may not always be able to run the Checking Program successfully in the middle of a task or after every single task sub-step. The assignment tells you where you can safely check your work.
You will create file system structure in your CLS home directory containing various directories and files. When you are finished the tasks, leave the files and directories in place on the CLS as part of your deliverables for your instructor to verify.
Assignments may be re-marked at any time on the CLS; you must have your term work available on the CLS right until term end. Do not delete any assignment work until after the term is over!
You can modify your work and check it with the Checking Program as often as you like before you submit your final mark to Blackboard. You can upload your marks to Blackboard as many times as you like before the due date. Partial marks are accepted.
Your instructor will also mark on the due date the work you do in your account on the CLS. Leave all your work on the CLS and do not modify it after you have submitted your final mark to Blackboard.
All course notes are available on the Internet and also on the CLS. You can learn about how to read and search these CLS files using the command line on the CLS under the heading Copies of the CST8207 course notes near the bottom of the page Course Linux Server.
All references to the Source Directory below are to the CLS directory ~idallen/cst8207/16f/assignment12/
and that name starts with a tilde character ~
followed by a user name with no intervening slash. The leading tilde indicates to the shell that the pathname starts with the HOME directory of the account idallen
(seven letters).
You do not have permission to list the names of all the files in the Source Directory, but you can access any files whose names you already know.
Most of the tasks below ask you to write a small executable shell script, based on the lecture notes and slides. None of the scripts need complex Boolean expressions (“||
” or “&&
” or -a
or -o
); they are all simple scripts with simple conditional logic.
Each script below must begin with the Standard Script Header you used for your previous script assignments. See the class notes.
Though the header is executable code, in the descriptions below we don’t count those lines, or any comment or blank lines, in the size of the script. We only count the new lines of code that you write.
For example, a “one-line script” is really several lines of header, a blank line, a block of several comment lines that Document Your Script, another blank line, and then your one line of actual script code. The description below calls this a one line script, even though it may contain a dozen lines.
Make sure that your script file is executable, so that it can be executed as ./scriptname.sh
from the shell command line.
Build up each script by adding a few lines and testing what you have added; don’t write the whole thing and try to debug it!
Run the given example tests on your scripts to make sure they work. Sample output for each of the scripts is given, so that you may check your work as you proceed.
Make sure your script handles all of the sample inputs given, especially the inputs containing shell metacharacters. (System crackers often attack your system using special characters as input.)
The examples below do not fully test your script; you will need to try other examples to make sure your scripts work properly for all possible inputs, especially inputs with blanks and shell meta-characters.
Remember to double quote all variable expansions to prevent GLOB and blank expansion that can cause syntax errors and other unwanted problems in your script.
Error messages must follow the Good Error Message format given below.
If you are having problems with your script and are getting error messages from the shell, review possible Script Problems.
For full marks, follow these task directions below exactly as written. READ ALL THE WORDS to work effectively and not waste your time.
Your instructor will also mark on the due date the work you do in your account on the CLS. Leave all your work on the CLS and do not modify it. Do not delete any assignment work from the CLS until after the course is over.
You must keep a list of command names used each week and write down what each command does, as described in the [List of Commands You Should Know]. Without that list to remind you what command names to use, you will find assignments very difficult.
Do a Remote Login to the Course Linux Server (CLS) from any existing computer, using the host name appropriate for whether you are on-campus or off-campus. All work in this assignment must be done on the CLS.
Base Directory: Make the CLS directory named ~/CST8207-16F/Assignments/assignment12
, in which you will create the files and scripts resulting from the following tasks. (You do not have to create any directories that you have already created in a previous assignment.) Spelling and capitalization must be exactly as shown:
check
check
symbolic link needed to run the Checking Program, as described in the section Checking Program below.This assignment12
directory is called the Base Directory for most pathnames in this assignment. Store your files and answers in this Base Directory, not in your HOME directory or anywhere else.
Use the symbolic link to run the Checking Program to verify your work so far.
Normally the Checking Program checks all the scripts. This can be slow if you are only interested in the check output for one script that you are working on. You can check just one or more individual scripts by giving the script names as arguments to the Checking Program:
$ ./check acol1.sh # only check this script
$ ./check acol1.sh acolNF.sh # only check these two scripts
Do not submit for marking the output of checking only a few scripts!
awk
wrapper script: extracting a column of inputIndexRecall in lecture that we used used the awk
program to extract the first space-delimited column of an input stream. We will develop a script named acol
(Awk COLumn) that extracts any column of input.
Review Properties of all Scripts, above.
You need to understand Shell Scripts and Control Structures to do these scripts.
The first four scripts have no error checking code. Only the final acol.sh
script needs to have full error checking.
acol1.sh
IndexCreate a one-line script named acol1.sh
that uses awk
to read its standard input and extract the first column, exactly as used in the lecture notes.
Make sure all the examples below work before you run the checking program! Examples:
$ echo a b c | ./acol1.sh
a
$ date
Sun Dec 4 02:11:41 EST 2016
$ date | ./acol1.sh
Sun
$ last | ./acol1.sh
idallen
idallen
kelleyt
donnelr
[...etc...]
Add comments to Document Your Script.
Check your work so far using the checking program symlink.
acol2.sh
IndexCreate a one-line script named acol2.sh
that uses awk
to read its standard input and extract the second column of input.
Make sure all the examples below work before you run the checking program! Examples:
$ echo a b c | ./acol2.sh
b
$ date
Sun Dec 4 02:11:41 EST 2016
$ date | ./acol2.sh
Dec
$ last | ./acol2.sh
pts/9
pts/14
pts/50
pts/50
[...etc...]
Add comments to Document Your Script.
Check your work so far using the checking program symlink.
acolNF.sh
IndexCreate a one-line script named acolNF.sh
that uses awk
to read its standard input and extract the last (NF
) column of input.
Make sure all the examples below work before you run the checking program! Examples:
$ echo a b c | ./acolNF.sh
c
$ echo a b d e f g h i j | ./acolNF.sh
j
$ date
Sun Dec 4 02:11:41 EST 2016
$ date | ./acolNF.sh
2016
$ last | ./acolNF.sh
(00:31)
(02:11)
(02:02)
(00:01)
[...etc...]
Add comments to Document Your Script.
Check your work so far using the checking program symlink.
acolnew.sh
IndexIt should be clear that having a separate script for every possible number of columns is not a good thing. Let’s write one script that takes as its only argument the column number we want awk
to print.
Create a one-line script named acolnew.sh
that uses awk
to read its standard input and extract the column of input given as an argument on the command line.
This new one-line script is a very small modification of your previous one-line scripts; you only need to change about a half-dozen characters in the one line to make it work.
Make sure all the examples below work before you run the checking program! Examples:
$ echo a b c d e f | ./acolnew.sh 1
a
$ echo a b c d e f | ./acolnew.sh 2
b
$ echo a b c d e f | ./acolnew.sh NF
f
The script takes a single argument that is either a number (e.g. 1
, 2
, etc.) or the string NF
and substitutes that argument directly into the awk
command line inside the script as the column to print, instead of using the previous hard-coded column number 1
, 2
, or NF
that you used in the three previous scripts.
Hint: Instead of hard-coding the column number in the awk
command line, as you did in the above three previous scripts, use the first script argument variable instead of the hard-coded number. You will need to adjust the Quoting around the awk
command arguments to allow the shell argument variable to be expanded by the shell but still keep hidden from the shell the $
used by awk
to select the column number. You cannot use the awk
-v
option here.
The new one-line script is a simple modification of your previous one-line script: it is still a one-line script; it does not yet do any input checking or argument processing. If you don’t supply an argument, or you pass something that awk
doesn’t understand, you will get an error from the shell or from awk
– this is expected:
# examples of errors when script or awk gets bad input:
$ echo a b c d e f | ./acolnew.sh
./acolnew.sh: 8: ./acolnew.sh: 1: parameter not set
$ echo a b c d e f | ./acolnew.sh ''
awk: { print $ }
awk: ^ syntax error
$ echo a b c d e f | ./acolnew.sh ' '
awk: { print $ }
awk: ^ syntax error
$ echo a b c d e f | ./acolnew.sh @
awk: { print $@ }
awk: ^ invalid char '@' in expression
If you supply more than one argument, the script won’t detect the error and will simply ignore the extra arguments – this is okay for this prototype script:
$ echo a b c | ./acolnew.sh NF these arguments after NF are ignored
c
This one-line script does not validate its input, so it gives cryptic errors if the argument is incorrect and it does not warn you if you give too many arguments. This is not a production-quality script. We will fix it in the next step.
Add comments to Document Your Script.
Check your work so far using the checking program symlink.
acol.sh
IndexCopy acolnew.sh
to acol.sh
and add full input validation to the new script.
The new script must check to make sure it has exactly one input argument and do validation of that one argument before using it with awk
to print the given column number. The only valid input allowed must be an integer from 1 to 231-1, or the string NF
.
You do not need to change the awk
line in your script at all. You only need to add input validation to the script, before calling awk
. The input validation will cause the script to exit if the input is bad, so that awk
doesn’t process bad input.
Your script should enforce the “only one argument” requirement. Print both a Good Error Message and Usage Message on stderr and exit with a bad error status if the number of arguments is not exactly one.
Also print an error and Usage message and exit if the one argument is equal to the empty string (''
), or if it is not all-digits or NF
.
Also print an error and Usage message and exit if a numeric argument contains more than 15 digits. (The built-in test
command can’t test very large integers!) Hint: You can do this check with a GLOB pattern.
Also print an error and Usage message and exit if a numeric argument is zero or greater than or equal to 231. (The awk
program can’t handle huge column numbers: awk: fatal: attempt to access field -2147483648
)
Make sure all the examples below work before you run the checking program! Examples:
$ echo a b c | ./acol.sh
...error message about wrong number of arguments prints here...
...usage message prints here...
$ echo $?
1
$ echo a b c | ./acol.sh 1 2 3 a b c >out
...error message about wrong number of arguments prints here...
...usage message prints here...
$ echo $?
1
$ echo a b c | ./acol.sh ''
...error message about empty argument prints here...
...usage message prints here...
$ echo $?
1
$ echo a b c | ./acol.sh ' '
...error message about non-digit, non-NF argument prints here...
...usage message prints here...
$ echo $?
1
$ echo a b c | ./acol.sh '*'
...error message about non-digit, non-NF argument prints here...
...usage message prints here...
$ echo $?
1
$ echo a b c | ./acol.sh 0000000000123456 # 16 digits
...error message about field number being more than 15 digits...
...usage message prints here...
$ echo $?
1
$ echo a b c | ./acol.sh 2147483648 # 2**31
...error message about field number being >= 2**31...
...usage message prints here...
$ echo $?
1
$ echo a b c | ./acol.sh 0
...error message about field number being zero...
...usage message prints here...
$ echo $?
1
$ echo a b c | ./acol.sh 1
a
$ echo $?
0
$ echo a b c | ./acol.sh NF
c
$ echo $?
0
$ echo a b c | ./acol.sh 2147483647 # 2**31 - 1
$ echo $?
0
$ echo a b c | ./acol.sh 000000000000002 # 15 digits
b
$ echo $?
0
Add comments to Document Your Script.
Check your work so far using the checking program symlink.
Extra Work: The
awk
program interprets numbers with leading zeroes as octal (base 8), which produces unexpected results:$ seq -s ' ' 1 10 1 2 3 4 5 6 7 8 9 10 $ seq -s ' ' 1 10 | ./acol.sh 10 10 $ seq -s ' ' 1 10 | ./acol.sh 010 8
Find a way to remove leading zeroes before the number gets passed to
awk
.
acol
symbolic linkIndexYou need to understand Symbolic Links, Shell Variables, and Search Path to do this task.
Make sure you have a personal bin/
directory in your account. (You created this directory in a previous assignment.)
Make sure your personal bin/
directory is added to the end of your search path every time you log in. (You did this in a previous assignment.)
Use a relative symlink to link the acol.sh
script into your personal bin/
directory under the name acol
. Execute the acol
program using this name in the bin/
directory to make sure the relative symlink works.
Make sure you can type acol
into the shell and have your shell find and run your script from in your bin/
directory using your search path.
Hints: Please Read All The Words in the sentence that begins with the words “Use a” above, especially the word that begins with the letter “r”.
Check your work so far using the Checking Program symlink.
Use your acol
script whenever you need to see just one column of data. Take your acol
script with you to your next job!
$ fgrep 'refused connect' /var/log/auth.log | acol NF | sort | uniq -c | sort -nr | head -n 3
71064 (116.31.116.28)
17217 (116.31.116.24)
9271 (116.31.116.23)
$ last | acol 1 | sort | uniq -c | sort -nr | head -n 5
192 idallen
150 user1234
121 user2345
117 user3456
110 user4567
$ last | acol 3 | sort | uniq -c | sort -nr | head -n 5
131 cpef0f249a3b983-
122 147.194.28.242
117 23-91-139-86.cpe
94 cpebc4dfb967133-
89 otwaon1056w-lp1
puzzle.txt
IndexYou need to understand Tar and Gzip and Permissions to do this task.
puzzle.txt
After you have run the Checking Program at least once, you will find created for you in the Source Directory a puzzle file named with your userid puzzle/abcd0001
.mystery
(where abcd0001
is replaced by your own userid). This file contains many layers of compression and tar
archiving.
Extract the ASCII text file from deep inside this puzzle file and save the ASCII text as file puzzle.txt
in your Base Directory.
Hints: Repeatedly unpack the compression and tar
archiving until you find an ASCII text file. I demonstrated this example in class after the lecture showing you the tar
command – see your class notes. The file
command will be helpful to decode the nested layers of the puzzle.
Check your work so far using the checking program symlink.
That is all the tasks you need to do.
Read your CLS Linux EMail and remove any messages that may be waiting. See Reading EMail for help.
Check your work a final time using the Checking Program below and save the standard output of that program into a file as described below. Submit that file (and only that one file) to Blackboard following the directions below.
When you are done, log out of the CLS before you close your laptop or close the PuTTY window, by using the shell exit
command:
$ exit
You must document your script with comment lines before you submit it. Script comment lines start with the comment or hashtag character #
and extend to the end of the line. You can (and must) use more than one comment line in your script.
Add at least five (or more) comment lines to each script containing the following five types of information, in the following order:
4.2.1 acol1.sh
-s
and a second argument of the script name, e.g. ./check -s acol1.sh
The Signing Key comment line must start with # KEY:
and will be about 70 characters long.Obey these rules for your script comments:
1>&2
to write messages to standard error instead of standard output.#
rather than making one huge long comment line.Here is a sample comment block for a hypothetical assignment number 99:
# Assignment 99 This is a Sample Comment Block
# 1.2 foo.sh
# Ian Allen 123456789 abcd0001@algonquinlive.com
# KEY: ==wMwATMskemnackve2NxIDO0N3Ygg2cuMHduVWb1dmchByN4YTOxcTO1QTM
# This is a script that demonstrates how to frob the widjet.
# If there are no widjets to frob, the script prints an
# error message end exits with status 2. Otherwise exit zero.
Make sure you do the correct placement of the comment block in the script file, as described above!
Good shell script error messages must obey these four rules:
1>&2
to send to stderr any output normally destined for stdout. See the examples below.
1>&2
is used on echo
statements, to send the text to standard error instead of standard output.$0
).
$0
.expecting one file name
).
found 3 (a b c)
$#
and their values $*
.Never say just missing argument
or illegal input
or invalid input
or too many
. Always specify exactly what is needed and how many is “too many” or “too few”. Here are examples:
echo 1>&2 "$0: Expecting 3 file names; found $# ($*)"
echo 1>&2 "$0: Student age '$student_age' is not between $min_age and $max_age"
echo 1>&2 "$0: Modify days '$moddays' must be greater than zero"
echo 1>&2 "$0: File '$file' does not exist; expecting accounting file"
Put quotes around anything entered by a user, otherwise your error messages may be confusing. Compare these example messages without and with quotes around the user input file name:
$ ./total.sh still
./total.sh: File still does not exist; expecting accounting file
Usage: ./total.sh account_file
$ ./total.sh still
./total.sh: File 'still' does not exist; expecting accounting file
Usage: ./total.sh account_file
After detecting an error, the usual thing to do is print a Good Error Message explaining the error, followed by a Usage message telling how to use the script, then exit the script with a non-zero return code. Don’t keep processing bad data!
The Usage message gives the syntax for correctly using the script, using man
page syntax to indicate optional and repeated arguments, e.g.:
Usage: ./script.sh [ first_line [ last_line ] ]
Usage: ./script.sh filename...
The name of the script is output using the $0
variable. Do not hard-code the name of a script inside the script.
Summary: Do some tasks, then run the Checking Program to verify your work as you go. You can run the Checking Program as often as you want. When you have the best mark, upload the single file that is the output of the Checking Program to Blackboard.
Since I also do manual marking of student assignments, your final mark may not be the same as the mark submitted using the current version of the Checking Program. I do not guarantee that any version of the Checking Program will find all the errors in your work. Complete your assignments according to the specifications, not according to the incomplete set of the mistakes detected by the Checking Program.
check
There is a Checking Program named assignment12check
in the Source Directory on the CLS. You can execute this program by typing its (long) pathname into the shell as a command name and paginating the (often long) output using less
:
$ ~idallen/cst8207/16f/assignment12/assignment12check | less
Create a symbolic link named check
in your Base Directory that links to the Checking Program in the Source Directory, as you did in a previous assignment. Use the symlink to check your work:
$ ./check | less
Checking only one of your scripts
Normally the Checking Program checks all the scripts. This can be slow if you are only interested in the check output for one script that you are working on. You can check just one or more individual scripts by giving the script names as arguments to the Checking Program:
$ ./check acol1.sh # only check this script $ ./check acol1.sh acolNF.sh # only check these two scripts
Do not submit for marking the output of checking only a few scripts!
When you are done, execute the above Checking Program as a command line on the CLS. This program will check your work, assign you a mark, and display the output on your screen.
You may run the Checking Program as many times as you wish, allowing you to correct mistakes and get the best mark. Some task sections require you to finish the whole section before running the Checking Program at the end; you may not always be able to run the Checking Program successfully after every single task step.
When you are done with this assignment, and you like the mark displayed on your screen by the Checking Program, you must redirect only the standard output of the Checking Program into the text file assignment12.txt
in your Base Directory on the CLS, like this:
$ ./check >assignment12.txt
$ less assignment12.txt
assignment12.txt
file name.YOUR MARK for
assignment12.txt
(containing the output from the Checking Program) from the CLS to your local computer.
YOUR MARK for
assignment12.txt
file from your local computer to the correct Assignment area on Blackboard (with the exact name) before the due date:
assignment12.txt
file from your local computer. Make sure the assignment file has the correct name on your local computer before you attach it. Attach only your assignment12.txt
file for upload. Do not attach any other file names.assignment12.txt
file on the Upload Assignment page, scroll down to the bottom of the page and use the Submit button to actually upload your attached assignment12.txt
file to Blackboard.Use only Attach File, Browse My Computer on the Upload Assignment page. Do not enter any text into the Write Submission or Add Comments boxes on Blackboard; I do not read them. Use only the Attach File, Browse My Computer section followed by the Submit button. If you need to comment on any assignment submission, send me EMail.
You can revise and upload the file more than once using the Start New button on the Review Submission History page to open a new Upload Assignment page. I only look at the most recent submission.
You must upload the file with the correct name from your local computer; you cannot correct the name as you upload it to Blackboard.
You will also see the Review Submission History page any time you already have an assignment attempt uploaded and you click on the underlined assignment12 link. You can use the Start New button on this page to re-upload your assignment as many times as you like.
You cannot delete an assignment attempt, but you can always upload a new version. I only mark the latest version.
Your instructor may also mark files in your directory in your CLS account after the due date. Leave everything there on the CLS. Do not delete any assignment work from the CLS until after the term is over!
I do not accept any assignment submissions by EMail. Use only the Blackboard Attach File, Browse My Computer. No word processor documents. Plain Text only.
Use the exact file name given above. Upload only one single file of Linux-format plain text, not HTML, not RTF, not MSWord. No fonts, no word-processing. Linux plain text only.
NO EMAIL, WORD PROCESSOR, PDF, RTF, or HTML DOCUMENTS ACCEPTED.
No marks are awarded for submitting under the wrong assignment number or for using the wrong file name. Use the exact 16-character, lower-case name given above.
WARNING: Some inattentive students don’t read all these words. Don’t make that mistake! Be exact.
READ ALL THE WORDS. OH PLEASE, PLEASE, PLEASE READ ALL THE WORDS!