1.1 GLOB patterns (review)Index

There are several major places where GLOB patterns are used:

$ ls *.txt
$ echo ?????.txt
$ touch [ab]*.txt

case "$1" in
/* ) type='Absolute Pathname' ;;
*  ) type='Relative Pathname' ;;
esac

$ find . -name '*.txt'
$ find . -name '?????.txt'
$ find . -name '[ab]*.txt'

1.2 Regular Expressions – Basic and ExtendedIndex

Regular Expressions (short form: regexp) are text matching patterns similar to GLOB patterns but more powerful. Regexp patterns use all the GLOB pattern matching characters and add more. The characters work slightly differently between GLOB and regexp.

Regexp are used by many Unix/Linux programs and programming languages such as grep, sed, awk, vim, less, more, man, Perl, python, etc.

In an editor (such as vim or sed), a Regular Expression may be used to select characters to be deleted, replaced, or exchanged:

:%s/colou*r/COLOUR/g                   # vim replacement regular expression

$ echo "Colouur bad.  Colour red.  Color tan." | sed -e 's/Colou*r/COLOUR/g'
COLOUR bad.  COLOUR red.  COLOUR tan.

Regexp have a Basic set of pattern matching characters and an Extended set of characters. The grep program family is a very popular user of both Basic and Extended Regular Expressions.

The grep command itself accepts Basic Regular Expression syntax, and needs backslashes in front of some operators to access Extended Regular Expression features. The egrep command accepts Extended Regular Expression syntax and does not need the backslashes. You can do the same text search using either command, but the syntax changes:

$ grep 'publickey for \(idallen\|cst8207[abc]\?\)' /var/log/auth.log   # Basic
$ egrep 'publickey for (idallen|cst8207[abc]?)' /var/log/auth.log      # Extended

From the section REGULAR EXPRESSIONS in the man page for the grep command:

Basic vs Extended Regular Expressions
  In basic regular expressions the meta-characters ?, +, {, |,
  (, and ) lose their special meaning; instead use the backslashed
  versions \?, \+, \{, \|, \(, and \).

Even the bash shell has extended syntax that allows the use of regular expressions instead of simple GLOB patterns.

IMPORTANT: Regular Expressions use some of the same special characters as GLOB patterns, but they mean different things! In particular, *, ?, and . work differently! There are others!

1.3 GLOB patterns are anchored; Regular Expressions floatIndex

GLOB patterns are said to be anchored to the start and end of the line; they must always match the entire text string (usually a file name) from the start to the end.

The GLOB pattern a*b matches only text that starts with a and ends with b – that GLOB pattern doesn’t match just the ab in the middle of xxxabxxx.

The modified GLOB pattern *a*b* now matches the whole text that contains a followed by b anywhere in the text. The modified GLOB pattern does match the entire text xxxabxxx.

Regular Expressions are not by default anchored. They “float” down the text and they may match anywhere in the text string unless you explicitly anchor them to either the start or end of the text using using regexp characters ^ and/or $.

The Regular Expression a.*b matches inside any text that contains a followed by b anywhere in the text. The floating regexp does match the ab in the middle of xxxabxxx.

The modified Regular Expression ^a.*b$ is now anchored to the start and end of the text. The modified expression now matches exactly the same text as the GLOB pattern a*b because it forces the a to match at the start and the b to match at the end. It does not match inside xxxabxxx.

You must remember to anchor the ends of your Regular Expressions if you want to be sure that they match the whole piece of text and not just some part of the text.

Summary:

• Unanchored regexp a*b matches (only) the text ab inside xxxabxxx.
• Anchored regexp ^a*b$ does not match the ab inside xxxabxxx because the a has to be at the start and the b has to be at the end. It does match the string aaaaab.

2.1 Basic Regular Expressions using * repetition (zero or more) and parenthesesIndex

Similar to an algebraic exponent, the asterisk/star * Regular Expression operator binds tightly to the immediately preceding Regular Expression and repeats it zero or more times. Parentheses (a feature of Extended Regular Expressions) can be used for grouping, e.g.

$ grep 'suc*eed' document.txt        # find sueed, suceed, succeed, succceed, etc.
$ grep 'Bar\(bar\)*a' document.txt   # find Bara, Barbara, Barbarbara, etc.
$ egrep 'Bar(bar)*a' document.txt    # use egrep Extended regexp syntax

Rhabarbara: https://www.youtube.com/watch?v=dD2mhVc6C_8

Parentheses need backslashes in front of them when using a program such as grep that uses Basic Regular Expression syntax. The egrep program accepts Extended Regular Expression syntax and does not need the backslashes.

2.2 Concatenating and repeating Regular Expressions using * and \(...\)Index

As with algebraic multiplication, there is no operator to concatenate Regular Expressions to match longer strings. Simple write one expression and follow it with the next one.

Similar to an algebraic exponent, the asterisk/star * Regular Expression operator binds tightly to the immediately preceding Regular Expression and repeats it zero or more times. Parentheses can be used for grouping, e.g.

2.3 Special Characters in Basic Regular ExpressionsIndex

Regular Expressions have more special characters than GLOB patterns. Some special characters need backslashes in front of them to enable them in Basic Regular Expressions.

2.4 Regular Expressions match anywhere in a line: anchoring with ^ and $Index

GLOB Patterns are said to be anchored to the start and end of the string being matched. The GLOB pattern a*b matches text axb but not abx or xab. The a has to be at the start, and the b has to be at the end.

To allow a GLOB pattern to be unanchored and match anywhere inside a string, you need to pad the GLOB with * on both sides:

$ echo a*b                  # anchored: matches axb not abx or xab
$ echo *a*b*                # now matches abx or xab or xabx or xaxbx

The GLOB pattern has to match the whole string, and may need * at each end to allow it do that.

Unlike GLOB Patterns, which are anchored, Regular Expressions are not anchored unless you make them so using the explicit anchor characters ^ and/or $. Unanchored Regular Expressions “float” down the string until a match is found, and they don’t have to extend to the end of the string.

Regular Expressions can match just a piece of text in the middle of a line; they don’t have to match the whole line.

The GLOB pattern a*b doesn’t match the string xabx because GLOB is anchored and has to match the whole string, but the Regular Expression a.*b does match inside the line, because it is unanchored at either end and floats down the string and matches the ab in the middle of string. The regexp starts unanchored (no ^ at the start) and thus “floats” down the string to do the match.

Use the line start ^ and line end $ meta-characters to anchor a Regular Expression to the start or end of a line. Here are some examples of how GLOB patterns and regexp compare:

GLOB        Regular Expression (may use anchors)
----        ------------------------------------
foo         ^foo$
bar[abc]    ^bar[abc]$
[!abc]      ^[^abc]$                 # note in complement GLOB uses ! vs. ^
foo?        ^foo.$
a*b         ^a.*b$
*foo*       foo                      # unanchored GLOB needs * at ends
*a*b*       a.*b                     # unanchored GLOB needs * at ends

Remember that an unanchored Regular Expression may match only part of a line, e.g. the text ab matches only the ab part of xxxabxxx, not the whole xxxabxxx. GLOB patterns must always match the entire line from start to end; they can’t match a substring inside a line the way regexp can.

4.1 Using ^ to complement a character class set: [^abc]Index

• The ^ used immediately inside the opening square bracket of a class complements the whole character class set: [^az3]. The resulting character class expression matches any single character that is not in the set.
• The complemented class [^az3] means “any single character that is not a, z, or 3”
• The ^ only works this way if it is the first character inside the square brackets, otherwise it has no special meaning.
• The classes [a^z3] or [az^3] or [az3^] all match one of a, z, 3, or ^
• Remember, a ^ used in a Regular Expression outside of square brackets has the special meaning “match at beginning of line”. Don’t confuse it with ^ used inside a character class.

• Note that GLOB patterns complement character sets using ! and not ^:

  GLOB        Regular Expression
  [!abc]      [^abc]

Don’t confuse GLOB with Regular Expressions.

4.2 Having closing ] as part of a character class setIndex

A ] character can be placed inside square brackets to be part of the character class set, but it has to be the first character in the set. []az3] means one of the four characters ], a, z, or 3 and [^]azh] means any single character that is not one of the four characters ], a, z, or 3.

Attempting to put a closing square bracket ] inside square brackets in any other position is a syntax error:

• [ab]d] is a failed attempt at [ab][d]
• [] is a failed attempt at []]

You can put an opening [ anywhere in a character class, e.g.

$ grep '[([{]` doc.txt     # search for lines with '(' or '[' or '{'

4.3 POSIX character classes – e.g. [:digit:]Index

POSIX Character Class expressions represent an entire range of characters, such as “all the digits” or “all the letters”. The classes have an awkward syntax: The POSIX class name is preceded by [: and followed by :], e.g. [:digit:]. These are the resulting class names:

• The exact content of each character class depends on the local language.
• Only for plain ASCII is it true that “letters” means English a-z and A-Z.
• Other languages have other “letters”, e.g. é, ç, etc.
• When we use the POSIX character classes, we are specifying the correct set of characters for the local language as per the POSIX description.

These POSIX class names only work inside an enclosing Regular Expression character class expression using (more) square brackets. What looks like double square brackets is really an enclosing square bracket character class expression containing a POSIX class name (which unfortunately also uses square brackets and colons as part of its name), e.g.

grep '[0123456789]'     # a digit (a list of all the digits)
grep '[[:digit:]]'      # a digit - the POSIX class name [:digit:] inside []
grep '[abcd[:digit:]]`  # a digit or letter a or b or c or d
grep '[ab[:digit:]cd]`  # same -- a digit or a or b or c or d
grep '[[:digit:]abcd]`  # same -- a digit or a or b or c or d

Of course you can use multiple POSIX class names inside the character class expression:

grep '[[:alpha:][:digit:]]`   # a letter or a digit
grep '[^[:alpha:][:digit:]]`  # *NOT* a letter or a digit

WARNING: You cannot interchange the [:alpha:] class and a list of all the upper- and lower-case letters; they are not always the same because the POSIX [:alpha:] class changes depending on the local language:

grep '[[:alpha:]]`   # a letter, using the POSIX class name [:alpha:]
grep '[a-zA-Z]'      # NOT THE SAME AS [:alpha:] - DO NOT USE !
grep '[abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ]' # NOT THE SAME

4.4 POSIX Regular Expression Examples, e.g. ^[[:digit:]]*$Index

These expressions could be given to grep:

• Any line containing nothing or only alphabetic characters from start to end:

  ^[[:alpha:]]*$

• Any line containing only alphabetic characters from start to end, but must have at least one such character (can’t be an empty line):

  ^[[:alpha:]][[:alpha:]]*$

• Any line that begins with a digit (followed by anything or nothing):

  ^[[:digit:]]

4.5 Character class ranges using [.-.]Index

• A dash - between two characters inside square brackets, e.g. [0-9], represents a range of characters between the two, unless the dash is first or last in the set of characters, e.g. [-09] or [09-]
• Both [-09] and [09-] mean one of the three characters 0, 9, or -
• The range expression[0-9] means any one character from the set of characters located between characters 0 and 9 inclusive.

What determines what characters line between other characters? The result depends on your current Locale and is not well-defined.

4.6 Do not use Alphabetic Ranges that depend on Locale, e.g. [a-z]Index

Do not use alphabetic ranges (e.g. [a-z])! The ranges change depending on your system Locale and may change in unexpected ways:

$ touch A B C Z a b c z

$ LC_ALL=C

$ echo *
A B C Z a b c z

$ echo [a-z]
a b c z

$ LC_ALL=en_CA.UTF-8

$ echo *
A a B b C c Z z

$ echo [a-z]
a B b C c Z z

• The range [a-z] meaning “any one character between a and z inclusive” used to mean something when there was only one ASCII English locale.
• Now that multiple locales exist, the meaning of “between a and z inclusive” is ambiguous because it means different things in different locales.
• Always use the POSIX character classes for complete letter ranges, e.g. [:alpha:].
• If you need to specify a partial list of characters, enumerate the list; do not use a range: use [abcdefgh] not [a-h]

5.1 Basic versus Extended Regular Expression syntax: \| vs. |Index

The difference between Basic and Extended Regular expressions is whether the program requires you to use a backslash to make use of the Extended features:

Basic:     . * ^ $ \   \| \? \+ \{ \} \( \)       # must use backslash
Extended:  . * ^ $ \    |  ?  +  {  }  (  )       # do *NOT* use backslash

The ordinary grep program uses Basic Regular Expressions, so you have to use backslashes in front of the Extended characters to turn on Extended features. The egrep Extended Regular Expression program (short for grep -E) doesn’t need the backslashes:

$ grep 'Accepted publickey for \(idallen\|cst8207[abc]\?\)' /var/log/auth.log
$ egrep 'Accepted publickey for (idallen|cst8207[abc]?)' /var/log/auth.log

Basic Regular Expressions are used in these programs and you need to use backslashes to turn on Extended features:

• vi, more, sed, awk, grep

Extended Regular Expressions are used in these programs and you do not need backslashes to enable Extended features:

• less (e.g. man pages)
• egrep and grep -E
• perl and grep -P

The perl program (and grep -P) has its own set of special Perl-compatible Regular Expression features, not described here.

5.2 Extended Feature: Repetition: ? + {n,m}Index

Extended Regular Expressions give you more options when repeating a preceding expression:

$ egrep 'colou?r' doc.txt                       # color or colour not colouur
$ egrep 'has +spaces' doc.txt                   # one or more spaces between
$ egrep '[0-9]{9}' doc.txt                      # 123456789
$ egrep '[0-9]{3}-[0-9]{3}-[0-9]{4}' doc.txt    # 123-456-7890
$ egrep '^.{80}$' doc.txt                       # 80 character lines
$ grep '^.\{,80\}$' doc.txt                     # 80 character or fewer lines

5.3 Extended Feature: Alternation (one or the other): ab|cdIndex

Extended Regular Expressions give you a way of matching one expression or another expression using the logical or bar | operator:

$ grep -E 'dog|cat'  doc.txt              # find lines with dog or cat
$ grep 'dog house\|cat fight' doc.txt     # find lines with "dog house" or "cat fight"

You can do a crude form of alternation using the -e option to give the alternatives (as many as you like) in the grep family of programs:

$ fgrep -e 'dog' -e 'cat' doc.txt         # find lines containing dog or cat
$ grep -e '^dog$' -e '^cat$' doc.txt      # find lines with *only* dog or cat

The or | operator binds very loosely. Everything else has higher precedence:

$ grep -E '^a|b$' doc.txt                 # lines starting with a or ending with b

5.4 Extended Feature: Grouping with parentheses a(b|c)dIndex

Parentheses ( and ) are an Extended feature that can be used to group Regular Expressions for repetition, and to override the precedence rules.

$ egrep 'ab|cd' doc.txt              # ab or cd
$ egrep 'a(b|c)d' doc.txt            # a followed by "b or c" followed by d

$ grep -E '^a|b$' doc.txt            # lines starting with a or ending with b
$ grep -E '^(a|b)$' doc.txt          # lines containing only a or only b

$ egrep 'Bar(bar)+a' doc.txt         # Barbara, Barbarbara, etc.

(Visit Barbara at the Rhababer-Barbara-Bar.)

5.5 Extended Feature: Tags or Backreferences; \1 \2 \3Index

Another extended regular expression feature allows you to match later what matched earlier in a pattern:

• When you use parentheses for grouping, you can refer to the n’th group using \n (backslash followed by the number n).
• The pattern (..)\1 means any sequence of two characters that repeats, e.g. abab or 1-1- or XyXy, etc.
• The \1 in the above example refers backward to the first group (parenthesized) expression.

8.1 POSIX character class names are indivisibleIndex

The POSIX class name includes the surrounding colons and square brackets and nothing should ever be placed inside those brackets. This is a common mistake:

grep '[[^:digit:]]'    # WRONG ! no longer a POSIX class name !
grep '[^[:digit:]]'    # correct - match any single non-digit character

Using what you think is a POSIX character class outside of the enclosing character class square brackets does not work. On some systems, grep will warn you that it doesn’t work:

$ grep '[:alnum:]'       # WRONG !
grep: character class syntax is [[:space:]], not [:space:]

On other systems, the character class expression will quietly match the list of characters inside the outer square brackets, i.e. match one of the characters :, a, l, n, u, or m!

8.2 Regexp matches are as long as possibleIndex

Any Regular Expression match will be as long as possible. They are called “greedy”:

• a.*c matches all of abc___abc – it doesn’t only match the first abc.
• You can turn off this greedy behaviour in some implementations of Regular Expressions. (See the perl expression *?, also available as grep -P.)

8.3 Don’t use repeat operators at line boundaries in grepIndex

All the expressions below match the same set of lines containing a letter a, but the first expression uses a lot less processing power than the others:

$ grep 'a'      file.txt    # this is the cleanest and fastest one
$ grep 'aa*'    file.txt
$ grep 'a.*'    file.txt
$ grep '.*a'    file.txt
$ grep '.*a.*'  file.txt

If you’re looking for lines containing a piece of text, don’t complicate the regexp with repeat operators that waste computer time but don’t change which lines the regexp finds.

8.4 Unix/Linux regex processing is line basedIndex

• Linux text files are usually processed line by line when matching Regular Expressions; regular expressions will not cross line boundaries.
• The newlines at the end of every line are not usually considered part of the text that can be matched.
• The vim editor is an exception – it has a special syntax for matching across line ends, e.g. abc\ndef and abc\_.def, but this doesn’t work anywhere else (so don’t worry about it here).

8.5 Regular Expressions match anywhere in a line: anchoring with ^ and $Index

Unlike GLOB Patterns, which are anchored, Regular Expressions are not anchored unless you make them so using the explicit anchor characters ^ and/or $. Unanchored Regular Expressions “float” down the string until a match is found, and they don’t have to extend to the end of the string.

$ echo a*b                  # anchored: matches axb not abx or xab
$ ls | grep '^a.*b$'        # equivalent anchored Regular Expression
$ ls | grep 'a.*b'          # NOT equivalent unanchored Regular Expression

Regular Expressions “float” down the string unless they are anchored.

8.6 Expressions matching zero length strings match everywhereIndex

• The repetition operator * means “zero or more”.
• A Regular Expression consisting of zero of anything can match anywhere and everywhere between all the characters in a line.

• For example, if you have a line with any 10 characters in it, the zero-length Regular Expression x* (meaning zero or more x characters) could match 11 times, before and after every one of the 10 characters (if it doesn’t match any of the characters themselves):

  $ echo '0123456789' | sed -e 's/x*/-/g'
  -0-1-2-3-4-5-6-7-8-9-

• The grep colour option and web tools such as http://regexpal.com cannot highlight matches of zero characters, but the matches are there!

• The vim editor will highlight the entire line when a zero-length expression matches between all the characters.

8.7 Quote all regexp to hide them from the shellIndex

This Regular Expression below sometimes works, and sometimes does not, depending on what file names match the aa* GLOB pattern in the current directory:

grep aa* foo.txt                      # no quotes, GLOB expands: bad idea

• The shell will try to do filename globbing on aa*, possibly changing it into existing filenames that begin with a before grep runs: we don’t want that.
• Use shell quoting around Regular Expressions; don’t let the shell GLOB expansion change the regex before grep sees the regex.

8.8 Alphabetic ranges are not well-defined in all LocalesIndex

• Do not use alphabetic dashed range expressions, e.g. [a-m], they do not match what you think they match (only lower-case letters) in many common locales.
• Use the POSIX character classes to match all characters, e.g. [[:lower:]]
• Specify all the characters in a partial range, e.g. use [abcdefghijklm] not [a-m]
• Numeric ranges such as [0-9] are accepted. (Are there any locales where [0-9] does not mean the ten digits zero through nine?)
• You can also explicitly set your Locale to be the old ASCII Locale, which makes alphabetic ranges safe to use: LC_ALL=C

9.1 Example: capitalize sentences repeatedly in a document using viIndex

Task: Any lower-case letter following a period and two spaces should be made upper-case. Easy to do using Regular Expressions in vi:

• To search forward in vi, type: /\. [[:lower:]]
• Then type 4~ to make four characters upper-case.
• Then type n (next match) and . (repeat change) as many times as necessary.
• The n command moves to the next occurrence, and . repeats the capitalization command.

9.2 Example: uncapitalize in middle of wordsIndex

Any upper-case character following a lower case character should be made lower case, e.g. uNcapitalize or aWkward or iN

• to search forward in vi, type: /[[:lower:]][[:upper:]]
• then type l to move one to the right (off of the lower-case letter)
• type ~ to change the capitalization
• type nl. as necessary
• the l is needed because vi will position the cursor on the first character of the match, which in this case is a character that doesn’t change.

Advanced: In vim you can also use the syntax /[[:lower:]][[:upper:]]/b1 to both match the text and move the cursor right one position. Then you can just repeat the two characters n. as many times as necessary. The vim editor has very advanced pattern search and cursor position capabilities; type :help regexp

10.1 http://lynda.comIndex

• Some students are already comfortable with the command line
• For those who aren’t, yet another tutorial source that might help is Lynda.com
• All Algonquin students have free access to Lynda.com
• Unix for Mac OSX users:

Lynda.com has a course on regular expressions

The problem is that it covers our material as well as some more advanced topics that we won’t cover

It is a good presentation, and the following chapters should have minimal references to the “too advanced” material

• Chapter 2 Characters
• Chapter 3 Character Sets
• Chapter 4 Repetition Expressions

10.2 Interactive Regular Expression TutorialIndex

For a quick interactive tutorial on Regular Expressions, see http://regexone.com/ but be aware that this tutorial uses some short-hand expressions that we don’t use in this course because they don’t work everywhere:

The tutorial does not use or understand the POSIX character classes that are more standard in Unix/Linux programs.