Your shell has a pathname-matching (wildcard) feature that makes operating on large numbers of pathnames easy:
$ cp a* .. # copy all names starting with 'a' to the parent directory
The Unix name for wildcard pattern matching is GLOBbing, from the idea that the pattern matches a “global” list of names. Other operating systems may call these wildcard characters.
GLOB patterns do not only match file names. The names being matched by a GLOB pattern can be anything: files, directories, symbolic links, etc. Sometimes we say that GLOB patterns match “file names”, but what we really mean is that they match any kind of name.
The shell will try to expand any tokens on the command line that contain unquoted GLOB characters into existing pathnames in the file system. The shell will try to match the GLOB patterns in those tokens against existing pathnames in the file system to produce a list of existing pathnames.
GLOB patterns cannot generate any names that do not exist. The GLOB patterns must always match existing names.
Characters that the shell will try to expand to match existing pathnames in the file system. These characters are sometimes called “wildcard” characters in other systems.
metacharacter (or meta-character)
A character that has a special meaning to the shell. The letter a or the digit 1 have no special meaning to the shell and they are not shell metacharacters. The symbol asterisk (or star) * is a metacharacter, and so are semicolons, blanks/spaces and many other symbols. You can usually turn a metacharacter into an ordinary non-special character by quoting it. (See below for “quoting”.)
token or word
A non-blank piece of the command line that can’t be further separated into other tokens by the shell. The unquoted two-word string one two contains two (blank-separated) tokens. The unquoted string one;two contains three tokens, because the shell treats the metacharacter semicolon ; specially and splits the string into three pieces: the word one, the meta-character ;, and the word two. The four-character string cd.. contains only one token. (The shell does not split the cd away from the .. part – it’s all treated as one word because neither letters nor periods are meta-characters to the shell.)
quoting
Turning off the special meaning of metacharacters by surrounding them with single or double quotes, or putting a backslash in front of them:
$ echo 'This is a semicolon; the quotes hide it from the shell.'
$ echo "This is a semicolon; the quotes hide it from the shell."
$ echo This is a semicolon\; the backslash hides it from the shell.
hidden names
Names that start with a leading period (dot – .) are not shown by default by some commands (e.g. ls) and leading periods are never matched by GLOB metacharacters, even GLOB lists that contain a period such as [.]. Names staring with periods are called “hidden” names, e.g. hidden files and hidden directories.
These are the GLOB metacharacters recognized and processed by the shells:
* - (asterisk, star) matches zero or more of any characters
? - (question mark) matches any one (1) character
[list] - (square brackets)
- Matches any one (1) character in the list of characters, e.g.
[abc] matches one a or one b or one c (only one of the three).
- WARNING: The list can contain a range of characters, but don't
use ranges until you read more on Internationalization!
- The list can be inverted/complemented by using ! at the start,
e.g. [!abc] means "any one character not a or b or c".
The shell always processes GLOB characters that it finds on the command line, even for commands that do not take pathnames. (The shell doesn’t know which commands do or do not take pathnames.) For example:
$ mail *
will call the mail program and give it all the non-hidden names in the current directory, which makes no sense because mail programs want email addresses, not pathnames. The shell can’t know which programs want pathnames and which ones don’t, so it always expands GLOB patterns!
GLOB metacharacters never match the leading period on any hidden names, so you don’t have to worry about matching the names . or .. accidentally.
3.1 Using * to match any number of any charactersIndex
As a GLOB metacharacter, the asterisk * matches zero or more of any character in a name, including spaces or other strange characters. The * never matches the leading period on a hidden name, so echo * never shows any names starting with a period.
The GLOB pattern *foo matches non-hidden names ending with foo, including foo, xxxfoo and 123foo.
The GLOB pattern foo* matches names beginning with foo, including foo, fooxxx and foo123.
The GLOB pattern *foo* matches non-hidden names containing foo anywhere, including foo, fooxxx and 123foo and ZZZfoo@@@.
3.2 Using ? to match only one single character, any characterIndex
As a GLOB metacharacter, the question mark ? matches exactly one of any character in a name, including a space or other strange character. The ? never matches the leading period on a hidden name, so echo ? never shows the current directory name . that is a single period.
The GLOB pattern ??? matches non-hidden names that are exactly three characters long.
The GLOB pattern ???* matches non-hidden names that are three or more characters long.
3.3 Using [list] to match single characters from a listIndex
As a GLOB metacharacter pair, the square brackets [list] match exactly one character in a name from a list of characters. The list of characters can never match the leading period on a hidden name, so echo [.] never shows the current directory name . that is a single period.
The GLOB pattern [abc] does not match the three-character name abc; it matches only the one-character names a, or b, or c:
$ touch a b c abc
$ echo [abc]
a b c
No matter how many characters are in the list, a [list] pattern will only match exactly one of the characters in the list, not more than one, not less.
The GLOB patterns [aA] and [a][A] are very different:
The GLOB pattern [aA] is one list, so it matches only one-character names. It matches any one character from the list [aA], so it matches any one-character name that is either a or A.
The odd GLOB pattern [a][A] is made up of two lists, so it matches only two-character names. It matches only the two-character name aA that is made up of a (taken from the first list) followed by A (taken from the second list). It only matches aA.
Having a GLOB square bracket list with only one character in it, e.g. [a], is not usually useful, since it matches exactly one character a, so rather than write *[a]bc use the equivalent and much simpler *abc that matches exactly the same names.
Aside: There are shell options that affect how GLOB patterns are evaluated, and some of these shell options (e.g. nullglob) may make single-character lists useful in special cases. Don’t worry about that until you are more advanced.
No matter how many characters are in the list, a [] pattern will only match exactly one of the characters, not more than one, not less.
3.3.1 Do not use ranges of letters in [...] listsIndex
You can use a dash - to indicate a range of digits inside a [] list, e.g. [1-5], but don’t use ranges of letters, e.g. [a-c] unless you fully understand the effects of your machine’s internationalization locale setting.
For many (most?) modern Linux machines with a modern locale setting (e.g. en_US.utf8), the trivial character range [a-c] actually matches the five characters: a A b B c! Don’t use character ranges!
Normally, GLOB patterns are case-sensitive so that abc* does not match any of the names ABC, aBc, Abc, etc. If you want to match both upper- and lower-case letters in names, make each letter into its own little two-character [] list so that the list matches either upper- or lower-case:
If you are matching any letters, not specific ones, you can use POSIX character classes such as [:upper:], [:lower:] or [:alpha:] inside the [] list:
$ touch a b c A B C abc ABC 1 2 3 123 1a 2b 3c x1 y2 z3
$ echo [[:lower:]]
a b c
$ echo [[:upper:]]
A B C
$ echo [[:alpha:]]
A B C a b c
$ echo [[:alpha:]]*
ABC B C a abc b c x1 y2 z3
$ echo *[[:alpha:]]
ABC B C a abc b c 1a 2b 3c
No matter how many characters are in the list, a [] pattern will only match exactly one of the characters, not more than one, not less.
Until you are sure you know how the shell uses GLOB patterns to match names, use the echo or ls command to see what names are being matched (if any). Before you do any of these commands using a GLOB pattern:
$ rm [abc]* # DON'T DO THIS until you verify the GLOB pattern!
$ touch [abc]* # DON'T DO THIS until you verify the GLOB pattern!
$ cp [abc]* ~/savedir/ # DON'T DO THIS until you verify the GLOB pattern!
try one of these harmless display commands to verify that the GLOB pattern matches the correct names:
$ echo [abc]* # this verifies that the GLOB pattern works
$ ls -d [abc]* # this verifies that the GLOB pattern works
(The -d option makes ls show only the names of any directories, not their contents as it usually does.)
If either of these harmless commands displays on your screen the correct list of matching names, then you can replace it with the rm command (or whatever command you really wanted to use).
If there are too many pathnames to view comfortably on your screen, you can use other commands to limit the display or count the names:
$ ls -d [aA]* | less # view all the pathnames in a pager program
$ ls -d [aA]* | head # look at first ten pathnames
$ ls -d [aA]* | tail # look at last ten pathnames
$ ls -d [aA]* | wc # count the pathnames; is the count right?
$ cp [aA]* ~/savedir/ # if GLOB is correct, use the actual command
Make sure that the GLOB pattern is correct before you use it.
Unlike DOS/Windows wildcarding, the GLOB features are done by the shell, not by the individual programs called by the shell.
The shell will do this GLOB expansion irrespective of the name of the command being used, even if the command being used does not accept or process pathnames. The shell does not know which commands expect pathnames; the shell always expands tokens that have GLOB patterns in them, no matter what the command is. Typing:
$ wc -l *
counts the lines in all the non-hidden names in the current directory, and using pathnames with wc is useful. Typing the following commands (that will get the same list of GLOB-expanded pathnames as wc, but that do not operate on pathnames) is probably not going to work well:
$ echo * # first verify what the GLOB pattern * matches
$ wc * # yes, wc expects pathnames and this works - good
$ kill * # probably wrong - kill expects process numbers!
$ mail * # probably wrong - mail expects mail addresses!
In all the command lines above, the shell GLOB expansion delivers the same pathnames to the command being typed. Two of the commands will likely not work, because these commands don’t take pathnames as arguments.
If you do not want GLOB processing to happen, hide the GLOB characters from the shell by using quoting – surround the token with single or double quotes or precede each GLOB metacharacter with a backslash:
$ echo *
a b c
$ echo "*"
*
$ echo '*'
*
$ echo \*
*
The first argument to the grep command is a regular expression pattern and is almost never intended to be expanded by the shell as a GLOB pattern, so you must quote the first argument to stop GLOB expansion:
$ grep '?' /etc/passwd
$ grep '.*sh' /etc/passwd
If you don’t want the shell to use GLOB on your command arguments, you must quote the arguments to commands so that the shell doesn’t see the metacharacters. Here are several examples using quoting to hide GLOB characters:
$ echo "*** Warning: assuming the worst ***"
$ mail -s "Coming to dinner?" idallen@idallen.ca
$ tr '[:lower:]' '[:upper:]' <lowercase >uppercase
$ find /usr/bin -name '*go*' -ls
The shell will always try to expand unquoted GLOB characters.
By default, GLOB patterns will match any existing pathname component – the pathname might be a file, a directory, or some other Unix pathname type (e.g. symbolic link, socket, fifo):
$ mkdir a ; touch b
$ echo *
a b
Be careful that the GLOB pattern expands to be pathnames compatible with the command you are using!
$ mkdir a ; touch b
$ rm *
rm: cannot remove 'a': Is a directory
$ mkdir a ; touch b
$ rmdir *
rmdir: failed to remove 'b': Not a directory
Each separate token found by the shell in the command line is examined for GLOB characters, and has those characters expanded separately. A single token containing GLOB patterns never produces duplicate names; every name appears exactly once. The only way for the same pathname to appear twice is for the shell to find two separate tokens containing GLOB characters and expand them separately.
Some of the command lines below produce duplicate or triplicate output; since, each blank-separated token is separately GLOB-expanded by the shell:
$ touch a b c
$ echo *
a b c
$ echo * *
a b c a b c
$ echo * * *
a b c a b c a b c
The command lines below produce identical output; since, only one token is found and expanded by the shell, and * means the same thing as ** or *** when there are no spaces between the metacharacters:
$ touch a b c
$ echo *
a b c
$ echo **
a b c
$ echo ***
a b c
If a GLOB pattern expansion matches and produces a name with a space or other shell metacharacter in it, that space or shell metacharacter is not seen or processed a second time by the shell. The space or other meta-character in the name is not re-interpreted by the shell and it does not produce extra command line arguments. Whatever the GLOB pattern expands to is kept together as a single pathname:
$ touch 'a b *' # create a five-character name with space and asterisk
$ echo *
a b * # the GLOB * matches the name 'a b *'
$ wc *
0 0 0 a b * # the expanded pathname is still one name
10 GLOB metacharacters never match leading periods (hidden names)Index
By default, GLOB metacharacters never match a leading period in a pathname component. (You can change this behaviour using shell options, but nobody does because of the side-effects noted below.)
To match a pathname component that starts with a period, you must explicitly put the real period at the start of the pattern. (This is why GLOB patterns never match the directory entries . or .. unless you start the pattern with a period.) The no-leading-period rule even applies to GLOB character lists that contain a period ([.]) – they are still GLOB patterns and do not match leading periods in pathnames!
This refusal to match hidden names is there so that people who type things such as rm -rf * don’t accidentally match and remove the contents of the hidden directory name .. from their accounts.
Beware that GLOB patterns .* and .? match the pathname .. and therefore rm -rf .* recursively removes your parent directory (if the rm command doesn’t check first). A safer pattern is rm -rf .??* that is at least three characters long and thus does not match . or .. (or any two-character names starting with a period).
11 Unmatched GLOB patterns are passed unchangedIndex
In Bourne-style shells (e.g. BASH on Linux), if the shell cannot match a GLOB pattern on the command line against any existing pathname, the GLOB pattern is silently left unchanged by the shell and passed unchanged to the command being used.
No error message is generated by the shell by the failed GLOB; the command runs with the unchanged argument still containing the unmatched GLOB metacharacters. This is almost never what you want. For example:
$ touch someverylongfilename.txt
$ ls
someverylongfilename.txt
$ cp /etc/passwd sme* # typimg error; should be some* !
$ ls
sme* someverylongfilename.txt # silently created a new file name
If GLOB matching fails, because no names match the pattern, the GLOB pattern is passed unchanged to the command. Usually the unexpanded GLOB pattern isn’t what you want. The command may produce an error message for the nonexistent pathname passed to it by the shell:
$ ls
happybirthday.txt
$ rm hp* # typimg error; should be ha* !
rm: cannot remove 'hp*': No such file or directory
$ rm ha* # this GLOB pattern matches
Another example of a GLOB pattern failing and being passed unchanged to the command (not useful):
$ ls
abc
$ wc abb* # typing error; should be ab*
wc: abb*: No such file or directory
$ wc ab* # this GLOB pattern matches
0 0 0 abc
C-Shells do produce error messages when GLOB patterns fail, and refuse to run the command. You can optionally make BASH behave this way, too, by setting the BASH failglob option using shopt built-in command (highly recommended!). That option will make it an error to use a GLOB pattern that doesn’t match anything.
12 Unquoted GLOB patterns are hidden problemsIndex
Because the shell normally doesn’t tell you that a GLOB expansion failed, you might not notice that a GLOB is quietly failing until the contents of the file system change one day, causing the pattern to expand.
Here are two examples, one using fgrep and one using find.
This example puts the text This is an *error* message. in a file and uses the fgrep command to search for the unsafe, unquoted GLOB string *error* in the file:
$ mkdir empty ; cd empty
$ echo 'This is an *error* message.' >file.txt
$ echo *error* # use echo to see the GLOB fail to expand
*error*
$ fgrep *error* file.txt # fgrep finds the line in the file
This is an *error* message.
Above, the unquoted *error* GLOB pattern doesn’t match any file names, so it gets passed unchanged as a string to the fgrep command, which searches for it in the file and prints the line containing it. This is good.
Now watch what happens to the identical fgrep command when we create just one file that triggers the unquoted GLOB pattern into expanding:
$ touch terrorized.txt
$ fgrep *error* file.txt # no output this time! Why?
You can again use echo to diagnose the problem:
$ echo *error* # use echo to see the GLOB expand
terrorized.txt
$ echo fgrep *error* file.txt # use echo to see the GLOB expand
fgrep terrorized.txt file.txt
The newly-expanding GLOB turns a working fgrep command into one that fails. The fgrep command may have been working correctly for years, until one day someone creates a file that triggers the GLOB pattern that causes the failure.
If you don’t want something that looks like a GLOB pattern to be expanded by the shell, make sure to quote it to hide the GLOB characters from the shell:
$ fgrep '*error*' file.txt # hide the GLOB with quoting
This is an *error* message.
The shell will always try to expand tokens that look like GLOB patterns used on command lines. Don’t rely on a GLOB failing if you don’t want GLOB characters to expand. Always quote arguments that contain characters that the shell might take as GLOB expansions (especially arguments to the fgrep and grep command)!
Above, the unquoted *go* GLOB pattern doesn’t match any file names, so it gets passed unchanged as a pattern to the find command, which uses it (a pattern) to find and display all names containing go in the /usr/bin directory.
Now watch what happens to the identical find command when we create some files that trigger the unquoted GLOB pattern into expanding:
$ touch Algonquin algorithm good # create three file names
$ find /usr/bin -name *go* # run exactly the same command as before
find: paths must precede expression: algorithm # COMMAND FAILS !!!
You can again use echo to diagnose the problem:
$ echo *go* # use echo to see the GLOB expand
Algonquin algorithm good
$ echo find /usr/bin -name *go* # use echo to see the GLOB expand
find /usr/bin -name Algonquin algorithm good
The newly-expanding GLOB turns a working find command into one that fails. The find command may have been working correctly for years, until one day someone creates files that trigger the GLOB pattern that causes the failure.
If you don’t want something that looks like a GLOB pattern to be expanded by the shell, make sure to quote it to hide the GLOB characters from the shell:
The shell will always try to expand tokens that look like GLOB patterns used on command lines. Don’t rely on a GLOB failing if you don’t want GLOB characters to expand. Always quote arguments that contain characters that the shell might take as GLOB expansions (especially arguments to the find command)!
13 GLOB patterns are matched before running the commandIndex
GLOB patterns are matched and expanded by the shell before the command is run.
This command below with two GLOB characters creates two files in an empty directory because neither GLOB character matches any files at the time the shell goes looking for them:
$ mkdir empty ; cd empty
$ touch ? * # GLOB doesn't match; creates two new files
$ ls
* ?
The apparently similar set of two touch commands below creates only one file because the second touch command GLOB pattern matches the file created by the first touch command:
$ mkdir empty ; cd empty
$ touch ? # GLOB doesn't match anything; creates new file
$ touch * # GLOB matches the ?; no new file created
$ ls
?
GLOB patterns are expanded on the command line by the shell before the command is run.
GLOB patterns apply in the directory indicated by the place in the pathname where they appear. For example (try these yourself):
$ echo * # match names in the current directory
$ echo ./* # match names in the current directory
$ echo ../* # match names in the parent directory
$ echo /* # match names in the ROOT directory
$ echo /usr/* # match names in the /usr/ directory
$ echo /usr/bin/* # match names in the /usr/bin directory
A GLOB pattern that appears to the left of any slash must match one or more directory names (or symbolic links to directory names); because, only directory names can appear to the left of slashes in valid pathnames, and GLOB patterns only match existing, valid pathnames. In the examples below, the GLOB pattern must match …
$ echo */ls # directories in the current directory that contain "ls"
$ echo ../*/ls # directories in the parent directory that contain "ls"
$ echo /*/ls # directories in the ROOT directory that contain "ls"
$ echo /usr/*/ls # directories in the /usr directory that contain "ls"
15 GLOB patterns do not match or span slashesIndex
GLOB characters can match any character in a pathname component, including spaces, newlines, and unprintable characters, but they do not match or cross the slashes that separate pathname components. Ordinary shell GLOB patterns do not match slashes in a pathname. (The find command can cross slashes, as can GLOB patterns using the bash shell option globstar.)
If a token containing GLOB patterns has two non-adjacent slashes, all the matched existing pathnames must also have exactly two slashes:
A GLOB pattern does not cross slashes in a pathname.
If the token containing the GLOB pattern has N non-adjacent slashes, all the matched existing pathnames will also have exactly the same number of slashes:
$ echo /* # matches names directly under the ROOT but no deeper
$ echo /bin/* # matches names directly under /bin/ but no deeper
$ echo /usr/bin/* # matches names directly under /usr/bin/ but no deeper
To match multiple levels of pathnames in a directory hierarchy, you must use a separate GLOB pattern between each slash:
$ ls /* # GLOB applies in / only (no deeper - one slash only)
$ ls /*/* # GLOB applies in / and then in /*/
$ ls /*/*/* # GLOB applies in / and then in /*/ and then in /*/*/
$ ls /usr/bin/* # GLOB applies only in /usr/bin/
$ ls /usr/*/* # GLOB applies in /usr/ and then in /usr/*/
The list of pathnames produced by GLOB patterns /* and /*/* have no pathnames in common. The first list contains only pathnames with one slash; the second list contains only pathnames with two slashes. The * GLOB pattern does not match the slash that separates pathname components.
Basic GLOB patterns matching non-hidden names in the current directory:
$ touch a ab abc abcde abcdef
$ ls
a ab abc abcde abcdef
$ echo *
a ab abc abcde abcdef
$ echo **
a ab abc abcde abcdef
$ echo ****
a ab abc abcde abcdef
$ echo * *
a ab abc abcde abcdef a ab abc abcde abcdef
$ echo ?
a
$ echo ??
ab
$ echo ? ?
a a
$ echo ?? ??
ab ab
$ echo ???
abc
$ echo ???*
abc abcde abcdef
$ echo *???
abc abcde abcdef
$ echo ?*?
ab abc abcde abcdef
$ echo *[cf]
abc abcdef
$ echo *c*
abc abcde abcdef
These shell patterns match all non-hidden names in the /tmp/idallen directory:
17 A GLOB pattern that matches only directory namesIndex
Given an existing directory dir1, this pathname argument is valid:
$ ls dir1/.
...files list here...
Given file name file1, this similar pathname argument is not valid:
$ ls file1/.
ls: cannot access file1/: Not a directory
A file name file1 cannot be used as if it were a directory. Only directory names can appear to the left of slashes in valid Unix pathnames.
If we replace the name dir1 in the token dir1/. with a GLOB pattern, as in */., the result can only expand to be a valid pathname if the GLOB pattern matches a directory.
Question: What does echo */. output, and how does it differ from the echo * pattern? We know that echo * echoes every non-hidden name in the current directory, both files and directory names. How does using */. change this?
Hint: Wildcards (GLOB patterns) only expand to match valid names, so the wildcard */. only matches names where */. is a valid pathname. The pattern */. can only be valid if the GLOB pattern to the left of a slash matches a particular kind of name. What is it?
Author:
| Ian! D. Allen, BA, MMath - idallen@idallen.ca - Ottawa, Ontario, Canada
| Home Page: http://idallen.com/ Contact Improv: http://contactimprov.ca/
| College professor (Free/Libre GNU+Linux) at: http://teaching.idallen.com/
| Defend digital freedom: http://eff.org/ and have fun: http://fools.ca/
