Updated: 2017-01-20 00:48 EST

1 Why Use Remote Command LinesIndexup to index

System administrators often need to run command lines on remote machines without doing a full remote login. We just want the output of the one command line; that is all. We don’t want to log in, run the command, and log out again. Or, we might want to run a command on a remote machine and redirect the output into a local file, which can’t be done using a full login.

For example, one might want to verify an account disk quota on a remote machine using the quota command remotely, without logging in. To run a command remotely, simply add the command line after the ssh hostname on the command line. You may need to accept the machine’s host key, and you will need to enter your remote system password:

$ ssh localhost hostname
abcd0001@localhost's password:
idallen-ubuntu
$ 

$ ssh cst8207.idallen.ca quota
abcd0001@cst8207.idallen.ca's password:
Disk quotas for user abcd0001 (uid 1000): 
 Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
/dev/sda1        52  204800  512000              15    5000    8000
/dev/sdc1       568  204800  512000             237    5000    8000
$ 

$ ssh cst8207.idallen.ca quota >quota.txt     # save output in a file
abcd0001@cst8207.idallen.ca's password:

$ wc quota.txt
4 30 255 quota.txt

The single command line is executed by a shell on the remote machine and the output is displayed. The remote output can be redirected into a file on the local machine. No full login to the remote machine is done.

Another example: Perhaps you might need to create a directory on several (dozen? hundred?) machines. A quick way to do this is with a shell for loop that runs an ssh remote command line to each machine without needing to log in, type each command manually, and then log out:

$ for h in host1 host2 host3 host4 host5 host6 host7 ; do
>   ssh $h.example.com mkdir newdir
> done

The above loop works best when you have enabled password-free public key logins to the remote machines, to avoid typing many passwords, but even without that it’s much faster than having to log in to each machine, execute the command, then log out!

2 Two Shells Read Every Command LineIndexup to index

Two shells are involved in reading and parsing every command line sent to a remote system using ssh:

  1. One shell is running on the local machine, reading your keyboard to collect the command line for ssh.
  2. Another shell is running on the remote machine to execute the command line collected and sent there by ssh.

The local shell processes the command line you type in, doing such things as local GLOB and variable expansion on the local machine, and then stripping off one layer of quotes and passing the result to to the ssh command, which will send the arguments to the remote machine.

The remote machine passes the stripped command line arguments to a remote shell, and that remote shell does more GLOB and variable expansion and quote stripping on those arguments on the remote machine.

The command line is processed by two shells.

For example, we can run multiple commands on the remote machine by passing semicolons in the command line sent to the remote shell. We have to hide these semicolons from the local shell by quoting them. Without the quoting, the semicolons would be seen and acted on by the local shell, creating two commands in the local shell, like this:

$ cd /tmp
$ ssh cst8207.idallen.ca hostname ; pwd          # unquoted semicolon
idallen-ubuntu                   # hostname executes on remote system
/tmp                             # pwd executes on local system

Above, the ssh sends only the hostname command to the remote system; the local shell splits the command line we typed on the semicolon and the pwd runs on the local system.

Using quotes to hide the semicolon, all the special characters can be sent to the remote shell, creating two commands on the remote machine:

$ ssh cst8207.idallen.ca "hostname ; pwd"        # quoted semicolon
idallen-ubuntu                 # hostname executes on remote system
/home/abcd0001                 # pwd also executes on remote system

This is what is happening in the ssh command above:

  1. Quotes hide the semicolon metacharacter from the local shell so that it gets passed to the ssh command as part of an ordinary argument. (The quotes are removed from the argument by the local shell before handing the argument to the ssh command. Quotes delimit the argument, but are never made part of it when the argument is passed to the command, as described in Quoting.)
  2. The ssh command then passes its argument string to the shell on the remote machine. (Remember: The string does not have quotes around it when ssh receives it and passes it to the remote machine.)
  3. The shell on the remote machine receives the argument string. It splits the string into tokens on unquoted blanks and acts on any unquoted shell metacharacters in the string, such as the semicolon used to put multiple commands on one line.
  4. The semicolon shell metacharacter is seen by the remote shell. It allows multiple commands to be written on one line. The remote shell executes two commands (separated by the semicolon) and sends the two outputs back to our screen.

The act of using SSH to send a command to a remote machine uses two shells. Each shell recognizes and strips a layer of quoting from the line being sent. You can see this by putting an echo in front of the command line, to see how the local shell strips one layer of quotes. The stripped line is what is actually being passed to the ssh command and then sent to the remote system’s shell:

$ echo ssh cst8207.idallen.ca "hostname ; pwd"   # insert echo in front
ssh cst8207.idallen.ca hostname ; pwd

Above, the echo on the screen shows us that the ssh command is sending the argument string hostname ; pwd to the remote machine. (The string received by ssh and passed to the remote machine has no quotes around it, because the local shell stripped the quotes.)

When that (unquoted) string argument hostname ; pwd is received by the shell on the remote machine, the remote shell will split the argument into three tokens (because shells split on unquoted blanks). The remote shell will see the unquoted semicolon metacharacter, and execute two separate commands on the remote machine. The output of the two commands will return and display on your local screen, where you could redirect it into a local file.

Two shells are involved in every ssh remote command. One layer of quotes used on an ssh command line hides the metacharacters from expansion by the local shell, but not from expansion by the remote shell.

3 Quoting Remote Command Lines TwiceIndexup to index

Two shells are involved in every ssh remote command. One layer of quotes used on an ssh command line hides the metacharacters from expansion by the local shell, but not from expansion by the remote shell.

Where things get tricky is when the command line you want to run on the remote machine contains shell metacharacters that need quoting on both the local machine and on the remote machine. One level of quoting doesn’t work:

$ echo "Here today ; gone tomorrow."
Here today ; gone tomorrow.

$ ssh localhost echo "Here today ; gone tomorrow."
Here today
bash: gone: command not found

To see the problem more clearly, again insert an echo into the start of the ssh command line to show what arguments are being passed to the ssh command for execution on the remote system:

$ echo ssh localhost echo "Here today ; gone tomorrow."
ssh localhost echo Here today ; gone tomorrow.

Above, the echo on the screen shows us that the ssh command is sending an unquoted semicolon to the shell on the remote system. The remote shell splits the command line in two on the unquoted semicolon and tries to execute a nonexistent command named gone.

To quote something on the remote machine in the remote shell requires careful thought. One level of quotes isn’t enough. You need one level of quotes to hide things from the shell you’re typing into on the local machine, and inside those quotes you need another level of quotes that get sent to the remote shell to do the quoting and hiding remotely as well.

Let’s modify our example and put in single quotes, inside the double quotes, to hide the semicolon from the shell on the remote system:

$ echo ssh localhost echo "Here today ';' gone tomorrow."
ssh localhost echo Here today ';' gone tomorrow.

Above, the echo on the screen shows us that the ssh command will send a single-quoted semicolon to the remote system, which will keep it from being interpreted as a shell metacharacter. We can remove the leading echo and confirm that it works as expected:

$ ssh localhost echo "Here today ';' gone tomorrow."
Here today ; gone tomorrow.

Before you think we’ve solved the problem, let’s explore a bit further. Spaces are also shell meta-characters, and the above remote echo is not receiving exactly the same arguments as its local version. The local echo is receiving only one argument; the remote echo is receiving five:

$ echo "Here today ; gone tomorrow."                 # one argument to echo
$ ssh localhost echo "Here today ';' gone tomorrow." # five arguments to echo

The remote shell executes an echo command with five arguments, not one, because the ssh command line is sent to the remote machine unquoted and the remote shell splits the line into tokens separated by spaces. This is a problem if the remote command is not expecting multiple arguments. To make the problem more obvious, let’s try touching a file containing a space both locally and remotely:

$ touch "my file"                                # creates one file
$ ssh localhost touch "my file"                  # creates two files!

Let’s remind ourselves what ssh is passing to the remote system by inserting an echo in front of the ssh command line:

$ echo ssh localhost touch "my file"             # insert echo in front
ssh localhost touch my file                      # two arguments to touch

Above, the echo on the screen shows us that the ssh command is sending a command line that will cause two separate arguments to the touch command, not one. There are no quotes being sent to the remote system to hide the space in the file name.

To fix the problem of the space in the name on the remote system, we need to hide the blanks from both the local shell and the remote shell by putting quotes inside of quotes. On easy way to do this is by putting single quotes inside double quotes:

$ echo ssh localhost touch "'my file'"           # single inside double
ssh localhost touch 'my file'                    # one argument to touch

Above, the echo on the screen shows us that the ssh command is now sending a command line containing one single-quoted argument to the touch command. We can remove the leading echo and the command will work as expected:

$ ssh localhost touch "'my file'"                # now creates one file

We need “quotes inside of quotes” to hide all the metacharacters on both the local system and the remote system.

We can also put double quotes inside single quotes to hide them. For example, suppose we want to run the command mkdir "Space Dir Name" on the remote machine. The command line below doesn’t work; it creates three directory names not one directory name with two spaces in it.

$ ssh localhost mkdir "Space Dir Name"          # creates three directories

Inserting a leading echo command at the start helps us see this:

$ echo ssh localhost mkdir "Space Dir Name"     # insert echo in front
ssh localhost mkdir Space Dir Name              # three arguments to mkdir

Above, the echo on the screen shows us that the ssh command is sending a command line that will cause three arguments to the mkdir command, not one argument inside quotes.

We have to hide the double quotes from the local shell so that they get sent as double quotes to the remote shell. One way to do this is to put the double quotes inside single quotes:

$ ssh localhost mkdir '"Space Dir Name"'        # creates one directory

Inserting a leading echo command at the start helps us see this:

$ echo ssh localhost mkdir '"Space Dir Name"'   # insert echo in front
ssh localhost mkdir "Space Dir Name"            # one argument to mkdir

Here is another example showing how one level of quotes isn’t enough to preserve metacharacters on the remote system:

$ ssh locahost echo "Hello         World"        # only one level of quotes
Hello World                                      # no quotes; blanks disappear

$ echo ssh locahost echo "Hello         World"   # insert echo in front
ssh locahost echo Hello         World            # no quotes are sent

$ echo ssh locahost echo "'Hello         World'" # put quotes inside quotes
ssh locahost echo 'Hello         World'          # one level of quotes remain

$ ssh localhost echo "'Hello         World'"     # remove the leading echo
Hello         World                              # quoted blanks are preserved

Quotes that we want sent to the remote shell themselves need quoting, to protect those quotes from the local shell.

Here is another example. We need to use a real asterisk (*) on the remote system, so it needs to be quoted on the remote system, as well as on the local system, to hide it from the shell on both systems.

The first step is to know how to hide the asterisk from the local shell. Either single or double quotes will work:

$ echo '*'                                       # works fine locally
*

Now we need to add the extra quoting needed to pass both the single quotes and the asterisk to the remote shell. To make this work we use echo to develop a correctly quoted ssh command line. When the echo command shows us a correctly quoted line, we know it will work remotely. We see this doesn’t work:

$ echo ssh localhost echo '*'                    # WRONG
ssh localhost echo *                             # WRONG - note lack of quotes

The above line echoed on the screen shows that no quotes are being passed to the remote shell; that won’t work on the remote machine because the remote shell will expand the unquoted asterisk. Let’s hide the single quotes and the asterisk inside double quotes, and then it works:

$ echo ssh localhost echo "'*'"                  # hide the single quotes
ssh localhost echo '*'                           # now it has quotes

$ ssh localhost echo "'*'"                       # remove the leading echo
*                                                # it works

3.1 Quoting Quotes in remote command linesIndexup to index

Suppose we want to echo a double quote on the remote system. The first step is to know how to hide the double quote from the local shell. Single quotes will work:

$ echo '"'                                       # double inside single
"

The command line below does not work; the first level of single quotes is stripped by the local shell and the double quote is sent alone to the remote system where it causes a syntax error:

$ ssh localhost echo '"'                         # doesn't work with ssh
bash: -c: line 0: unexpected EOF while looking for matching '"'
bash: -c: line 1: syntax error: unexpected end of file

Inserting a leading echo command at the start helps us see this:

$ echo ssh localhost echo '"'                    # insert echo in front
ssh localhost echo "                             # only one quote is being sent

We need to quote the single quotes so that they get passed to the remote system to protect the double quote from the remote shell. The way we did this above was to put the whole string inside double quotes, but we can’t do that if there is a double quote inside the string we want to surround with double quotes. The quotes won’t match up correctly and we will get an input continuation prompt from the local shell:

$ ssh localhost echo "'"'"                       # unmatched quote
>                                                # unmatched double quote!

One solution is to hide the internal double quote from the local shell:

$ ssh localhost echo "'\"'"                      # hide embeded double quote
"                                                # works!

Another solution is to leave single quotes around the double quote to hide it from the local shell and send two more single quotes to the remote shell using backslashes to hide them locally:

$ ssh localhost echo \''"'\'                     # add two single quotes
"                                                # works!

Inserting a leading echo command at the start helps us see this:

$ echo ssh localhost echo \''"'\'                # insert echo in front
ssh localhost echo '"'                           # correctly quoted

Above, the echo on the screen shows us that the ssh command is sending a single-quoted double quote to the shell on the remote system. The single quotes will hide the double quote from the remote shell, as desired.

3.2 Quoting Variables in remote command linesIndexup to index

Things get more complicated if we are trying to protect something that looks like a variable expansion in a remote command line.

The following example shows how the shell variable $$ is protected by single quotes when used locally, but it expands to be the process ID when the command line is sent to the remote system:

$ echo 'It costs $$.'                            # single quotes stop expansion
It costs $$.

$ ssh localhost echo 'It costs $$.'              # doesn't work remotely
It costs 18029.                                  # $$ variable expands

$ echo ssh localhost echo 'It costs $$.'         # insert echo in front
ssh localhost echo It costs $$.                  # unprotected $$ variable

Above, the echo on the screen shows us that the ssh command is sending an unquoted $$ variable to the remote shell, which will expand it. To prevent this expansion by the remote shell, we also need to send single quotes to the remote shell, to protect the $$ variable.

You might think that you could simply double-quote the single-quoted string to protect the single quotes, as we did above, so that the single quotes would prevent the variable expansion on the remote system, but it’s not that easy:

$ ssh localhost echo "'It costs $$.'"            # surround with double quotes
It costs 25761.                                  # doesn't work!

$ echo ssh localhost echo "'It costs $$.'"       # insert echo in front
ssh localhost echo 'It costs 25761.'             # $$ variable expanded locally

Above, the echo on the screen shows us that the ssh command is now getting the single quotes correctly, but it is also getting an already-expanded $$ variable, because the local shell expanded $$ in the now-double-quoted string! We need to tell the local shell not to expand the $$ inside the double-quoted string, and one way to do that is to use backslashes to hide the dollar signs inside the double-quoted string:

$ ssh localhost echo "'It costs \$\$.'"          # protect $$ from expansion
It costs $$.                                     # works!

$ echo ssh localhost echo "'It costs \$\$.'"     # insert echo in front
ssh localhost 'It costs $$.'                     # correctly quoted

Another way to send single quotes to the remote shell is to individually quote (using backslashes) the single quotes on either side of the single-quoted string being sent, just as we did in the case of the single-quoted double quote earlier:

$ echo ssh localhost echo \''It costs $$.'\'     # add two more single quotes
ssh localhost echo 'It costs $$.'                # this looks good

$ ssh localhost echo \''It costs $$.'\'          # remove the leading echo
It costs $$.                                     # it works!

$ ssh localhost echo "'"'It costs $$.'"'"        # another way to use single quotes
It costs $$.                                     # it works!

The last example above shows that you can also double-quote the single quotes to hide them from the local shell, instead of using backslashes to hide them.

3.3 Quoting both variables and quotes in remote command linesIndexup to index

Things get much more complicated if we are trying to protect both variable expansions and different types of quotes all in a single remote command line. Getting the mixed quoting right is tricky even when you only have to quote for one local shell.

Suppose we want to echo the string It's "not" easy $$. locally. This string contains both single and double quotes, and also dollar signs. All of these have to be hidden from the local shell:

$ echo "It's \"not\" easy \$\$."                 # use backslashes
It's "not" easy $$.

$ echo 'It'"'"'s "not" easy $$.'                 # use alternating quotes
It's "not" easy $$.

Making this same echo work for a remote shell is not easy, since we need to quote the quotes so that the remote shell gets all the correct quoting shown above.

A strategy to get different types of quotes sent to the remote shell is to single-quote the double quotes and double-quote the single quotes. We always start using a leading echo to see what would be sent to the remote shell first:

$ echo ssh localhost echo "'It'"'"'"'"'"'"'s "'"not" easy $$.'"'"
ssh localhost echo 'It'"'"'s "not" easy $$.'     # looks good

$ ssh localhost echo "'It'"'"'"'"'"'"'s "'"not" easy $$.'"'"  # remove the leading echo
It's "not" easy $$.                              # it works!

You can also use backslashes to escape individual characters so that they are each sent to the remote shell:

$ echo ssh localhost echo \'It\'\"\'\"\'s\ \"not\"\ easy\ \$\$.\'
ssh localhost echo 'It'"'"'s "not" easy $$.'     # looks good

$ ssh localhost echo \'It\'\"\'\"\'s\ \"not\"\ easy\ \$\$.\'  # remove the leading echo
It's "not" easy $$.                              # it works!

When the leading echo command shows us a correctly quoted line on the screen, we know it will work as a remote command without the leading echo. Always check your complex quoting with echo, first!

4 Using a second shell instead of remote loginIndexup to index

Sending a command line to a remote login shell works similarly to sending a command line from the current shell into another local bash shell through a pipeline. A pipeline is useful and faster for working the bugs out of a remote command line before you use it remotely through ssh since no passwords or remote connections are needed to pipe into bash:

$ ssh localhost "date ; whoami ; echo 'Hello  World'"   # slow using ssh
abcd0001@localhost's password:
Tue Apr 15 23:35:29 EDT 2014
abcd0001
Hello  World

$ echo "date ; whoami ; echo 'Hello  World'" | bash -u  # fast using bash
Tue Apr 15 23:36:29 EDT 2014
abcd0001
Hello  World

Above, we can see that in both the ssh case and the pipeline case two shells are involved and two levels of quotes are stripped. What works in the pipeline case will also work in a remote ssh command line. Use bash to help you debug your quoting quickly.

Here is a previous example showing how one level of quotes isn’t enough to preserve metacharacters on the remote system:

$ echo echo "Hello         World" | bash -u      # only one level of quotes
Hello World                                      # no quotes; blanks disappear
$ echo echo "'Hello         World'" | bash -u    # two layers of quotes
Hello         World                              # quoted blanks are preserved

Another example:

$ ssh localhost echo Eat\\\ \\\ \\\;\\\ \\\ Pray\\\ \\\ \\\;\\\ \\\ Linux
Eat  ;  Pray  ;  Linux

$ echo echo Eat\\\ \\\ \\\;\\\ \\\ Pray\\\ \\\ \\\;\\\ \\\ Linux | bash -u
Eat  ;  Pray  ;  Linux

While you are testing and developing a properly-quoted remote command line, you can get the same quoting output results more quickly using echo and a pipeline into a second copy of bash instead of using the much slower ssh. Once you get the bash quoting working, switch to using ssh.

5 Using eval instead of remote loginIndexup to index

The bash shell (and most other shells) contains a keyword that lets you strip two layers of quotes from a single command line.

Use the shell keyword eval in place of ssh localhost to get the same effect. The example below shows using both remote shell, a second bash shell, and eval to test a quoted string:

$ ssh localhost echo Eat\\\ \\\ \\\;\\\ \\\ Pray\\\ \\\ \\\;\\\ \\\ Linux
Eat  ;  Pray  ;  Linux

$ echo echo Eat\\\ \\\ \\\;\\\ \\\ Pray\\\ \\\ \\\;\\\ \\\ Linux | bash -u
Eat  ;  Pray  ;  Linux

$ eval echo Eat\\\ \\\ \\\;\\\ \\\ Pray\\\ \\\ \\\;\\\ \\\ Linux
Eat  ;  Pray  ;  Linux

Using eval removes two levels of quotes just the way that ssh localhost or a pipeline into bash does. You can use eval for testing because it’s faster and doesn’t need us to type in any ssh remote password.

6 Using argv.sh to count argumentsIndexup to index

You can use the argv.sh script from the Class Notes to verify that a string is one single argument to the remote echo command:

$ eval ./argv.sh Eat\\\ \\\ \\\;\\\ \\\ Pray\\\ \\\ \\\;\\\ \\\ Linux
Argument 0 is [./argv.sh]
Argument 1 is [Eat  ;  Pray  ;  Linux]
Author: 
| Ian! D. Allen, BA, MMath  -  idallen@idallen.ca  -  Ottawa, Ontario, Canada
| Home Page: http://idallen.com/   Contact Improv: http://contactimprov.ca/
| College professor (Free/Libre GNU+Linux) at: http://teaching.idallen.com/
| Defend digital freedom:  http://eff.org/  and have fun:  http://fools.ca/

Plain Text - plain text version of this page in Pandoc Markdown format

Campaign for non-browser-specific HTML   Valid XHTML 1.0 Transitional   Valid CSS!   Creative Commons by nc sa 3.0   Hacker Ideals Emblem   Author Ian! D. Allen