% File transfer to/from Unix/Linux machines % Ian! D. Allen -- -- [www.idallen.com] % Fall 2015 - September to December 2015 - Updated 2018-08-30 21:55 EDT - [Course Home Page] - [Course Outline] - [All Weeks] - [Plain Text] Introduction to File Transfer ============================= This file explains how to copy files between Unix/Linux machines and between Unix/Linux and other machines running Windows or Mac OSX. The [Course Linux Server] is an example of a Unix/Linux machine. For security reasons, SFTP file transfer *to* the CLS is not permitted. You may only transfer files *from* the CLS to your local machine. Connecting from other machines to the Course Linux Server --------------------------------------------------------- Usually, you will be using some local machine (e.g. your own laptop or desktop machine running Windows or OSX) and you will want to connect to the [Course Linux Server] (**CLS**) to transfer a file. This means you need to know the *network name* of the CLS. It has two names: The CLS is located on the Internet at address `cst8207.idallen.ca` (a public address). This address is visible anywhere on the Internet, giving you access to the machine without needing to use the Algonquin College VPN. Use it from home or off-campus. The CLS also has an on-campus private IP address `cst8207-alg.idallen.ca` usable only via the VPN or while on campus at Algonquin College. Use the private address when you are on-campus. Connecting from the Course Linux Server to other machines --------------------------------------------------------- Rarely, you might be logged in to the CLS and want to connect from the CLS (the local machine) to another machine (the remote machine) to transfer a file. Connecting from the local Course Linux Server out to a remote machine (e.g. using a command-line `ftp` or `ssh` command) requires that the remote machine have its own accessible IP address -- the remote machine should not itself be behind a firewall, unless you have arranged a pass-through port on that firewall. The remote machine must be running a file transfer server of some sort, to receive a connection from the CLS. This is not usually the case, so most file transfer is done by running the file transfer software on the local machine and connecting to the CLS as the remote machine. Locked out of the CLS --------------------- All file transfer programs need your CLS account name and CLS account password. If you fail to provide both accurately, you will be locked out of the CLS after several failed connection attempts. A common error is to fail to provide your account name to the file transfer program, so that it uses a blank name. Don't do that. Always make sure your CLS userid is being used. If your IP address gets locked out, follow the directions to [get your IP address unlocked]. Text File Line End Differences ------------------------------ Text files contain lines of text delimited (or terminated) by an end-of-line character (or characters). The line end character(s) in text files is not the same between Unix/Linux, Windows, and some versions of MacOS. - A text file written on Unix/Linux contains only a linefeed (`LF`) (sometimes called newline `NL`) character at the end of each text line. - A text file written on early MacOS contains only a carriage-return (`CR`) character at the end of each text line. - A text file written on DOS or Windows contains *both* a `CR` character *and* a `LF` character at the end of every line. Here is a Linux example using the `od` file dumper command to make the newline character visible: $ echo hi >foo $ wc foo 1 1 3 foo # third character is a NEWLINE (NL or LF) $ od -c foo # dump the file in character form 0000000 h i \n # \n means NEWLINE 0000003 The Unix/Linux command `file` can identify text files that have `CR` (Macintosh) or `CRLF` (DOS/Windows) line terminators: $ file macintosh unix windows macintosh: ASCII text, with CR line terminators unix: ASCII text windows: ASCII text, with CRLF line terminators Line-end differences may result in "staircasing" text (missing carriage returns) if you send a Unix/Linux text file to a Windows printer from some programs (e.g. Notepad). MacOS files may end up over-printed all on one line, due to the missing line feed characters! Print a small sample first, and on Windows try using *Write* or *Wordpad* to read or print a Unix/Linux file instead of the old *Notepad*. > Some file transfer programs may optionally translate line-endings if they > recognize that the file being transferred is a text file. Unix/Linux/OSX SCP and SFTP -- Secure Copy Program, Secure FTP ============================================================== > To copy files using Windows, scroll down to the [Microsoft Windows Users] > section. This section is for Unix/Linux/OSX and Cygwin users that already have the `scp` and `sftp` commands installed. The SCP and SFTP programs can be used to transfer files between Unix-like machines when a userid/password is required (e.g. to/from the [Course Linux Server]). Both these programs use the underlying SSH (Secure SHell) protocol that encrypts both your password and the data being transferred. Unix/Linux/OSX: Copy a file from a remote machine to the local machine ---------------------------------------------------------------------- Open a terminal window on your local machine. (Do *not* log in to the remote machine; the terminal commands given below must execute locally.) In the local terminal window, use SCP to copy a file from the remote machine to a file on the local machine, as follows. The first file name below is the file name on the remote machine; the second file name is the name it will be copied to (or use a name of dot `.` to keep the same name locally): Syntax: scp -p userid@remote.host.name:remote_file local_file Examples (use your own userid and your own remote host name): $ scp -p abcd0001@cst8207.idallen.ca:happy.txt happy.txt $ scp -p abcd0001@cst8207.idallen.ca:/etc/passwd . $ scp -p cst8207-alg.idallen.ca:/tmp/foo mydir/bar You will be prompted to enter your password for the remote machine. - The `-p` option preserves the modify time of the transferred file. - The `userid` part is your login userid on `remote.host.name`. You can leave off the `userid@` if your remote userid is the same as your userid on the local machine. - The `remote_file` may be an absolute pathname on `remote.host.name`, or it may be a pathname relative to your home directory on `remote.host.name`. - Using dot `.` for a local file will cause the local file name to be the same as the base name of the remote file. > If the remote machine is behind a firewall and requires a special port to > be used, the `-P` option (upper case `P`) must be used to set the firewall > pass-through port: (Use the actual port number for your firewall.) > > $ scp -P 2222 -p abcd0001@example.com:dodo.txt happy.txt Unix/Linux/OSX: Copy a file from the local machine to a remote machine ---------------------------------------------------------------------- To copy from local to remote via a Unix/Linux or Cygwin command line, just reverse the order of the arguments to SCP. If the remote name is a dot `.`, the remote file name will be the same as the base name of the local file. Syntax: scp -p local_file userid@remote.host.name:remote_file Examples (use your own userid and your own remote host name): $ scp -p happy.txt abcd0001@cst8207.idallen.ca:happy.txt $ scp -p /etc/passwd abcd0001@cst8207.idallen.ca:. $ scp -p mydir/bar cst8207-alg.idallen.ca:/tmp/foo To specify a special port number: $ scp -P 2222 -p happy.txt abcd0001@example.com:dodo.txt Unix/Linux/OSX: Using SFTP (includes Cygwin on Windows) ------------------------------------------------------- The SFTP program is a cover for SSH and SCP that makes it look like you are using the insecure FTP program; however, the actual connection and transfer is done using the secure SSH protocol. From a Unix/Linux (or Cygwin) command line, you can start SFTP like this: $ sftp abcd0001@cst8207.idallen.ca Connecting to cst8207.idallen.ca... abcd0001@cst8207.idallen.ca's password: sftp> help [... output similar to using insecure FTP ...] sftp> quit If you are familiar with insecure FTP (see below), SFTP will operate much the same way. As with insecure FTP, you can list the contents of remote directories and transfer files both ways (using `put` and `get`) on the same connection. Some versions of SFTP use `-P` to set the port number; others have an awkward way to specify the port number: $ sftp -oPort=2222 example.com Connecting to example.com... idallen@example.com's password: sftp> help [... output similar to using insecure FTP ...] sftp> quit Unix/Linux Insecure FTP -- File Transfer Protocol (do not use) ============================================================== > Avoid the standard insecure FTP program -- it sends passwords in clear text > across the Internet. (You are slightly safer using the FTP program locally > here at school, but realize that anyone snooping packets on your local > network will still see your password.) The Course Linux Server does not support insecure FTP, but you can use SFTP instead with many of the same command meanings. The old way to move files between machines was the insecure FTP (File Transfer Protocol) program. FTP is an insecure form of file transfer; because, any password you type into insecure FTP is visible across the network. Don't use insecure FTP for transfer between machines requiring userids and passwords over an insecure network (e.g. the Internet). If you log in to the Course Linux Server (using secure SSH), you can then use the insecure `ftp` or `lftp` commands on the Server to connect *out* from the Server to other remote machines (e.g. insecure FTP to your home computer or to ACADUNIX), if those other machines accept insecure FTP connections. If you set up your home computer with an insecure FTP server \[be careful!\], you may use the insecure `ftp` command on the Course Linux Server to connect to your home machine, if your home machine has a public IP address and isn't behind a firewall or NAT router. (Use SFTP instead.) Once you have an insecure FTP connection set up, you can copy files in either direction using the `put` and `get` commands, as you wish. Many Internet sites support a form of `anonymous` insecure FTP that lets you connect to a site without requring a password, using the special insecure FTP userid `anonymous` or `ftp`. Since it requires no password, this form of insecure FTP is safe to use over the Internet. It is how software is often provided for download to Unix/Linux users. Command-line insecure FTP is a *subsystem* kind of program with its own set of subcommands. Once inside the insecure FTP program, your prompt becomes `ftp>`. Inside insecure FTP, the `help` command will list the possible insecure FTP commands available, and `help commandname` will give you a bit more help on the given FTP command name. The Unix manual page for insecure FTP (`man ftp`) explains the individual insecure FTP subcommands in much better detail. Do not confuse Unix commands with insecure FTP subcommands. Pay attention to which program is prompting you for input. To quit FTP, type `quit`. The command-line insecure FTP program is also available under Windows. (You may need to install insecure FTP from the Windows CDROM.) The list of insecure FTP commands is slightly different; but, the basic commands (ls, cd, get, put) are the same as for Unix. (You can run command-line insecure FTP from a DOS window or using the *Run* dialog box.) Remember: FTP is not secure. Insecure FTP will not transfer entire directories; it may only be used to transfer files one at a time. (There are ways to get insecure FTP to fetch multiple files at once; but, the files must all be in the same directory; you can't fetch multiple directories. See the help for the `mget` insecure FTP subcommand.) Below is an example command-line insecure FTP session to a public FTP server. This insecure FTP command could be run on a Unix/Linux machine or under Windows. $ ftp ftp.gnu.org Connected to ftp.gnu.org. 220 GNU FTP server ready. 530 Please login with USER and PASS. Name (ftp.gnu.org:idallen): anonymous # NOTE: special userid used 230 Login successful. # NOTE: no password needed! Remote system type is UNIX. Using binary mode to transfer files. ftp> help [... many lines of FTP commands show here ...] ftp> help ls ls list contents of remote directory ftp> help get get receive file ftp> help cd cd change remote working directory ftp> ls 150 Here comes the directory listing. [... many files show here ...] ftp> cd gnu ftp> ls 150 Here comes the directory listing. [... many files show here ...] ftp> cd chess ftp> ls 150 Here comes the directory listing. [... many files show here ...] ftp> get README.gnuchess local: README.gnuchess remote: README.gnuchess 150 Opening BINARY mode data connection for README.gnuchess (89 bytes). 226 File send OK. 89 bytes received in 0.0066 seconds (13 Kbytes/s) ftp> quit 221 Goodbye. $ ls -l README.gnuchess -rw-r--r-- 1 idallen idallen 89 Oct 2 00:49 README.gnuchess FTP Binary Mode vs. Text Mode file transfer ------------------------------------------- Insecure FTP can transfer files in either *text* or *binary* mode. Almost always use *binary* mode, which makes an exact copy of the file. *Text* mode can be used to translate line ends when copying plain text files between dissimilar systems, e.g. between Unix and Windows, but the line-end translation will corrupt all other non-text files (i.e. images sent in text mode will be corrupted). Insecure FTP vs. the Unix Shells -------------------------------- The syntax of insecure FTP commands is not the same as the syntax of Unix commands. This insecure FTP command doesn't do what you think it does: ftp> ls -l filename output to local-file: filename? If you answer `yes` to this prompt, you will copy the output of `ls -l` into the file `filename` in your current directory, erasing what was there before. This is probably not what you want. Don't do it. The insecure FTP command names resemble Unix command names; but, they are *not* Unix commands. The syntax is different. You are not typing into a shell, you are typing into the insecure FTP program. Be careful. Also, don't type insecure FTP commands into Unix and expect that they will work, e.g. the BASH shell doesn't understand `put filename`. Microsoft Windows Users ======================= MS Windows does not ship with any secure file transfer programs such as SCP or SFTP, even though versions exist that are open source and free software. You can buy expensive commercial versions of SSH/SCP/SFTP for Windows from various vendors; or, you can download and install some free (source code available) programs: - [FileZilla]: a multi-platform GUI SFTP, FTP and SCP client. - [WinSCP]: a Windows GUI SFTP, FTP and SCP client. - [PuTTY]: a program suite that contains Windows command-line (DOS window) programs that work like SSH, SCP, and SFTP. The FileZilla GUI Client ------------------------ [FileZilla] is a free multi-platform graphical SFTP, FTP, and SCP client. This client is already installed on the Windows computers in the Algonquin T126 labs. In the FileZilla **Quickconnect** bar, enter the Public (or Private) address `sftp://cst8207.idallen.ca` in the **Host:** box and your Algonquin userid in the **Username:** box to connect to the Course Linux Server. Accept the remote SSH host key when prompted to do so. Upon successful connection, the left side of the window becomes the **Local site:** and the right side of the screen becomes the **Remote site:**. Highly recommended for sysadmin: Under the Transfer menu, turn on *Preserve timestamps of transferred files*. You can now drag items from Local to Remote or vice-versa. You can right-click on items for other options. The WinSCP GUI Client --------------------- [WinSCP] is a free graphical SCP/SFTP client for Windows. It is one of the easiest ways to copy files from the Course Linux Server, but it cannot be automated or scripted. For use in scripts, see [the PuTTY suite of programs]. > Here is a brief 2m 52s [Lynda.com tutorial on using PuTTY and WinSCP]. (You > need a free Lynda.com account to see this tutorial.) This tutorial does > *not* set the full set of required PuTTY options! **WinSCP** has two GUI interface modes: **Commander** and **Explorer**. Use the default drag-and-drop **Explorer** interface if you are not familiar with the two-pane format of *Norton Commander*. Use the **Commander** interface if you want to move files quickly using mouse-free keyboard shortcuts. ![WinSCP Connection] One you log in to the CLS with WinSCP, you can drag items from the CLS on the right to your local machine on the left. The "Refresh" button refreshes the CLS pane on the right to show new files. When transferring files from the CLS to your local machine and then up to Brightspace, make sure the *Transfer Type* is set to *Binary*. (This is the default.) Do *not* transfer the files from the CLS to your local machine and then up to Brightspace using the *Text* Transfer Type. ### Transfer Type -- use Binary not Text **Warning**: WinSCP will optionally translate files between Unix/Linux/Windows [Text File Line-End Differences] if the file name is a recognized text file, e.g. the name ends in `.txt`, and you select a *Text* Transfer Type. If you transfer a file from Linux to Windows using the *Text* Transfer Type, the result will be a Windows/DOS format text file, not a Linux format text file. Do *not* change Linux text files from the CLS into Windws/DOS text files as you transfer files from the CLS to Brightspace; Windows/DOS files will not be accepted! ### Editing files using the WinSCP GUI WinSCP lets you text-edit files directly from the GUI, but you won't learn any Linux editing skills if you do that. Be careful that you don't make edits to files that you later overwrite with older versions by doing a file transfer. ### WinSCP Error due to: too large SFTP packet If you get a WinSCP Error similar to this "Received too large SFTP packet": ![WinSCP Error Message] go to your `.bashrc` file and insert this exact line at the beginning: [ -z "${PS1-}" ] && return Your `.bashrc` and `.bash_profile` files must not produce any output when called in non-interactive mode, otherwise the output confuses the SFTP protocol used by WinSCP. The PuTTY suite of programs (PSCP, PSFTP) ----------------------------------------- [PuTTY] is a graphical telnet/SSH client and a suite of command-line (DOS window) file transfer clients. > Here is a brief 2m 52s [Lynda.com tutorial on using PuTTY and WinSCP]. (You > need a free Lynda.com account to see this tutorial.) This tutorial does > *not* set the full set of required PuTTY options! If you download and install the full PuTTY program suite under Windows (PuTTY comes with an executable auto-installer that will do this for you), you will find the programs PSCP and PSFTP under the installation directory (usually under `C:\Program Files\PuTTY`). Start up a DOS command prompt, change to this directory, and run the secure commands you need to copy files to/from other systems. See below for examples of how to do this. Unless you change your DOS search PATH, you will only be able to execute the PSCP and PSFTP commands from the directory into which you downloaded them. When transferring files between Windows and Unix/Linux machines, remember that pathnames on the Windows machines contain backward slashes while pathnames on Unix/Linux machines contain forward slashes. For example, you might find yourself typing something like this: psftp> put "d:\folder\myfile.txt" "public_html/dir/page.txt" local:d:\dir\myfile.txt => remote:/home/abcd0001/public_html/page.txt Windows pathnames contain backslashes and the Unix/Linux pathnames contain forward slashes. Some versions of PSFTP also accept forward slashes for Windows pathnames. You must surround the pathnames with double quotes if the pathnames contain blanks. ### PSCP **PSCP** is a command-line copy program, similar to the Unix/Linux SCP program. If the remote machine is behind a firewall and requires a gateway and special port to be used, the -P option (upper case P) must be used to set the firewall pass-through port. Leave it out otherwise. The backslash at the end of a line below indicates that the line continues. Type what is written all on one line without the backslash. C:\> cd "C:\Program Files\PuTTY" C:\Program Files\PuTTY> pscp -h ...the -h displays a short help listing here... C:\Program Files\PuTTY> \ pscp -P 2222 "abcd0001@example.com:dir/foo.txt" "folder\bar.txt" abcd0001@example.com's password: ...only use the -P option if you need the special port number... ...you may be asked to accept the host key here (say yes)... ...file transfers remote "dir/foo.txt" to local "folder\bar.txt"... - You must use your own userid on the remote machine. You must replace "`example.com`" with the machine name or IP address to which you wish to connect. The `-P` option sets a non-standard port number, if you need it. - Windows pathnames should contain backslashes and the Unix/Linux pathnames contain forward slashes. Some versions of PSCP accept forward slashes for Windows pathnames. You must surround the pathnames with double quotes if the pathnames contain blanks. ### PSFTP **PSFTP** is a secure command-line FTP-like program, similar to standard FTP. If the remote machine is behind a firewall and requires a special port to be used, the -P option (upper case P) must be used to set the firewall pass-through port. Leave it out otherwise. C:\> cd "C:\Program Files\PuTTY" C:\Program Files\PuTTY> psftp -h ...the -h displays a short help listing here... C:\Program Files\PuTTY> psftp -P 2222 abcd0001@example.com ...only use the -P option if you need the special port number... ...you may be asked to accept the host key here (say yes)... abcd0001@example.com's password: Remote working directory is /home/abcd0001 psftp> help ...short help listing displays here... psftp> ls ...listing of directory displays here... psftp> get ".bashrc" "foo.txt" remote:/home/abcd0001/.bashrc => local:foo.txt psftp> put "d:\dir\myfile.txt" "public_html/page.txt" local:d:\dir\myfile.txt => remote:/home/abcd0001/public_html/page.txt psftp> quit ...file "foo.txt" is now in the current directory... - You must use your own userid on the remote machine. You must replace `example.com` with the machine name or IP address to which you wish to connect. The `-P` option sets a non-standard port number, if you need it. - Windows pathnames should contain backslashes and the Unix/Linux pathnames contain forward slashes. Some versions of PSFTP accept forward slashes for Windows pathnames. You must surround the pathnames with double quotes if the pathnames contain blanks. - The options to the PuTTY Windows version of SFTP (named PSFTP) are not the same as the options to the Unix/Linux version of SFTP. In particular, the option `-P` has different meanings! ### Example SCP and SFTP Windows Command Lines First, here are some typical PSCP command lines for file transfer from a local (Windows) computer to the `public_html` directory of the `abcd0001` account on the Course Linux Server. The first line uses the Public gateway and special port 2222; the second uses the Private IP address (via the VPN or On-Campus): pscp -P 2222 d:\dir\image.jpg abcd0001@cst8281.idallen.ca:public_html/a10/image.jpg pscp d:\dir\image.jpg abcd0001@10.50.254.148:public_html/a10/image.jpg Second, here is the same transfer using the PSFTP command instead of PSCP: psftp -P 2222 abcd0001@cst8281.idallen.ca password: psftp> put d:\dir\image.jpg public_html/a10/image.jpg psftp> help psftp abcd0001@10.50.254.148 password: psftp> put d:\dir\image.jpg public_html/a10/image.jpg psftp> help 1. You must replace abcd0001 with your own userid. You will be asked for your password on the Course Linux Server. If you are asked to accept the server encryption key, say `yes`. 2. You must remember to insert the web directory name `public_html` into all your file names for the Course Linux Server, since that is where the web server looks in your account. Files put into your home directory will not be visible on the Web. 3. Slashes go backwards for Windows pathnames and forwards for Unix pathnames. 4. The *psftp* and *pscp* commands may not be in your Windows DOS search PATH. You can add the directory containing these commands to your DOS search PATH, or you can change to the directory containing these commands when you want to run them, or you can type the absolute path of the command names if you aren't in the right directory. Windows Insecure FTP (do not use) --------------------------------- The Course Linux Server does not support insecure FTP. MS Windows has a command-line version of insecure FTP available from a DOS prompt or in a DOS window. You can also download various graphical insecure FTP clients. Many recommend the insecure programs *FileZilla* or *WS_FTP*. ### Using Windows GUI via Windows Explorer Some versions of Windows also let you use an insecure FTP URI to connect to an insecure FTP server and log in and transfer files, e.g. using this form of URI: ftp://ftp.algonquincollege.com/ The Windows Explorer (not Internet Explorer!) will let you open an insecure FTP URI such as the one above and drag-and-drop files between your machine and a remote machine graphically. Underneath, Windows is using the insecure FTP protocol and your data and passwords are visible to anyone who can snoop your network connection. Don't use this. 1. You must log in with your own Linux userid. You will be asked for your FTP password. 2. Your password and data are not encrypted when you use insecure FTP. Do not use this method on an untrusted network (i.e. Internet). 3. You may be able to access a file via FTP that cannot be displayed in a web browser, since the FTP program is logged in as your account name and the web browser accesses your files as *other*. You must ensure that your files have read permissions for *other* after you transfer them to the Course Linux Server. 4. Windows Explorer, using insecure FTP, may create directories and files with the wrong Linux permissions. Directories under your `public_html` directory must be readable and searchable (not writable!) by *other*. Files under your `public_html` directory must also be readable (not writable or executable!) by others. Inaccessible files and directories will generate `Permission Denied` errors in your web browser. Files and directories with unwanted *write* permissions will allow other users to delete or erase your web pages. File Transfer Hacks =================== Here are some ways to move files around without using a file transfer program: Use EMail for text files ------------------------ If the file you want to take home from Unix is a text file (not an image), you can usually EMail it to yourself somewhere using a command-line Unix EMail program with standard input redirected to come from the file you want. Unix mail programs that work this way are `mutt`, `Mail`, `mailx`, and `mail`. For example: $ mail user@example.com <.bashrc $ mail abcd0001@algonquincollege.com <.bashrc You can only send **text files** this way, and you can only send one file per mail message using file input redirection (unless you concatenate many files together first). See `man mail` for further details. To send binary programs via email you must encode them as ASCII first and decode them after receiving them. See `man uuencode`. -- | Ian! D. Allen, BA, MMath - idallen@idallen.ca - Ottawa, Ontario, Canada | Home Page: http://idallen.com/ Contact Improv: http://contactimprov.ca/ | College professor (Free/Libre GNU+Linux) at: http://teaching.idallen.com/ | Defend digital freedom: http://eff.org/ and have fun: http://fools.ca/ [Plain Text] -- plain text version of this page in [Pandoc Markdown] format [www.idallen.com]: http://www.idallen.com/ [Course Home Page]: .. [Course Outline]: course_outline.pdf [All Weeks]: indexcgi.cgi [Plain Text]: 015_file_transfer.txt [Course Linux Server]: 070_course_linux_server.html [get your IP address unlocked]: 070_course_linux_server.html#getting-locked-out-of-the-server [Microsoft Windows Users]: #microsoft-windows-users [FileZilla]: http://filezilla-project.org/ [WinSCP]: http://winscp.net/eng/ [PuTTY]: http://www.chiark.greenend.org.uk/~sgtatham/putty/ [the PuTTY suite of programs]: #the-putty-suite-of-programs-pscp-psftp [Lynda.com tutorial on using PuTTY and WinSCP]: https://www.lynda.com/Linux-tutorials/Access-Linux-CLI-from-Windows/517442/570824-4.html [WinSCP Connection]: data/winscp.png "WinSCP" [Text File Line-End Differences]: #text-file-line-end-differences [WinSCP Error Message]: data/winscperror.png "WinSCP" [Pandoc Markdown]: http://johnmacfarlane.net/pandoc/