------------------------- Week 14 Notes for CST8165 ------------------------- -Ian! D. Allen - idallen@idallen.ca - www.idallen.com Remember - knowing how to find out an answer is more important than memorizing the answer. Learn to fish! RTFM! (Read The Fine Manual) Keep up on your readings (Course Outline: average 4 hours/week homework) ------------------------------------------------------------------------- Linux Lab work (only works with on-campus/VPN access to 10.50.254.148): http://tools.ietf.org/html/rfc1939 (23 pages) 1. Send email to abcd0001@localhost.localdomain via SMTP server 10.50.254.148 where abcd0001 is replaced by your Algonquin student userid. - this SMTP server is liberal in accepting LF line ends! - you can make up any envelope From address you like - you can make up any message To/From addresses you like - you can also send email this way to your classmates (be polite) See Notes: smtp_session.txt * $ nc -v 10.50.254.148 25 Connection to 10.50.254.148 25 port [tcp/smtp] succeeded! 220 idallen-alinux ESMTP Postfix (Ubuntu) * EHLO ... see the sample session in smtp_session.txt ... ... etc ... * QUIT 221 Bye $ 2. Fetch and delete the email using "nc" to the POP3 TCP port. See RFC Section 10: Example POP3 Session - this POP3 server is liberal in accepting LF line ends! - login with your Algonquin userid using USER and PASS - your password is the letter C followed by the last 7 digits of your Algonquin student number - you must login to the server with ssh first, since Dovecot will not allow unencrypted POP3 access from off-machine * $ ssh abcd0001@10.50.254.148 * $ nc -v localhost 110 Connection to localhost 110 port [tcp/pop3] succeeded! +OK Dovecot ready. * USER abcd0001 * PASS C1234567 +OK Logged in. ... etc ... * QUIT +OK Logging out. ------------------------------------------------------------------------- File Transfer Protocol - FTP ---------------------------- http://tools.ietf.org/html/rfc114 (Apr 1971, 17 pages, no index) http://tools.ietf.org/html/rfc454 (Feb 1973, 35 pages, no index) http://tools.ietf.org/html/rfc686 (May 1975 "leaving well enough alone") http://tools.ietf.org/html/rfc691 (May 1975 "one more try on the FTP") http://tools.ietf.org/html/rfc765 (Jun 1980, 70 pages, no index!) http://tools.ietf.org/html/rfc959 (Oct 1985, 69 pages, no index!) - another text-based protcol (you can run it with "nc") - commands are not case-sensitive - but it's tricky to handle the parallel data connection at the same time... "The objectives of FTP are 1) to promote sharing of files (computer programs and/or data), 2) to encourage indirect or implicit (via programs) use of remote computers, 3) to shield a user from variations in file storage systems among hosts, and 4) to transfer data reliably and efficiently. FTP, though usable directly by a user at a terminal, is designed mainly for use by programs. Q: Give two (of four) objectives of the FTP protocol. Q: T/F, FTP was designed for humans to use directly (e.g. via netcat). 2.3. THE FTP MODEL (p.8) - nice ASCII art diagram PI - protocol interpreter DTP - data transfer process - two types of FTP: Active FTP and Passive FTP (see below) http://en.wikipedia.org/wiki/File_Transfer_Protocol "FTP runs exclusively over TCP. The FTP server always listens on port 21 for incoming connections from FTP clients. A connection to this port from the FTP Client forms the control stream on which commands are passed to the FTP server from the FTP client and on occasion from the FTP server to the FTP client. For the actual file transfer to take place, a different connection is required which is called the data stream. Depending on the transfer mode, the process of setting up the data stream is different. 1. In active mode, the FTP client opens a random port (> 1023), sends the FTP server the random port number on which it is listening over the control stream and waits for a connection from the FTP server. When the FTP server initiates the data connection to the FTP client it binds the source port to port 20 on the FTP server. [So, the client initiates the initial control connection to the server and the server "calls back" the client to transfer data.] 2. In passive mode [the client first tells the server to use passive mode using the FTP "PASV" command, then] the FTP Server opens a random port (> 1023), sends the FTP client the port on which it is listening over the control stream and waits for a connection from the FTP client. In this case the FTP client binds the source port of the connection to a random port greater than 1023. [So, the client initiates the initial control connection to the server and also initiates the data connection to the server.] Summary: - Active FTP: client opens a port and tells the server to connect to it. - client control connects to server; server data connects back to client - server binds outgoing data stream to privileged data port 20 - client needs to open a port in the firewall to let in the server - Passive FTP: client tells server to use passive mode using "PASV" command; server opens a port and tells the client to connect to it. - client control connects to server; client data connects to server - both ends of data stream use unprivileged port numbers - no incoming port needs to be opened in a firewall - but the server's firewall needs a random port opened to permit the client to connect to the data stream Q: Describe how FTP works (including port numbers), and including a description of the difference between FTP "active" and "passive" modes. http://slacksite.com/other/ftp.html "One of the most commonly seen questions when dealing with firewalls and other Internet connectivity issues is the difference between active and passive FTP and how best to support either or both of them. Hopefully the following text will help to clear up some of the confusion over how to support FTP in a firewalled environment. - The diagram on this site is wrong: no [4] is sent from client - FTP uses the TELNET protocol on the control connection Q: What protocol does FTP use for its control connection? - control channel can extend across multiple servers (Fig. 2 p.9) http://en.wikipedia.org/wiki/FXP Q: T/F, the FTP control channel can chain across multiple servers. Q: T/F, an FTP client can arrange to transfer data from one remote server to another without routing this data through the client's connection. - FTP servers can be told to open ports on *any* machine! http://en.wikipedia.org/wiki/FTP_bounce_attack http://www.cert.org/tech_tips/ftp_port_attacks.html "To conform with the FTP protocol, the PORT command has the originating machine specify an arbitrary destination machine and port for the data connection. However, this behavior also means that an attacker can open a connection to a port of the attacker's choosing on a machine that may not be the originating client. Making this connection to an arbitrary machine for unauthorized purposes is the FTP bounce attack. Q: What command in the FTP protocol set permits an "FTP bounce attack", and how is that command used to do the attack? - The FTP protocol commands only have one argument! - a rename requires RNFR followed by RNTO (silly) Q: How do you implement a "rename" using the FTP command set? - server responses are 3-digit codes followed by message text - client "should wait" for server reply before issuing next command (5.4 p.49) Q: T/F, FTP is a pipelining/streaming protocol - you can send many commands without waiting for server responses in between. - multi-line server responses use the SMTP '-' trick, e.g. 220-message for the first line, and code followed by space (e.g. 220 end) for the last line of the multi-line response - but, unlike SMTP, the intervening lines after the first status line may look like anything (e.g. output from "STAT")! - if a server response generates something that looks like it might prematurely end the response, the line must be prefaced with "some neutral character" (e.g. a space) Q: How does the FTP server produce a multi-line response to a client? - FTP reply codes are structured (Section 4.2 p.36) 1yz Positive Preliminary reply 2yz Positive Completion reply 3yz Positive Intermediate reply 4yz Transient Negative Completion reply 5yz Permanent Negative Completion reply x0z Syntax x1z Information x2z Connections x3z Authentication and accounting x4z Unspecified x5z File system Q: T/F, FTP 3-digit response codes have assigned meanings for both the first and second digit. [do not memorize these codes!] - command response sequences (5.4 p.49) - state diagrams (6 p.54) - see RETR, p.51 125, 150 (110) 226, 250 425, 426, 451, 551, 552 Using netcat on the control connection -------------------------------------- Because FTP has a separate data connection, you would need two netcat sessions going to transfer a file, and you would have to get the port numbers right. The FTP control connection is essentially the same as Telnet - it uses the same protocol. Q: Why does FTP need two parallel netcat sessions to run, when SMTP, POP3, HTTP, and TELNET only need one netcat? Note: The FTP protocol STAT command does the same as LIST; but, it receives the server info over the control connection instead of over the data connection; this makes it useful on a (single) netcat connection. Some basic directory listing is possible with STAT, even without a data connection; you can even STAT a directory (such as "."). Q: What is the difference between the FTP "STAT" and "LIST" commands? Q: T/F, since the FTP "LIST" returns data via the data connection, there is no way to use FTP to see a directory listing of a remote site without opening a data connection first. Q: T/F, the FTP "STAT" command works even if you have a firewall that prevents FTP data channels from being set up Using "STAT" on the control connection: $ nc -v ftp.idallen.ca 21 ftp.idallen.ca [72.18.159.15] 21 (ftp) open 220---------- Welcome to Pure-FTPd [TLS] ---------- 220-You are user number 3 of 50 allowed. 220-Local time is now 20:46. Server port: 21. 220-This is a private system - No anonymous login 220-IPv6 connections are also welcome on this server. 220 You will be disconnected after 15 minutes of inactivity. ---> HELP 214-The following SITE commands are recognized ALIAS CHMOD IDLE 214 Pure-FTPd - http://pureftpd.org/ ---> USER idallen 331 User idallen OK. Password required ---> PASS XXXX 230-User idallen has group access to: idallen 230 OK. Current restricted directory is / ---> STAT .twmrc 213-STAT -rw-r--r-- 1 32096 idallen 14706 Jun 21 2002 .twmrc 213 End. ---> NOOP 200 Zzz... ---> QUIT 221-Goodbye. You uploaded 0 and downloaded 0 kbytes. 221 Logout. or 421 Timeout - try typing a little faster next time Active FTP session with server responses ---------------------------------------- Section 7, p.59 Many FTP commands have options that let you see the server/client dialog on the control connection. Below is a sample "Active" FTP session. Note the encoding of the 16-bit PORT into two 8-bit integers. The high byte is sent first, followed by the low byte, - e.g. 164,39 means 164*256 + 39 = 39424 Q: An FTP server sends you this PORT string: 1,10 What PORT number is that? (1*256 + 10 = 266) Q: What is wrong with this FTP PORT string: 1,300 $ ftp -vd ftp.idallen.ca Connected to ftp.idallen.ca. 220---------- Welcome to Pure-FTPd [TLS] ---------- 220-You are user number 1 of 50 allowed. 220-Local time is now 23:13. Server port: 21. 220-This is a private system - No anonymous login 220-IPv6 connections are also welcome on this server. 220 You will be disconnected after 15 minutes of inactivity. Name (ftp.idallen.ca:idallen): idallen ---> USER idallen 331 User idallen OK. Password required Password: ---> PASS XXXX 230-User idallen has group access to: idallen 230 OK. Current restricted directory is / ---> SYST 215 UNIX Type: L8 Remote system type is UNIX. Using binary mode to transfer files. ftp> ls i.sh ---> PORT 192,168,1,250,164,39 # 164*256 + 39 = 39424 200 PORT command successful ---> LIST i.sh 150 Connecting to port 39424 -rwxr-xr-x 1 32096 idallena 1737 Oct 19 2005 i.sh 226-Options: -a -l 226 1 matches total ftp> get i.sh local: i.sh remote: i.sh ---> TYPE I 200 TYPE is now 8-bit binary ---> PORT 192,168,1,250,172,40 # 172*256 + 40 = 44072 200 PORT command successful ---> RETR i.sh 150 Connecting to port 44072 226-File successfully transferred 226 0.132 seconds (measured here), 12.86 Kbytes per second 1737 bytes received in 0.15 secs (11.5 kB/s) ftp> quit ---> QUIT 221-Goodbye. You uploaded 0 and downloaded 2 kbytes. 221 Logout. In an Active FTP session, the client sends the IP address and port to the server and the server initiates a data stream connection back to the client on that address and port. (Usually the address and PORT are on the client machine; but, they could be on any machine.) Q: My FTP client wants to set up an Active session with a remote server and have the server connect to IP address 1.2.3.4 and port 513. What FTP command does the client send to the server? (PORT 1,2,3,4,2,1) Passive FTP session with server responses ----------------------------------------- Same as above, but in PASV mode the client tells the server to use PASV mode where the server then tells the client what port to connect to. The client is responsible for making both the initial outgoing control connection and then the outgoing data connection to the server: $ ftp -vd -p ftp.idallen.ca Connected to ftp.idallen.ca. 220---------- Welcome to Pure-FTPd [TLS] ---------- 220-You are user number 1 of 50 allowed. 220-Local time is now 23:24. Server port: 21. 220-This is a private system - No anonymous login 220-IPv6 connections are also welcome on this server. 220 You will be disconnected after 15 minutes of inactivity. Name (ftp.idallen.ca:idallen): idallen ---> USER idallen 331 User idallen OK. Password required Password: ---> PASS XXXX 230-User idallen has group access to: idallen 230 OK. Current restricted directory is / ---> SYST 215 UNIX Type: L8 Remote system type is UNIX. Using binary mode to transfer files. ftp> get i.sh local: i.sh remote: i.sh ---> TYPE I 200 TYPE is now 8-bit binary ---> PASV 227 Entering Passive Mode (72,18,159,15,164,151) # 164*256 + 151 = 42135 ---> RETR i.sh 150 Accepted data connection 226-File successfully transferred 226 0.000 seconds (measured here), 24.03 Mbytes per second 1737 bytes received in 0.02 secs (85.7 kB/s) ftp> ls i.sh ---> TYPE A 200 TYPE is now ASCII ---> PASV 227 Entering Passive Mode (72,18,159,15,218,170) # 218*256 + 170 = 55978 ---> LIST i.sh 150 Accepted data connection -rwxr-xr-x 1 32096 idallen 1737 Oct 19 2005 i.sh 226-Options: -a -l 226 1 matches total ftp> quit ---> QUIT 221-Goodbye. You uploaded 0 and downloaded 2 kbytes. 221 Logout. Q: My FTP client wants to set up a Passive session with a remote server. What command to the server triggers a Passive session? Q: What response does an FTP server give to a PASV command, and what do the six numeric arguments in the response mean? Q: A server responds to PASV from a client with (1,2,3,4,2,3). What IP address will the client connect to? (1.2.3.4) What port will the client connect to at that IP address? (515) =============================================================================== Network Address Translation (NAT) --------------------------------- http://tools.ietf.org/html/rfc2663 (Aug 1999, 30 pages, "informational") http://tools.ietf.org/html/rfc3022 (Jan 2001, 16 pages, "informational") This is how your home router works. "Basic NAT and NAPT are two variations of traditional NAT, in that translation in Basic NAT is limited to IP addresses alone, whereas translation in NAPT is extended to include IP address and Transport identifier (such as TCP/UDP port or ICMP query ID). NAT and NAPT allow machines on a private, internal network to send their packets through a NAT router. Network Address Translation modifies the *source* address (and with NAPT, possibly the source port) of TCP or UDP packets leaving the router for the Internet, and it modifies the *destination* address (and possibly the destination port) of reply packets returning to the router from the Internet. The saved source address/port is placed into the destination address/port of the reply packets when they return. NAT/NAPT are inherently "man-in-the-middle" attacks, made possible by early TCP design decisions not to encrypt parts of the protocol. Example of NAT: Consider a Client on an internal network (e.g. 192.168.0.0/24) connecting through a single Router to a remote Server. The Router has the only public Internet IP address (e.g. given by your ISP). There are four IP addresses involved in the connection, A,B,C,D: A. client - internal network - e.g. 192.168.0.2 B. router - internal network interface: e.g. 192.168.0.1 C. router - external network interface: e.g. 206.47.37.39 D. server - public Internet address - e.g. 72.18.159.15 INTERNAL CLIENT NAT ROUTER INTERNET SERVER --------------- ------------- --------------- 1. a client src=A generates a packet destined for server dst=D [src=A,dst=D] --> 2. the packet [A,D] heads to the router B interface on its way to the Internet --> [src=A,dst=D] 3. the router appplies NAT to src=A, saving and replacing src=A with src=C [src=C,dst=D] --> 4. the new packet [C,D] exits C and continues to the server D via the Internet --> [src=C,dst=D] 5. the packet arrives at D; server generates a reply packet from src=D to dst=C <-- [src=D,dst=C] 6. the reply packet [D,C] returns to the router C interface via the Internet [src=D,dst=C] <-- 7. the router appplies un-NAT to dst=C, replacing dst=C with saved dst=A <-- [src=D,dst=A] 8. the reply packet [D,A] exits B and continues to the internal client dst=A [src=D,dst=A] <-- 1. Client A creates a packet destined for server D with [src=A, dst=D] 2. Client routing table says to send packet via router gateway B 3. Packet arrives at router internal interface B on the way to D: - Router applies NAT, removing and remembering the internal src address A and changing the src address A to be its own external interface address C in the IP packet header: [A,D] becomes [C,D] - Router saves the src address A in a table for later use (Every NAT connection needs its own separate saved address!) 4. Packet leaves router external interface C with new [src=C, dst=D] Packet traverses Internet and arrives at server dst D. 5. Server constructs a reply packet with [src=D, dst=C] This reply packet is destined for the router's public Internet address. 6. Reply packet traverses Internet and arrives at router interface C: 7. Router recognizes that this packet is a reply to the earlier packet for which the router saved the src address A. The router takes the saved src address A out of its table and puts it back into the dst address of the reply packet: [D,C] becomes [D,A] 8. Reply packet leaves router internal interface B with new [src=D, dst=A] Reply packet arrives at internal client interface A. Round trip complete. - Neither the client nor the server have any idea that the packets were changed by the router. - The server thinks it is talking directly to the router (not the client). - The remote server has no idea that the *actual* source of the packets is the client behind the router. - The *data* portion of packets is untouched by plain NAT/NAPT. Only the TCP/IP headers are altered by the router, not the data. - TCP checksums also have to be fixed! (pseudo-header) - IP packet fragments cannot be translated (missing header info) - in "NAPT" (with Port Translation), the router also saves and changes the TCP/UDP port numbers on packets leaving the router for the Internet, and the router replaces the saved port numbers (as well as replacing the saved IP addresses) as the reply packets return - NAT provides some security against unexpected incoming TCP/UDP packets. - your private net is hidden behind the router - mappings from Internet to private net cannot be triggered from the Internet side of the router; the internal net client must initiate the first packet through the NAT process in the router - NAT can also map ICMP packets to/from an external network Q: How does NAT work? Describe (with pictures) the changes made to a packet as it leaves a client, traverses a NAT-ing router to a server, and then the reply packet returns from the server to the router to the client. Show the source and destination addresses at each interface. Q: T/F, a router's NAT translation table can be initialized either by a packet going from internal net to external, or by a packet going from external net to internal http://tools.ietf.org/html/rfc2663 (considerations, Aug 1999) "NAT devices are application unaware in that the translations are limited to IP/TCP/UDP/ICMP headers and ICMP error messages only. NAT devices do not change the payload of the packets, as payloads tend to be application specific. Q: T/F, a basic NAT device changes IP addresses only in the IP header, not in the data portion of a packet. "NAT devices (without the inclusion of Application Level Gateways [ALGs]) do not examine or modify transport payload. For this reason, NAT devices are transparent to applications in many cases. There are two areas, however, where NAT devices often cause difficulties: 1) when an application payload includes an IP address, and 2) when end-to-end security is needed. Q: T/F, if an application protocol puts a local IP address into the data portion of a packet, a plain NAT router (no ALG) won't modify it; so, the protocol may not work (because the local IP address won't be translated to work on the Internet) On designing new protocols: http://tools.ietf.org/html/rfc3235 (Jan 2002, 13 pages, "informational") "This document discusses those things that application designers might wish to consider when designing new protocols. While many common Internet applications will operate cleanly in the presence of Network Address Translators, others suffer from a variety of problems when crossing these devices. Q: Why is it a Very Bad Idea for a protcol to put IP information in the data portion of a packet (as FTP does with its PORT command)? Active FTP through NAT using a smart router ------------------------------------------- http://tools.ietf.org/html/rfc2616 - section 7.1 p.24 "7.1. FTP support" Normally (without an Application Level Gateway [ALG]), NAT doesn't change anything in the *data* in each packet. For active FTP, which sends a local FTP client IP address to the remote server in the data part of the PORT command packet, plain NAT isn't clever enough to fix the IP address hidden in the FTP PORT command packet's data. The server gets the local client internal IP address from the PORT command and is unable to use it to contact the client; because, the IP address in the data packet is the client's *internal* network address, not the Internet-visible address of its NAT router. To fix this, better routers have "application specific gateways [ALGs]" built in that reach into the *data portion* of the FTP PORT packet and modify it, replacing the client internal address in the PORT command with the public external address of the router (and fixing all the TCP/IP checksums and TCP window sizes as well!). - note that this sort of ALG intervention requires unencrypted data packets! http://tools.ietf.org/html/rfc2616 "[The] "PORT" command and "PASV" response in FTP control session payload identify the IP address and TCP port that must be used for the data session it supports. The arguments to the PORT command and PASV response are an IP address and a TCP port in ASCII. An FTP ALG is required to monitor and update the FTP control session payload so that information contained in the payload is relevant to end nodes. The ALG must also update NAT with appropriate data session tuples and session orientation so that NAT could set up state information for the FTP data sessions. "Because the address and TCP port are encoded in ASCII, this may result in a change in the size of packet. For instance, 10,18,177,42,64,87 is 18 ASCII characters, whereas 193,45,228,137,64,87 is 20 ASCII characters. If the new size is same as the previous, only the TCP checksum needs adjustment as a result of change of data. If the new size is less than or greater than the previous, TCP sequence numbers must also be changed to reflect the change in length of FTP control data portion. A special table may be used by the ALG to correct the TCP sequence and acknowledge numbers. The sequence number and acknowledgement correction will need to be performed on all future packet of the connection. Q: T/F, an ALG in a router may need to change the size of an FTP packet to fully implement NAT. Example of NAT through a router with an FTP ALG: Below are two ethereal TCP stream dumps. The first dump shows the TCP stream as seen on the local net side of the router (the side connected to the FTP client). The second dump shows the same TCP stream as seen on the external Internet side of the router. Note that the ALG in the router has changed (and remembered!) the IP address in the PORT command. The ALG in a smart router has to notice the FTP traffic and reach into the data portion of the control stream TCP packet to replace the actual PORT IP address 192.168.1.250 with the Internet address 206.47.37.39 (and then fix all the TCP window and checksum information!). Dump 1: This TCP stream trace is taken on the client side of the router. Traffic leaving host 192.168.1.250 heading to router (from ethereal): 220---------- Welcome to Pure-FTPd [TLS] ---------- 220-You are user number 1 of 50 allowed. 220-Local time is now 23:34. Server port: 21. 220-This is a private system - No anonymous login 220-IPv6 connections are also welcome on this server. 220 You will be disconnected after 15 minutes of inactivity. ---> USER idallena 331 User idallena OK. Password required ---> PASS XXXX 230-User idallena has group access to: idallena 230 OK. Current restricted directory is / ---> SYST 215 UNIX Type: L8 ---> PORT 192,168,1,250,181,11 # 181*256 + 11 = 46347 200 PORT command successful ---> LIST i.sh 150 Connecting to port 46347 226-Options: -a -l 226 1 matches total ---> TYPE I 200 TYPE is now 8-bit binary ---> PORT 192,168,1,250,237,226 # 237*256 + 11 = 60898 200 PORT command successful ---> RETR i.sh 150 Connecting to port 60898 226-File successfully transferred 226 0.000 seconds (measured here), 21.22 Mbytes per second ---> QUIT 221-Goodbye. You uploaded 0 and downloaded 2 kbytes. 221 Logout. Dump 2: The same TCP stream traffic as above, seen on the other side of the router. Traffic leaving the router 206.47.37.39 heading to Internet (from ethereal): 220---------- Welcome to Pure-FTPd [TLS] ---------- 220-You are user number 1 of 50 allowed. 220-Local time is now 23:34. Server port: 21. 220-This is a private system - No anonymous login 220-IPv6 connections are also welcome on this server. 220 You will be disconnected after 15 minutes of inactivity. ---> USER idallena 331 User idallena OK. Password required ---> PASS XXXX 230-User idallena has group access to: idallena 230 OK. Current restricted directory is / ---> SYST 215 UNIX Type: L8 ---> PORT 206,47,37,39,181,11 # note new IP addr from the ALG 200 PORT command successful ---> LIST i.sh 150 Connecting to port 46347 226-Options: -a -l 226 1 matches total ---> TYPE I 200 TYPE is now 8-bit binary ---> PORT 206,47,37,39,237,226 # note new IP addr from the ALG 200 PORT command successful ---> RETR i.sh 150 Connecting to port 60898 226-File successfully transferred 226 0.000 seconds (measured here), 21.22 Mbytes per second ---> QUIT 221-Goodbye. You uploaded 0 and downloaded 2 kbytes. 221 Logout. ============================================================================= Amusing real-world postal mail spam revenge story: http://schneier.com/crypto-gram-0304.html#1 SMTP open vs. closed relays --------------------------- http://en.wikipedia.org/wiki/Open_mail_relay An open mail relay is an SMTP (e-mail) server configured in such a way that it allows anyone on the Internet to relay (i.e. send) e-mail through it. http://www.ordb.org/faq/ An open mail relay occurs when a mail server processes a mail message where neither the sender nor the recipient is a local user. http://whatis.techtarget.com/definition/0,,sid9_gci782509,00.html An open relay (sometimes called an insecure relay or a third-party relay) is an SMTP e-mail server that allows third-party relay of e-mail messages. By processing mail that is neither for nor from a local user, an open relay makes it possible for an unscrupulous sender to route large volumes of spam. In effect, the owner of the server -- who is typically unaware of the problem -- donates network and computer resources to the sender's purpose. http://email.about.com/library/glossary/bldef_open_relay.htm Definition: When the Internet was young, people trusted each other (or there simply weren't too many of them) and everybody was allowed to use everybody else's SMTP server. The server would take the email message and forward it to the intended recipient. Spammers exploited that openness. They sent their unsolicited messages through somebody else's mail server, thereby masking their real identity to the recipient. This is why most mail servers will not accept messages for relaying today. Servers that will accept such messages are known as open relays. E-mail Authentication http://en.wikipedia.org/wiki/E-mail_authentication Ensuring a valid identity on an e-mail has become a vital first step in stopping spam, forgery, fraud, and even more serious crimes. An essential second step will be ensuring the entity has a good reputation. Unfortunately, the Simple Mail Transfer Protocol (SMTP) that handles most e-mail today was designed in an era when users of the Internet were mostly honest techies who expected others to be equally honest. This article will explain how e-mail identities are forged and the steps that are being taken now to prevent it. Open Relays are not the major problem any more: http://www.imc.org/ube-relay.html - random sample of 500 SMTP servers in August 2002 "To date, there have been mostly anecdotal reports on how many publicly-known SMTP servers allowed anyone to relay through them. Because the reported percentages varied widely, and the test methodologies went unstated, IMC tested a large random sample of SMTP servers in January, 1998 to see how many of them allowed relaying from users not within their realm. IMC repeated the test in July, 1998, July, 1999, January 2001, and August 2002 to show the trends in open relays. The results show that fewer than 1% of mail servers that are named in mail addresses allowed relaying in August 2002, a very sharp reduction from over 6% from a year and a half earlier. In the same 18 months, the amount of spam a typical Internet users receives has gone up significantly. This shows that the amount of spam is not related to the number of open relays available. T/F: Open Relays are a major source of SPAM on the Internet. * checking for open relay using netcat: - connect to the server and see if you can use the server to send yourself an email (where "xxx" and "yyy" are both addresses that are foreign to the network on which the SMTP server resides): $ nc -v localhost smtp EHLO somedomain.ca MAIL FROM: RCPT TO: Q: What is an SMTP "open relay"? Q: How can I use nc to tell if an SMTP server is an "open relay"? "Limiting Unsolicited Bulk Email (UBE)" http://www.imc.org/imc-spam/ "IMC's members have expressed a strong interest in helping to come up with solutions to the problem of unsolicitied bulk email (UBE), better known as "spam". The use and abuse of UBE is spreading rapidly, and many Internet users are complaining loudly about the very negative effects it has on them. Anti-Spam Recommendations for SMTP MTAs http://tools.ietf.org/html/rfc2505 - footnote mentions the Monty Python origin of the term "spam" - done at SMTP level: "Our basic assumption is that refuse/accept is handled at the SMTP layer and that an MTA that decides to refuse a message should do so while still in the SMTP dialogue. First, this means that we do not have to store a copy of a message we later decide to refuse and second, our responsibility for that message is low or none - since we have not yet read it in, we leave it to the sender to handle the error. Q: Give two reasons why refusing spam during the SMTP dialog (refusing to accept the email) is a Good Thing. - suggests using 4xx temporary fail codes; however: "However, 4xx Temporary Errors may have unexpected interaction with MX-records. If the receiving domain has several MX records and the lowest preference MX-host refuses to receive mail with a "451" Response Code, the sending host may choose to - and often will - use the next host on the MX list. [...] Our intent was to make the offending mail stay at the offending sender's host and fill up his mqueue disk, but it all ended up at our friend, the next lowest preference MX-host. Q: What is a major drawback to refusing spam using SMTP Temporary Errors? SMTP Service Extension for Authentication ----------------------------------------- SASL: Simple Authentication and Security Layer SMTP AUTH http://tools.ietf.org/html/rfc2554 http://www.faqs.org/rfcs/rfc2554.html Section 4 p.3: - After a successfule AUTH SASL negotiation, the client starts over from scratch with EHLO again. - Clients can try multiple AUTH commands if first fails. T/F: If a client fails an SMTP AUTH authentication, it must disconnect and reconnect to try again. Brute forcing AUTH using Microsoft guest account: http://www.spamcop.net/fom-serve/cache/372.html For example, by default, Microsoft Exchange 5.5, 2000, 2003 and the Exchange server with IIS/5 set up a guest account. This allows anyone to connect to and use the server. Even if you have set up "require authentication" (meaning the user must supply a username and password) the guest account will allow the user to send mail through the server even if their login fails. Q: Why didn't "require authentication" improve the security of early Microsoft Exchange servers? Internet Mail Consortium - extensive email archives by topic http://www.imc.org/ Current Draft Protocols - Stopping SPAM --------------------------------------- Overview slide: http://mipassoc.org/csv/CSV-Intro-03dc.pdf "A Set of Guidelines for Mass Unsolicited Mailings and Postings (spam*)" http://tools.ietf.org/html/rfc2635 - Section 2 documents the Monty Python origin of the term "spam" "No anti-UBM measure for SMTP-based Internet mail works" http://homepages.tesco.net/~J.deBoynePollard/FGA/smtp-anti-ubm-dont-work.html History of anti-spam (Sep 04): http://www.circleid.com/posts/sender_id_a_tale_of_open_standards_and_corporate_greed_part_i/ http://www.circleid.com/posts/sender_id_a_tale_of_open_standards_and_corporate_greed_part_ii/ Internet Architecture Board (IAB) Internet Research Task Force (IRTF) Anti-Spam Research Group (ASRG) - chartered in March 2003 - the chairs of the ASRG were reluctant to send the idea over to the IETF for standardization until it can be better determined that the idea actually had merit. - Meng Wong, CTO of PoBox.com, forks SPF in Summer 2003 - ASRG started a dedicated subgroup (LMAP) to merge all the varied proposals - failed: trying to do engineering (IETF) instead of research (IRTF) - IETF created MARID - MTA Authorization Records In DNS - Eventually SPF and Caller-ID proposals would merge in the May of 2004, and the combination become known as Sender-ID. - Microsoft then revealed it applied for patents on the technology - SPF authors considered filing a defensive patent "Microsoft is claiming IPR in proposals that may very well not even be theirs, which evolved in an open discussion, is asking for a restrictive license, and refusing to consider the market truth-that most of email server software is FOSS and might not be able to use this standard. - http://new.openspf.org/blobs/spf-community-position IETF MARID group - "MTA Authorization Records In DNS" http://en.wikipedia.org/wiki/MARID http://www.imc.org/ietf-mxcomp/ mailing list http://tools.ietf.org/html/rfc4408 Apr06 experimental http://new.openspf.org/RFC_4408/Errata http://new.openspf.org/Specifications (history chart of RFC) http://tools.ietf.org/html/draft-ellermann-spf-options-01 - SPF Version 1 - applies to the MAIL FROM and HELO SMTP identities only - does not try to parse headers in message body http://tools.ietf.org/html/rfc4406 Apr06 experimental - Sender ID (spf2.0) - Microsoft merger of SPF and Caller ID - same idea as other RFC 2822 layer protocols like DomainKeys IM (DKIM) - uses PRA - parses RFC2822 headers inside message body http://tools.ietf.org/html/rfc4405 Apr06 experimental - "Responsible Submitter" SMTP EHLO extension "SUBMITTER" http://tools.ietf.org/html/rfc4407 Apr06 experimental - Purported Responsible Address (PRA) - used by Sender ID - no mention of CSV Certified Server Validation in standards - MARID shutdown September 2004 (7 months!) http://www.imc.org/ietf-mxcomp/mail-archive/msg05054.html "Concluding a group without it having achieved its goals is never a pleasant prospect, and it is always tempting to believe that just a small amount of additional time and energy will cause consensus to emerge. After careful consideration, however, the working group chairs and area advisor have concluded that such energy would be better spent on gathering deployment experience. - comments on MARID failure http://www.imc.org/ietf-mxcomp/mail-archive/msg05091.html "We need to recognize that the old design philosophy in SMTP 2821, a "relaxed internet spirit required for wide deployment with less emphasis with security" no longer applies today. http://www.imc.org/ietf-mxcomp/mail-archive/msg05055.html "Secondly, the co-chairs/AD allowed this working group to try and create a standard, instead of standardizing existing practices. It is far easier to reach a rough consensus on what people *ARE* doing that what people *SHOULD BE* doing. Even if you don't like what people are doing, it is very useful to give clear descriptions of what is being done. http://new.openspf.org/Press_Release/2005-03-23 - SPF rejects co-opting efforts by Microsoft Sender ID - March 2005 EMail authentication proposals: ------------------------------ ** SPF - sender policy framework http://www.openspf.org/ http://new.openspf.org/Introduction - how it works http://en.wikipedia.org/wiki/Sender_Policy_Framework - domains publish "txt" records in the DNS listing the valid IP addresses that are allowed to send email for that domain (some say this an abuse of the purpose of the txt record) - these are called "SPF" records, e.g. "host -t txt idallen.ca" TXT "v=spf1 ip4:66.11.175.96/30 ?all" - IP address block 66.11.175.96/30 can send mail FROM idallen.ca - recipients of email can check the IP address of the connection and the envelope FROM address against the published SPF records Q: How does Sender Policy Framework (SPF) help validate incoming email? http://www.msexchange.org/tutorials/Sender-Policy-Framework.html http://www.msexchange.org/tutorials/Sender-Policy-Framework.html?printversion "If a spammer legitimately has an account in that domain, or owns the domain, they can still send email. This is a real problem so that some experts expect a massive growth in the registration of one-way domains for spammers to go around SPF and other techniques. But this is not a problem concerning the SPF implementation - it is a problem in general. Q: Why doesn't SPF guarantee that the incoming email is not spam? Big problem with SPF: - Mail can't be forwarded, since forwarding changes the IP address of the mail sender to one not authorized by the SPF records... Q: What is the major problem with the SPF verification system? - The SPF people came up with a "Sender Rewriting Scheme" that would rewrite the envelope FROM addresses and permit forwarding; but, it breaks a long-standing tradition that MTAs must *never* modify envelope FROM addresses: http://lwn.net/Articles/187736/ - June 2006 "The SPF folks have suggested solutions for these problems, but many of them require fundamental changes in how MTAs operate. The Sender Rewriting Scheme (SRS) proposal in particular breaks longstanding email tradition by having forwarding MTAs change the envelope sender as they forward email. Opponents of SPF not only argue that changing this tradition is a bad idea, but also that it is very unlikely to be widely implemented any time soon. Additionally, Mail User Agents (MUAs) would need to learn about SRS encoding in order to parse sender addresses for filtering email at the user end. Q: What part of the SMTP protocol does SRS change? Q: What is the major objection/impediment to adopting SPF with SRS? http://homepages.tesco.net/~J.deBoynePollard/FGA/smtp-spf-is-harmful.html "SPF is harmful. The architectural ramifications of it are so extensive and will have such significant changes on the ways that people can access and can use Internet mail, that it would actually be less costly to switch to an entirely new architecture such as IM2000 Internet mail than it would be to switch to SPF and deal with all of its consequences properly. [...] Most people who have analysed SPF in detail have come to the conclusion that it is a deeply flawed scheme that should be avoided outright. [...] It's trivial for malicious senders to create throwaway domain names with published DNS data, that they supply, declaring any SMTP Relay clients that they like to be (as far as SPF is concerned) "legitimate". And there's no shortage of such domain names to use once and then throw away. Q: T/F, if an email passes the SPF test (the IP matches the FROM domain's SPF record), you know the message isn't spam. http://www.w3.org/Mail/spf/ "Due to issues with SPF and mail forwarding, we intend to leave our SPF record in this state for the forseeable future, so our record is useful mainly for whitelisting. (mail with an 'SPF pass' status from w3.org is most likely legitimate, but other mail can be subject to more scrutiny, e.g. using heuristic-based filters.) Q: T/F, if an email with an envelope address from a reputable domain (e.g. www.w3.org) passes the SPF test (the IP matches the envelope FROM domain's SPF record), you can be reasonably sure the FROM address is valid. - finding SPF records: $ host -t txt gmail.com gmail.com descriptive text "v=spf1 redirect=_spf.google.com" Q: Give an example of a command-line that looks up the SPF record for domain foo.com. $ host -t txt _spf.google.com _spf.google.com descriptive text "v=spf1 ip4:216.239.56.0/23 ip4:64.233.160.0/19 ip4:66.249.80.0/20 ip4:72.14.192.0/18 ?all" Q: In an SPF record, what does an IP address and net mask mean, e.g. "v=spf1 ip4:216.239.56.0/23"? $ dig dyndns.org txt [...] ;; ANSWER SECTION: dyndns.org. 86400 IN TXT "v=spf1 mx/24 a:lists.dyndns.com ip4:63.208.196.0/24 ptr:opensrs.net include:outbound.mailhop.org ~all" ALSO: aol.com, microsoft.com, hotmail.com, dyndns.org, LiveJournal.com, OReilly.com, SAP.com, Spamhaus.org, Symantec.com, Ticketmaster.com, w3.org http://new.openspf.org/SPF_vs_Sender_ID - comparison of the two approaches "SPF can be compared to other SMTP layer protocols like CSV/CSA. Sender ID can be compared to other RFC 2822 layer protocols like DomainKeys IM (DKIM - RFC4871). http://tools.ietf.org/html/rfc4871 Q: T/F, SPF operates at the SMTP protocol level (RFC2821) and does not change the email message body. Q: T/F, Sender-ID operates on the Internet Message level (RFC2822). ** SRS - Sender Rewriting Scheme for SPF forwarding http://www.openspf.org/srs.html http://www.openspf.org/srspng.html - how it works (messy!) - needed to pass SMTP FROM address through SPF forwarders - very invasive! Handling SRS needs MUA/MTA rewrites... - same goals as Sender ID but does not alter message body Q: T/F, a serious impediment to the full adoption of SPF is the need to rewrite MUAs/MTAs to handle the SRS rewriting of envelope addresses. ** Sender ID = Caller ID + SPM (Microsoft patent issues) http://www.silicon.com/research/specialreports/thespamreport/0,39025001,39131378,00.htm "However, this Microsoft effort to push adoption of Sender ID is likely to fail, certainly with such a short deadline, said Jonathan Penn, an analyst at Forrester Research. "Hotmail is in no position to dictate that organisations adopt Sender ID," he said. http://www.circleid.com/posts/sender_id_a_tale_of_open_standards_and_corporate_greed_part_i/ "Now forward twenty years to 2002: John Postel passed away, the Internet standards process is now done by the IETF under ISOC instead of DOD, Apple and IBM market shares are almost nil, the Soviet Union has collapsed, Linux and open source software now run majority of the Internet infrastructure, and the phrase "Evil Empire" is now used to refer to Microsoft. But the original two documents defining the email system of what has become a worldwide network now simply called "the Internet" are still here. Every single email message sent today still follows the original guidelines and format set out over 20 years ago by two scientists working on behalf of the US Government. http://itmanagement.earthweb.com/columns/executive_tech/article.php/3444571 - December 2004 comment on CSV alternative to Sender ID or Domain Keys: "Levine quotes from the text of the applications to show that Microsoft claims not just patent rights on anything similar to Sender ID, but also on spam filters that compute scores based on the content of messages. That's not the kind of patent that standards bodies have ever wanted anyone to have on an Internet protocol. Q: T/F, the adoption of Sender ID is impeded by Microsoft patent issues. ** CSV - "Certified Server Validation" http://mipassoc.org/csv/ - was called "Client SMTP Validation" "CSV originally stood for "Client SMTP Validation". However, market(ing) feedback suggested that "Certified Server Validation" is more useful to folks who are trying to understand the nature of the service, without requiring that they be email geeks... http://mipassoc.org/csv/CSV-Comparison.html - compare CSV with SPF and Sender-ID (CSV is local and simpler) http://mipassoc.org/csv/draft-ietf-marid-csv-intro-02.html (expired Aug 05) http://tools.ietf.org/html/draft-crocker-csv-csa-00 (expired April 06) - drafts submitted to IETF (expired) "This specification defines a mechanism to permit session-time verification that a connecting SMTP client is authorized to request service as a mail transfer client. The mechanism uses a DNS SRV [RFC2782] record as a basis for verifying that the associated domain name is authorized to act as an SMTP client. The mechanism is small, simple and useful. http://itmanagement.earthweb.com/columns/executive_tech/article.php/3444571 - article Dec 2004 - Hello 'Certified Server,' Goodbye Spam Q: T/F, CSV can only certify the currently connecting SMTP client; it is not a complicated end-to-end validation system. ** DNA - "Domain Name Accreditation" http://tools.ietf.org/html/draft-ietf-marid-csv-dna-02 (expired Aug 05) http://tools.ietf.org/html/draft-ietf-marid-csv-intro-02 (expired Aug 05) ** BATV: Bounce Address Tag Validation http://mipassoc.org/batv/ http://mipassoc.org/csv/CSV-Intro-03dc.pdf ** Compatible Low-level Email Authentication and Responsibility (CLEAR) http://mipassoc.org/clear/index.html - combine CSV and BATV ** DKIM: Domain Keys Identified Mail http://www.dkim.org/ http://www.dkim.org/info/dkim-faq.html http://www.dkim.org/ietf-dkim.htm - IETF submission http://tools.ietf.org/html/rfc4871 - DKIM is claimed to be an upgrade of Yahoo's DomainKeys; DKIM was produced by an industry consortium in 2004. It merged and enhanced DomainKeys, from Yahoo! and Identified Internet Mail, from Cisco. - DKIM places its parametric information into RFC2822 header fields that are typically not shown to the recipient. Therefore DKIM's can be entirely invisible to recipients. Q: Does DKIM operate at the SMTP protocol level or at the Internet Message level? SPAM solutions response form (checklist): http://craphound.com/spamsolutions.txt Q: T/F, The current state of RFCs for spam filtering is: Let the market decide. (In 2006 there is no agreed-upon single standard for what is best.) ============================================================================= DNS-based IP block lists - DNSBL -------------------------------- http://en.wikipedia.org/wiki/DNSBL http://www.spamhaus.org/dnsbl_function.html "A DNSBL (commonly referred to as a 'Blocklist") is a database served as a DNS Zone able to be queried in realtime by Internet mail servers for the purpose of obtaining an opinion on the origin of incoming email. The role of a DNSBL such as Spamhaus' SBL/XBL/PBL Advisory system is to provide an opinion, to anyone who asks, on whether a particular IP Address meets Spamhaus' own policy for acceptance of inbound email. Every Internet network that chooses to implement spam filtering is, by doing so, making a policy decision governing acceptance and handling of inbound email. The Receiver unilaterally makes the choices on whether to use DNSBLs, which DNSBLs to use, and what to do with an incoming email if the email message's originating IP Address is "listed" on the DNSBL. The DNSBL itself, like all spam filters, can only answer whether a condition has been met or not. Spamhaus does not tell a 3rd-party mail system what to do with an item of email, the 3rd-party mail system asks Spamhaus for an opinion and Spamhaus responds to that request with its opinion. In effect the receiving mail server asks the Spamhaus DNSBL "Does this Sender's IP Address exist on the Spamhaus database?", the Spamhaus DNSBL simply responds with a "Yes" if present or, if not present does not respond at all (no response means "we have no opinion on that IP Address"). http://www.de.sorbs.net/lookup.shtml "Since those initial 78,000 proxies, the SORBS DNSBL has grown to an astounding 3 million listed hosts (that's less than 0.07% of the possible addresses on the internet - statistics correct as of June 2004). SORBS has also expanded over time to include hacked and hijacked servers, formmail scripts, trojan infestations (particularly those with backdoors), and more recently made the move to pre-emptively list all dynamically allocated IP address space. "The SORBS DNSBL is just list of numbers, nothing more, nothing less. The significance of these numbers is that they are related to hosts on the Internet whose condition/settings have included the particular vulnerabilities which we seek to eliminate, i.e. open relays, open proxies, etc. As a prospective user of the SORBS lists the most important question you need to ask yourself is: Do I understand the listing criteria for the list(s) I plan to use? Q: What does DNSBL stand for and how is a DNSBL implemented? Q: T/F, DNSBLs actually block IP addresses. Q: T/F, DNSBLs only offer opinions about IP addresses, they can't actually block anything. Problems: http://en.wikipedia.org/wiki/DNSBL "Additionally, it may be tricky to get a mistakenly listed IP address removed. For example, to request removal from the DUL provided by dynablock.njabl.org, you are supposed to send an email to removals at mail.njabl.org [3] but that address is in turn protected by the same DUL you are asking to be removed from. Q: Why might it be hard to get your IP address out of a DNSBL? Comparison: http://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists How do you actually query the DNS? ---------------------------------- Web based interface: http://www.ioncannon.net/dnsbl/ Try 81.66.180.11 http://en.wikipedia.org/wiki/DNSBL See "DNSBL Queries When a mail server receives a connection from a client, and wishes to check that client against a DNSBL (let's say, spammers.example.net), it does more or less the following: http://sorbs.net/ http://www.dnsbl.us.sorbs.net/using.shtml (return codes) $ host 11.180.66.81.dnsbl.sorbs.net 11.180.66.81.dnsbl.sorbs.net has address 127.0.0.10 $ host -t txt 11.180.66.81.dnsbl.sorbs.net 11.180.66.81.dnsbl.sorbs.net descriptive text "Dynamic IP Addresses See: http://www.sorbs.net/lookup.shtml?81.66.180.11" http://njabl.org/use.html (return codes) "Non-dial-up range entries will often have a descriptive TXT record which should indicate why the entry was added. Try 81.66.180.11 $ host 11.180.66.81.combined.njabl.org 11.180.66.81.combined.njabl.org has address 127.0.0.3 $ host -t txt 11.180.66.81.combined.njabl.org 11.180.66.81.combined.njabl.org descriptive text "Dynamic/Residential IP range listed by NJABL dynablock - http://njabl.org/dynablock.html" http://www.spamcop.net/ http://www.spamcop.net/bl.shtml?81.66.180.11 (web access) Try 81.66.180.11 $ host 11.180.66.81.bl.spamcop.net 11.180.66.81.bl.spamcop.net has address 127.0.0.2 $ host -t txt 11.180.66.81.bl.spamcop.net 11.180.66.81.bl.spamcop.net descriptive text "Blocked - see http://www.spamcop.net/bl.shtml?81.66.180.11" Q: If bl.spamcop.net is a DNSBL, give a command-line that would query this DNSBL for the address 1.2.3.4. Q: If bl.spamcop.net is a DNSBL, give a command-line that would query this DNSBL for the reason that address 1.2.3.4 was put in the DNSBL. Q: T/F, there is a standard for DNSBL responses, so that a given response always means the same thing from every DNSBL. SpamAssassin RBL checking ------------------------- SpamAssassin is a package that will look at an email message (RFC2822) and analyse the message headers. One service it can provide is looking up mail relay addresses ("Received:" header lines) in DNSBLs: X-Spam-Report: * 2.0 RCVD_IN_SORBS_DUL RBL: SORBS: sent directly from dynamic IP address * [81.66.180.11 listed in dnsbl.sorbs.net] * 1.6 RCVD_IN_BL_SPAMCOP_NET RBL: Received via a relay in bl.spamcop.net * [Blocked - see ] * 1.9 RCVD_IN_NJABL_DUL RBL: NJABL: dialup sender did non-local SMTP * [81.66.180.11 listed in combined.njabl.org] Received: from [81.66.180.11] (helo=m11.net81-66-180.noos.fr) by server320.tchmachines.com with esmtp (Exim 4.52) id 1GmGuf-00030Z-C3 for idallen@idallen.ca; Mon, 20 Nov 2006 16:42:49 -0500 Q: What part of an email message does SpamAssassin extract and send to various DNDBLs for opinions? ============================================================================= //EOF - end of notes CST8165 // see http://www.pengalert.com/ =============================================================================