User:David MacQuigg/Sandbox/MailTransfer03
Email message transfer
This article is a subtopic in a cluster of articles under Email system. We assume the reader understands that parent article and its terminology. This article will go into more detail on how email systems work at the machine level, including the processes and protocols used to transfer messages from one machine to the next.
Figure 1 shows the same ideal system as in the parent article, but this time the blocks are machines that handle the message on its way from Author to Recipient. The links between blocks are TCP connections. The labels beside these links are the protocols used to transfer messages over the link. The labels within the blocks designate the processes handling the messages, one per machine in this idealized system (except for the machine associated with the Mailstore, which is usually accessed by two independent processes). This is an application layer view of an email network.
We do not include routers in this figure, because they are part of a lower network layer. Routers are transparent to processes operating on the application layer. That means it doesn't matter how many routers are between one relay and the next. At the level of Figure 1, it is always "one hop" from machine to machine. The unit of transfer is a complete message, not a data packet. Complete messages are stored temporarily at each relay (although only the permanent Mailstore is illustrated).
We also do not show the Border to the open Internet, or any grouping of machines by their administrative role. By separating these concerns, and dealing with them at a higher level, we can reduce the complexity of our machine-level model. All relays between submission and delivery can be treated the same. They receive a message using the SMTP protocol, store it temporarily in a queue, perform some function associated with the administrative level, then send it to the next relay, again using SMTP. The TCP connections may be over the open Internet, or within a private network. At the level of Figure 1, it makes no difference.
As with the parent article, Figure 1 is only one of many posible systems. Most systems have many more relays than shown here, a fact which we illustrate with the dotted line between two of the relays. You can see how many relays handled a message by looking at the lines labeled "Received:" in the message header. There should be one for each relay. Often the purpose of these relays is unclear to anyone outside an organization. It may be something as simple as consolidation of the mail from multiple departmental servers to one corporate server.
In this article, we will briefly explain the processes and protocols shown in Figure 1, and how the system tries to provide reliable delivery even with expected failures. Finally, we will introduce some of the acronyms and jargon needed to understand the literature on email systems.
| Term | Definition |
|---|---|
| MUA | Message User Agent; an email program such as Mozilla Thunderbird or Microsoft Outlook that runs on the user's machine. Nowadays an MUA can even reside in a web browser or mobile phone. |
| MSA | Mail Submission Agent; the process which receives messages directly from an MUA, and performs the functions of an Agent responsible for mail submission. |
| MTA | Message Transfer Agent; the software on the server side for moving email messages around and forwarding them to other email server hosts |
| MDA | Mail Delivery Agent; the server that accepts mail for a user from a remote MTA and holds it until the user's mail client (their MUA) downloads the message |
| DSN | Delivery Status Notification (aka Bounce); A message generated by any process handling a message in transit, and sent to the Return Address. DSNs are typically sent when a failure occurs in transit. |
| MDN | Message Disposition Notification (aka Return Receipt); A message generated by the recipient's MUA, providing information on post-delivery handling, such as notice that the message has been displayed. |
| SMTP | Simple Mail Transfer Protocol; the protocol used to transfer mail from one mail system to another. Uses port 25 or 587 for unencrypted message transfer. |
| POP | Post Office Protocol; A protocol where a client connects, downloads mail from the server and then deletes that mail from the server. Mail that is downloaded then "sticks" on the computer the user retrieves their mail from. Contrast with IMAP. |
| IMAP | Internet Message Access Protocol; IMAP differs from POP in that messages are left on the server; this allows a user to "float" between different clients at different locations but still have access to all their mail |
Email System > Message Transfer
This subtopic provides a brief explanation of the Simple Mail Transfer Protocol (SMTP) used to move email messages across the Internet. A complete explanation of SMTP is found in RFC-5321 [Klensin08]. We will assume the reader understands the basic operation of the email system and the role of Mail Relays, as described in the parent article.
Message transfer at each "hop" is done by establishing a TCP connection between a Client SMTP process initiating the transfer and a Server SMTP process receiving the message. The initial transfer is done using a Client on the message Author's machine, typically a part of his email program. Intermediate transfers involve Relays having both Server and Client processes. The final transfer to a recipient's machine uses the POP or IMAP protocols. Why not use SMTP for the final transfer? SMTP is a "push" protocol. The sender initiates the transfer. This won't work on the final transfer, because recipient machines are not online all the time. The recipient must initiate the final transfer from the mailstore to his own machine, using a "pull" protocol like POP or IMAP.
After establishing a TCP connection, the message transfer is guided by a sequence of plain-text commands from the Client and reply codes from the Server. The purpose of the commands is to provide "envelope" information so that the message can be handled without having to read its contents. This separation of function allows the email system to work reliably and efficiently, without putting any constraints on the content or syntax of the message itself. The content may even be encrypted, making it totally unintelligible to the mail handling system.
A good way to understand the message transfer process is to send a small message and issue the commands manually using the 'telnet' program available from the command prompt window on most machines. Telnet is a general-purpose program for communication using TCP. Using telnet, you should be able to connect to your email service provider on port 25, the standard email service port (or port 587 if your provider expects you to authenticate your identity).
Here are the steps in a typical message transfer:
1) Establish a TCP connection. 2) Establish a mail session. 3) Provide a Return Address for the next message. 4) Provide a Recipient Address for the next message. 4a) Repeat step 4 for additional recipients. 5) Transfer the message and all its attachments. 5a) Repeat from 3 for additional messages. 6) Terminate the mail session. 7) Close the TCP connection.
and here is what an actual email session looks like (names changed to avoid abuse). $ is the command prompt. C: means Client, and S: means Server:
$ telnet example.org 25 S: 220 example.org ESMTP Sendmail 8.13.1/8.13.1; Wed, 30 Aug 2006 07:36:42 -0400 C: HELO mailout1.phrednet.com S: 250 example.org Hello ip068.subnet71.gci-net.com [216.183.71.68], pleased to meet you C: MAIL FROM:<xxxx@example.com> S: 250 2.1.0 <xxxx@example.com>... Sender ok C: RCPT TO:<yyyy@example.com> S: 250 2.1.5 <yyyy@example.com>... Recipient ok C: DATA S: 354 Enter mail, end with "." on a line by itself From: Dave\r\nTo: Test Recipient\r\nSubject: SPAM SPAM SPAM\r\n\r\nThis is message 1 from our test script.\r\n.\r\n S: 250 2.0.0 k7TKIBYb024731 Message accepted for delivery C: QUIT S: 221 2.0.0 example.org closing connection Connection closed by foreign host. $
In this session, we have used the five most basic SMTP commands (HELO, MAIL FROM, RCTP TO, DATA, and QUIT) to communicate with a server running Sendmail, one of the most popular mail server programs. The commands and the numerical response codes are standard for all server programs, but the response text following the standard codes may differ. Complete details of these and other less common commands are found in [Klensin08].
Here is a step-by-step explanation of the session above:
1) The telnet program requests a TCP connection to port 25 at the IP address of a server for example.org. Telnet uses a DNS query to find this address.
220 is the standard three-digit reply code for an email server to accept a connection request. If this were an automated process instead of telnet, the Client machine would read the standard code and ignore the rest of the line, which is intended for humans reading a log file. There is no standard form for the information after a reply code. The administrator at example.org might decide, for example, that it is not a good idea to advertise exactly what version of Sendmail he is running. If a vulnerability is discovered in that version, within hours there could be a hundred criminals scanning the Internet for any systems running that version.
2) The HELO command [1] requests a mail session and identifies the Client machine. The identifier should end in the domain name registered to the organization or individual who is responsible for this machine.
250 is the reply code for OK (the command was run without error). Following that code Sendmail provides a more complete greeting message with information (the IPname and IP address of the Client machine) that might be useful to the sender if there is a problem. The IP address is the source address of the TCP connection. The IPname is found by a Reverse DNS query on the IP address.
Notice that the IPname assigned by the Client's network owner can be different than the HELO name used by the Client, so an IPname is not a good way to identify the Client. Network owners often assign thousands of these names using a script which generates the name from the IP address. Savvy mail admins will avoid using these names in their HELO commands, because some anti-spam filters treat these automated names as a sign of spam.
3) The MAIL FROM command provides a Return Address [2] for an error report if there is a problem at any Relay between the sender and the recipient. The Return Address is usually the same as the From Address in the headers of the message, but it can be different. It might, for example, be re-written by a Forwarder so as to intercept any error reports from downstream Agents. It might also be null, indicating that no error reports should be sent. This option should always be used for the error messages themselves, avoiding the risk of one error message generating another, ad infinitum.
The 250 reply code is common to four of the commands in this session. Notice there is an additional "enhanced status code" on some of these replies. These enhanced codes [3] were standardized after years of experience using just the original reply codes. They provide a more detailed machine-readable classification of the various replies to a command. Enhanced codes are optional, and all mail servers should work with just the original three-digit codes.
4) The RCPT TO command specifies the address of one recipient. This command is repeated for each recipient. Any or all of the recipients can be rejected, and delivery will be attempted for those recipients whose address was accepted. If none are accepted, the entire message is rejected.
One of the functions of an email Relay is to group recipient addresses so that only one copy of a message must be sent to each group at a single destination. This can be a problem if some of those addresses were designated BCC by the message author. SMTP makes no distinction between TO, CC, and BCC addresses. Senders should never assume that BCC addresses are truly hidden.
5) The DATA command initiates the transfer of the message data, which can include headers, plain text, HTML text, and various other blocks of data encoded so that even binary data can be sent as simple strings of ASCII characters. Raw binary data cannot be sent, since there might be confusion if the end-of-line sequence (\r\n) occurs anywhere in the data. Headers come first, terminated by a blank line. The blank line in our example appears as two consecutive end-of-lines. The end of the entire message appears as a "." on a line by itself (\r\n.\r\n).
6) The QUIT command terminates the mail session.
7) The TCP connection is then closed by the mail server.
Notes
- ↑ An alternative command EHLO, is actually seen more often. This is a request to use ESMTP, an "extended" version of the original SMTP that supports new functionality. If the Server doesn't support ESMTP, the command fails, and the ESMTP Client repeats the request using the old HELO command. This awkward procedure was necessry because the syntax of the original HELO command did not allow for future options.
- ↑ The Return Address is also known as the Return Path or the Reverse Path. "Path" however, is a misnomer, since this is really a destination point, not a complete path. We prefer Return Address, since this SMTP envelope item has the same function as the return address on a paper envelope.
- ↑ G. Vaudreuil (1996) RFC-1893 Enhanced Mail System Status Codes, http://www.ietf.org/rfc/rfc1893.txt
Related Articles
Email System
Message Formats
Email Authentication
Bibliography
[Klensin08] J. Klensin, ed. (2008) "Simple Mail Transfer Protocol", RFC-5321, http://tools.ietf.org/html/rfc5321.
[PnD07] L. Peterson, B. Davie (2007) "Computer Networks: A Systems Approach" 4th ed. Sect. 9.1.1 "Electronic Mail", ISBN 0-12-370548-7.
[Resnick08] P. Resnick, ed. (2008) "Internet Message Format", RFC-5322, http://tools.ietf.org/html/rfc5322.
[Stevens94] W.R. Stevens (1994) "TCP/IP Illustrated, vol. 1, The Protocols", Chapter 28, "SMTP", ISBN 0-201-63346-9.
[Wikipedia08] Simple Mail Transfer Protocol. More details on SMTP, including history, abuse, and related protocols.
Possible Additional Topics
ESMTP - RFC-5321
Port 587 - RFC-4409
Reply Codes
550, 450 - greylisting
Options
MAIL FROM