User:David MacQuigg/Sandbox/MailTransfer03

Email processes and protocols

This article is a subtopic in a cluster of articles under Email system. We assume the reader understands that parent article and its terminology. This article will go into more detail on how email systems work at the machine level, including the processes and protocols used to transfer messages from one machine to the next.

Figure 1 shows the same ideal system as in the parent article, but this time the blocks are machines that handle the message on its way from Author to Recipient. The links between blocks are TCP connections. The labels beside these links are the protocols used to transfer messages over the link. The labels within the blocks designate the processes handling the messages, one per machine in this idealized system (except for the machine associated with the Mailstore, which is usually accessed by two independent processes). This is an application layer view of an email network.

(CC) Diagram: David MacQuigg
Figure 1 Machines, processes and protocols in an ideal email system.

We do not include routers in this figure, because they are part of a lower network layer. Routers are transparent to processes operating on the application layer. That means it doesn't matter how many routers are between one relay and the next. At the level of Figure 1, it is always "one hop" from machine to machine. The unit of transfer is a complete message, not a data packet. Complete messages are stored temporarily at each relay (although only the permanent Mailstore is illustrated).

We also do not show the Border to the open Internet, or any grouping of machines by their administrative role. By separating these concerns, and dealing with them at a higher level, we can reduce the complexity of our machine-level model. All relays between submission and delivery can be treated the same. They receive a message using the SMTP protocol, store it temporarily in a queue, perform some function associated with the administrative level, then send it to the next relay, again using SMTP. The TCP connections may be over the open Internet, or within a private network. At the level of Figure 1, it makes no difference.

As with the parent article, Figure 1 is only one of many posible systems. Most systems have many more relays than shown here, a fact which we illustrate with the dotted line between two of the relays. You can see how many relays handled a message by looking at the lines labeled "Received:" in the message header. There should be one for each relay. Often the purpose of these relays is unclear to anyone outside an organization. It may be something as simple as consolidation of the mail from multiple departmental servers to one corporate server.

In this article, we will briefly explain the processes and protocols shown in Figure 1, and how the system tries to provide reliable delivery and deal with the failure modes known at the time it was designed. Finally, we will introduce some of the acronyms and jargon needed to understand the literature on email systems.

Mail Handling Processes

With the exception of the Author and Recipient processes, all the processes in Figure 1 are relays. They receive, store, and forward messages to the next relay, or to a final destination. In this discussion, however, we will use the term Relay to mean just a simple SMTP relay process.

Relay processes are spawned by programs like Sendmail or Postfix, typically one process for each message. This allows most messages to be processed in seconds, even while there are a few messages taking minutes to process. A busy server might have hundreds of Relays running simultaneously, each Relay receiving a message one packet at a time, waiting for an outgoing connection, or performing the unique administrative function that was assigned to it.

The unique processing at each Relay is typically done by a subprocess or plug-in, written independently of the Relay, and called through a standard interface to the Relay. This allows programmers with expertise in each specific function to focus on their own program and not worry about all the details of message handling that are common to every Relay.

Why not make every process in Figure 1 a simple Relay with a plug-in? It doesn't work when there are requirements not satisfied by SMTP. The Delivery process doesn't use SMTP at all. This is because SMTP is a "push" protocol. The sending relay intitiates the transfer, and the receiving relay is expected to be always on line. This is not the case with most Recipient processes, however. A Recipient may be off line for days. The final transfer from the Mailstore to the Recipient's machine must be done at the Recipient's request. This requires a "pull" protocol (POP, IMAP, or HTTP).

The Submission process could use a standard Relay with SMTP over port 25 (and many do), but there is an advantage to using a distinct process with a unique port (587 instead of 25). Messages arriving on port 587 are all submissions, not a mix of submissions and transfers. Connections on port 587 are always authenticated. The submission process can expect it, and the Author process must always provide it. The parentheses around (SMTP) in Figure 1 show that the protocol on this connection is a modified version of SMTP known as the Submission Protocol.

A Gateway process looks like an SMTP Relay on the SMTP side, but is designed to transparently move messages from an SMTP environment to some other environment. There is less need for gateways, now that almost all email services are using SMTP.

The Mailstore provides "permanent" storage for the Recipient's messages. This is different than the temporary storage at each Relay, where messages wait in a queue only as long as it takes to get an available connection to the next Relay. The final Relay process is shown running on the same machine as the Delivery process. Both must access the same Mailstore.

The Zombie process is not part of the email system, but we include it in Figure 1 for discussions on security. A zombie process typically runs on a home computer, without the knowledge or consent of the computer's owner. It imitates a legitimate Relay, sending exact replicas of legitimate SMTP commands and message headers, faking everything but the IP address of the legitimate Relay. There are millions of these zombies on the Internet, organized into huge "botnets" controlled by hidden "botmasters". The most common defense is to block connections from any IP address known to be a zombie. New zombies keep popping up so fast, however, that spam is a thriving industry, in spite of the IP blacklists.

SMTP and other Mail Transfer Protocols

Message transfer protocols define the interactions at each "hop" in moving a message from Author to Recipient. At one time, every Email Service Provider (ESP) had its own protocols, and could not exchange messages with other ESPs. This forced users to have multiple accounts, one at each ESP having a Recipient to whom they would like to send a message. These proprietary protocols have now been replaced by three universal standards, SMTP, POP, and IMAP. Here is a brief explanation of SMTP, the Simple Mail Transfer Protocol.

Message transfer via SMTP is done by establishing a TCP connection between a Client SMTP process initiating the transfer and a Server SMTP process receiving the message. The initial transfer is done using a Client on the message Author's machine, typically a part of his email program. Intermediate transfers involve Relays having both Server and Client functionality. The final transfer to a Recipient's machine uses the POP or IMAP protocols. SMTP "pushes" messages to the next Relay. POP and IMAP "pull" messages from a mailstore at the destination.

After establishing a TCP connection, the message transfer is guided by a sequence of plain-text commands from the Client and reply codes from the Server. The purpose of these commands is to provide "envelope" information so that the message can be handled without having to read its contents. This separation of function allows the email system to work reliably and efficiently, without putting any constraints on the content or syntax of the message itself. The content may even be encrypted, making it totally unintelligible to the mail handling system.

A good way to understand the SMTP process is to send a small message and issue the commands manually using the 'telnet' program available from the command prompt window on most machines. Telnet is a general-purpose program for communication using TCP. Using telnet, you should be able to connect to your email service provider on port 25, the standard email service port (or port 587 if your provider expects you to authenticate your identity).

Here are the steps in a typical message transfer:

1) Establish a TCP connection.
2) Establish a mail session.
3) Provide a Return Address for the next message.
4) Provide a Recipient Address for the next message.
4a) Repeat step 4 for additional Recipients.
5) Transfer the message and all its attachments.
5a) Repeat from 3 for additional messages.
6) Terminate the mail session.
7) Close the TCP connection.

For more details on this process, see the related article on SMTP.

Reliability Concerns

Reliable operation of an email system requires that there be feedback to an Author, either a notice that something has failed along the way, or a confirmation that the message was displayed on the Recipient's machine. Bounce messages provide the notice of failure. Return Receipts provide confirmation that the message was displayed, even if not read.

Bounce messages can be generated by any process that fails to handle the message as expected. This could be a Relay that cannot make a connection to the destination server, a Recipient whose mailbox is full, or any number of other causes that fall into the "permanent" category. Temporary failures that last only a few hours are not generally reported, as those are considered normal for email.

Bounce messages are sent via SMTP to the Return Address provided in step 3 of the procedure above. This need not be the same as the Author's address, but most often it is, because each Relay simply copies whatever was provided with the incoming message. Some Agents prefer to take a more active role, however, and be notified of any downstream problems. This can be done by "re-writing" the Return Address so that bounces are intercepted.

Re-writing of Return Addresses is also done to avoid spam disguised as a bounce. By adding an authentication code to the Return Address, it is possible to automatically discard any bounce message with an invalid code. See the documents on SRS and BATV for details. Unfortunately, not many Agents do this, preferring instead to report the sender of the bounce as a spammer. This has led to a practice of simply discarding messages without notice to anyone. The result has been a terrible degradation in the reliability of email. Authors can no longer be confident that no bounce means their message was delivered.

Bounce messages are considered a "last resort" to be used when a Relay has accepted a message, and only later discovers it cannot be delivered. Simple errors, like no-such-recipient, should cause an immediate reject during the SMTP session. Rejects avoid the problem of fake Return Addresses, but may still leave the Author uninformed, if the reject occurs "downstream" in the chain of Relays. Rejects cannot be propagated "upstream", because SMTP sessions are terminated at each hop once after the message is accepted.

The move away from bounce messages has led to greater reliance on Return Receipts. An Author can request a Return Receipt by adding a line to the message headers

  Disposition-Notification-To: author@example.com
  

Most email programs have a button or menu command to do this, and a pop-up window that appears to the Recipient when a message with this line is displayed. The Recipient then has an option to ignore the request, or click a button that sends the confirmation. Return Receipts are sent just like normal messages to the address in the request.

Email System Terminology

Term	Definition
MUA	Mail User Agent; a program such as Mozilla Thunderbird, Microsoft Outlook or MUTT that can be used by an Author to compose and send, or by a Recipient to retrieve and read email messages. An MUA can also be a process, such as found in many web browsers or mobile phones.
MSA	Message Submission Agent; a machine, program or process that receives messages directly from an MUA, and performs the functions of an Agent responsible for mail submission.
MTA	Message Transfer Agent; a Relay; a machine, program or process that receives and stores a message, performs some administrative function, and forwards it to the next Relay or Mailstore.
MDA	Message Delivery Agent; the server that accepts mail for a user from a remote MTA and holds it until the user's mail client (their MUA) downloads the message
DSN	Delivery Status Notification (aka Bounce); A message generated by any process handling a message in transit, and sent to the Return Address. DSNs are typically sent when a permanent failure occurs in transit.
MDN	Message Disposition Notification (aka Return Receipt); A message generated by the recipient's MUA, providing information on post-delivery handling, such as notice that the message has been displayed.
SMTP	Simple Mail Transfer Protocol; the protocol used to transfer mail from one mail system to another. Uses port 25 or 587 for unencrypted message transfer.
POP	Post Office Protocol; A protocol where a client connects, downloads mail from the server and then deletes that mail from the server. Mail that is downloaded then "sticks" on the computer the user retrieves their mail from. Contrast with IMAP.
IMAP	Internet Message Access Protocol; IMAP differs from POP in that messages are left on the server; this allows a user to "float" between different clients at different locations but still have access to all their mail

Table 1 shows some acronyms commonly used in discussions of email systems. Four of these (MUA, MSA, MTA, and MDA) may be a bit confusing. The confusion arises over the word agent, which is used to mean a process, a program, or a machine, rather than an individual or organization. The use of plain English words like actor and agent to mean something else is common in computer science. This distortion of meaning occurs naturally, as specialists try to be concise in their communications. Instead of saying "MSA process", meaning the process that performs the functions of an Agent that handles mail submission, it was easier to just use MSA as an acronym. It works as long as the context is clear.

In these articles, we talk about the whole spectrum of "agents", from organizations and individuals to programs and processes, so we can't just use the common jargon without clarification. We also need to make distinctions based on function, like transmitter vs receiver, instead of just calling every relay an MTA.

Related Articles

Email System
Message Formats
Email Authentication

Bibliography

[Crocker09] D. Crocker (2009) "Internet Mail Architecture", RFC-5598, http://ietf.org/rfc/rfc5598.txt.

[Gellens06] R. Gellens, J. Klensin (2006) "Message Submission for Mail", RFC-4409, http://ietf.org/rfc/rfc4409.

[Klensin08] J. Klensin, ed. (2008) "Simple Mail Transfer Protocol", RFC-5321, http://ietf.org/rfc/rfc5321.txt.

[PnD07] L. Peterson, B. Davie (2007) "Computer Networks: A Systems Approach" 4th ed. Sect. 9.1.1 "Electronic Mail", ISBN 0-12-370548-7.

[Resnick08] P. Resnick, ed. (2008) "Internet Message Format", RFC-5322, http://ietf.org/rfc/rfc5322.txt.

[Stevens94] W.R. Stevens (1994) "TCP/IP Illustrated, vol. 1, The Protocols", Chapter 28, "SMTP", ISBN 0-201-63346-9.

[Wikipedia08] Simple Mail Transfer Protocol. More details on SMTP, including history, abuse, and related protocols.

Possible Additional Topics

  ESMTP - RFC-5321
  Port 587 - RFC-4409
  Reply Codes
    550, 450 - greylisting
  Options
    MAIL FROM