User:David MacQuigg/Sandbox/MailTransfer03: Difference between revisions

From Citizendium
Jump to navigation Jump to search
imported>David MacQuigg
(Rev 03)
 
imported>David MacQuigg
 
(17 intermediate revisions by the same user not shown)
Line 1: Line 1:
== Email message transfer ==
== Email processes and protocols ==


This article is a [[CZ:Related articles|subtopic]] in a cluster of articles under [[Email system]].  We assume the reader understands that parent article and its terminology.  This article will go into more detail on how email systems work at the machine level, including the processes and protocols used to transfer messages from one machine to the next.
This article is a [[CZ:Related articles|subtopic]] in a cluster of articles under [[Email system]].  We assume the reader understands that parent article and its terminology.  This article will go into more detail on how email systems work at the machine level, including the processes and protocols used to transfer messages from one machine to the next.


Figure 1 shows the same ideal system as in the parent article, but this time the blocks are machines that handle the message on its way from Author to Recipient.  The links between blocks are [[TCP connections]].  The labels beside these links are the protocols used to transfer messages over the link.  The labels within the blocks designate the processes handling the messages, one per machine in this idealized system (except for the machine associated with the Mailstore, which is usually accessed by two independent processes).  This is an [[application layer]] view of an email network.
Figure 1 shows the same ideal system as in the parent article, but this time the blocks are machines that handle the message on its way from Author to Recipient.  The links between blocks are [[TCP connections]].  The labels beside these links are the protocols used to transfer messages over the link.  The labels within the blocks designate the processes handling the messages, one per machine in this idealized system (except for the machine associated with the Mailstore, which is usually accessed by two independent processes).  This is an [[application layer]] view of an email network.
{{Image|Message transfer.png|left|700px|'''Figure 1  Machines, processes and protocols in an ideal email system.'''}}


We do not include routers in this figure, because they are part of a lower [[network layer]].  Routers are transparent to processes operating on the application layer.  That means it doesn't matter how many routers are between one relay and the next.  At the level of Figure 1, it is always "one hop" from machine to machine.  The unit of transfer is a complete message, not a data packet.  Complete messages are stored temporarily at each relay (although only the permanent Mailstore is illustrated).
We do not include routers in this figure, because they are part of a lower [[network layer]].  Routers are transparent to processes operating on the application layer.  That means it doesn't matter how many routers are between one relay and the next.  At the level of Figure 1, it is always "one hop" from machine to machine.  The unit of transfer is a complete message, not a data packet.  Complete messages are stored temporarily at each relay (although only the permanent Mailstore is illustrated).
Line 9: Line 11:
We also do not show the Border to the open Internet, or any grouping of machines by their administrative role.  By [[separating these concerns]], and dealing with them at a higher level, we can reduce the complexity of our machine-level model.  All relays between submission and delivery can be treated the same.  They receive a message using the SMTP protocol, store it temporarily in a queue, perform some function associated with the administrative level, then send it to the next relay, again using SMTP.  The TCP connections may be over the open Internet, or within a private network.  At the level of Figure 1, it makes no difference.
We also do not show the Border to the open Internet, or any grouping of machines by their administrative role.  By [[separating these concerns]], and dealing with them at a higher level, we can reduce the complexity of our machine-level model.  All relays between submission and delivery can be treated the same.  They receive a message using the SMTP protocol, store it temporarily in a queue, perform some function associated with the administrative level, then send it to the next relay, again using SMTP.  The TCP connections may be over the open Internet, or within a private network.  At the level of Figure 1, it makes no difference.


As with the parent article, Figure 1 is only one of many posible systems.  Most systems have many more relays than shown here, a fact which we illustrate with the dotted line between two of the relays.  You can see how many relays handled a message by looking at the lines labeled "Received:" in the message header.  There should be one for each relay.  Often the purpose of these relays is unclear to anyone outside an organization.  It may be something as simple as consolidation of the mail from multiple departmental servers to one corporate server.
As with the parent article, Figure 1 is only one of many posible systems.  Most systems have many more relays than shown here, a fact which we illustrate with the dotted line between two of the relays.  You can see how many relays handled a message by looking at the lines labeled "Received:" in the [[message header]].  There should be one for each relay.  Often the purpose of these relays is unclear to anyone outside an organization.  It may be something as simple as consolidation of the mail from multiple departmental servers to one corporate server.
 
In this article, we will briefly explain the processes and protocols shown in Figure 1, and how the system tries to provide reliable delivery and deal with the failure modes known at the time it was designed. Finally, we will introduce some of the acronyms and jargon needed to understand the literature on email systems.
 
=== Mail Handling Processes ===
 
With the exception of the '''Author''' and '''Recipient''' processes, all the processes in Figure 1 are relays.  They receive, store, and forward messages to the next relay, or to a final destination.  In this discussion, however, we will use the term '''Relay''' to mean just a simple SMTP relay process.
 
Relay processes are [[spawned]] by programs like [[Sendmail]] or [[Postfix]], typically one process for each message.  This allows most messages to be processed in seconds, even while there are a few messages taking minutes to process.  A busy server might have hundreds of Relays running simultaneously, each Relay receiving a message one packet at a time, waiting for an outgoing connection, or performing the unique administrative function that was assigned to it.
 
The unique processing at each Relay is typically done by a subprocess or plug-in, written independently of the Relay, and called through a standard interface to the Relay.  This allows programmers with expertise in each specific function to focus on their own program and not worry about all the details of message handling that are common to every Relay.
 
Why not make every process in Figure 1 a simple Relay with a plug-in?  It doesn't work when there are requirements not satisfied by SMTP.  The '''Delivery''' process doesn't use SMTP at all.  This is because SMTP is a "push" protocol.  The sending relay intitiates the transfer, and the receiving relay is expected to be always on line.  This is not the case with most Recipient processes, however.  A Recipient may be off line for days.  The final transfer from the Mailstore to the Recipient's machine must be done at the Recipient's request.  This requires a "pull" protocol (POP, IMAP, or HTTP).
 
The '''Submission''' process could use a standard Relay with SMTP over port 25 (and many do), but there is an advantage to using a distinct process with a unique port (587 instead of 25).  Messages arriving on port 587 are all submissions, not a mix of submissions and transfers.  Connections on port 587 are always authenticated.  The submission process can expect it, and the Author process must always provide it.  The parentheses around (SMTP) in Figure 1 show that the protocol on this connection is a modified version of SMTP known as the [[Submission Protocol]].
 
A '''Gateway''' process looks like an SMTP Relay on the SMTP side, but is designed to transparently move messages from an SMTP environment to some other environment.  There is less need for gateways, now that almost all email services are using SMTP.


In this article, we will briefly explain the processes and protocols shown in Figure 1, and how the system tries to provide reliable delivery even with expected failures. Finally, we will introduce some of the acronyms and jargon needed to understand the literature on email systems.
The '''Mailstore''' provides "permanent" storage for the Recipient's messages.  This is different than the temporary storage at each Relay, where messages wait in a queue only as long as it takes to get an available connection to the next Relay.  The final Relay process is shown running on the same machine as the Delivery process. Both must access the same Mailstore.


{{Image|Message transfer.png|right|700px|'''Figure 1  Machines, processes and protocols in an ideal email system.'''}}
The '''Zombie''' process is not part of the email system, but we include it in Figure 1 for discussions on security. A zombie process typically runs on a home computer, without the knowledge or consent of the computer's owner.  It imitates a legitimate Relay, sending exact replicas of legitimate SMTP commands and message headers, faking everything but the IP address of the legitimate Relay.  There are millions of these zombies on the Internet, organized into huge "botnets" controlled by hidden "botmasters".  The most common defense is to block connections from any IP address known to be a zombie.  New zombies keep popping up so fast, however, that spam is a thriving industry, in spite of the [[IP blacklists]].


{| class="wikitable sortable" width="35%" align="right"
=== SMTP and other Mail Transfer Protocols ===
! Term
! Definition
|-
| MUA
| Message User Agent; an email program such as [[Mozilla Thunderbird]] or [[Microsoft Outlook]] that runs on the user's machine. Nowadays an MUA can even reside in a web browser or mobile phone.
|-
| MSA
| Mail Submission Agent; the process which receives messages directly from an MUA, and performs the functions of an Agent responsible for mail submission.
|-
| MTA
| Message Transfer Agent; the software on the server side for moving email messages around and forwarding them to other email server hosts
|-
| MDA
| Mail Delivery Agent; the server that accepts mail for a user from a remote MTA and holds it until the user's mail client (their MUA) downloads the message
|-
| DSN
| Delivery Status Notification (aka Bounce); A message generated by any process handling a message in transit, and sent to the Return Address.  DSNs are typically sent when a failure occurs in transit.
|-
| MDN
| Message Disposition Notification (aka Return Receipt); A message generated by the recipient's MUA, providing information on post-delivery handling, such as notice that the message has been displayed.
|-
| SMTP
| Simple Mail Transfer Protocol; the protocol used to transfer mail from one mail system to another. Uses port 25 or 587 for unencrypted message transfer.
|-
| POP
| Post Office Protocol; A protocol where a client connects, downloads mail from the server and then deletes that mail from the server. Mail that is downloaded then "sticks" on the computer the user retrieves their mail from. Contrast with IMAP.
|-
| IMAP
| Internet Message Access Protocol; IMAP differs from POP in that messages are left on the server; this allows a user to "float" between different clients at different locations but still have access to all their mail
|}


== Email System > Message Transfer ==
Message transfer protocols define the interactions at each "hop" in moving a message from Author to Recipient.  At one time, every Email Service Provider (ESP) had its own protocols, and could not exchange messages with other ESPs.  This forced users to have multiple accounts, one at each ESP having a Recipient to whom they would like to send a message.  These proprietary protocols have now been replaced by three universal standards, [[SMTP]], [[POP]], and [[IMAP]].  Here is a brief explanation of SMTP, the Simple Mail Transfer Protocol.
This subtopic provides a brief explanation of the Simple Mail Transfer Protocol (SMTP) used to move email messages across the Internet.  A complete explanation of SMTP is found in RFC-5321 [Klensin08].  We will assume the reader understands the basic operation of the email system and the role of Mail Relays, as described in the [[Email System|parent article]].


Message transfer at each "hop" is done by establishing a [[TCP]] connection between a Client SMTP process initiating the transfer and a Server SMTP process receiving the message.  The initial transfer is done using a Client on the message Author's machine, typically a part of his email program.  Intermediate transfers involve Relays having both Server and Client processes.  The final transfer to a recipient's machine uses the [[POP]] or [[IMAP]] protocols.  Why not use SMTP for the final transfer?  SMTP is a "push" protocol.  The sender initiates the transfer.  This won't work on the final transfer, because recipient machines are not online all the time.  The recipient must initiate the final transfer from the mailstore to his own machine,  using a "pull" protocol like POP or IMAP.
Message transfer via SMTP is done by establishing a [[TCP]] connection between a Client SMTP process initiating the transfer and a Server SMTP process receiving the message.  The initial transfer is done using a Client on the message Author's machine, typically a part of his email program.  Intermediate transfers involve Relays having both Server and Client functionality.  The final transfer to a Recipient's machine uses the [[POP]] or [[IMAP]] protocols.  SMTP "pushes" messages to the next Relay.  POP and IMAP "pull" messages from a mailstore at the destination.


After establishing a TCP connection, the message transfer is guided by a sequence of plain-text commands from the Client and reply codes from the Server.  The purpose of the commands is to provide "envelope" information so that the message can be handled without having to read its contents.  This separation of function allows the email system to work reliably and efficiently, without putting any constraints on the content or syntax of the message itself.  The content may even be encrypted, making it totally unintelligible to the mail handling system.
After establishing a TCP connection, the message transfer is guided by a sequence of plain-text commands from the Client and reply codes from the Server.  The purpose of these commands is to provide "envelope" information so that the message can be handled without having to read its contents.  This separation of function allows the email system to work reliably and efficiently, without putting any constraints on the content or syntax of the message itself.  The content may even be encrypted, making it totally unintelligible to the mail handling system.


A good way to understand the message transfer process is to send a small message and issue the commands manually using the 'telnet' program available from the command prompt window on most machines.  Telnet is a general-purpose program for communication using TCP.  Using telnet, you should be able to connect to your email service provider on port 25, the standard email service port (or port 587 if your provider expects you to authenticate your identity).
A good way to understand the SMTP process is to send a small message and issue the commands manually using the 'telnet' program available from the command prompt window on most machines.  Telnet is a general-purpose program for communication using TCP.  Using telnet, you should be able to connect to your email service provider on port 25, the standard email service port (or port 587 if your provider expects you to authenticate your identity).


Here are the steps in a typical message transfer:
Here are the steps in a typical message transfer:
Line 62: Line 49:
  3) Provide a Return Address for the next message.
  3) Provide a Return Address for the next message.
  4) Provide a Recipient Address for the next message.
  4) Provide a Recipient Address for the next message.
  4a) Repeat step 4 for additional recipients.
  4a) Repeat step 4 for additional Recipients.
  5) Transfer the message and all its attachments.
  5) Transfer the message and all its attachments.
  5a) Repeat from 3 for additional messages.
  5a) Repeat from 3 for additional messages.
Line 68: Line 55:
  7) Close the TCP connection.
  7) Close the TCP connection.


and here is what an actual email session looks like (names changed to avoid abuse). $ is the command prompt. C: means Client, and S: means Server:
For more details on this process, see the related article on [[SMTP]].


$ telnet example.org 25
==== Post Office Protocol (POP) ====
S: 220 example.org ESMTP Sendmail 8.13.1/8.13.1; Wed, 30 Aug 2006 07:36:42 -0400
C: HELO mailout1.phrednet.com
S: 250 example.org Hello ip068.subnet71.gci-net.com [216.183.71.68], pleased to meet you
C: MAIL FROM:<xxxx@example.com>
S: 250 2.1.0 <xxxx@example.com>... Sender ok
C: RCPT TO:<yyyy@example.com>
S: 250 2.1.5 <yyyy@example.com>... Recipient ok
C: DATA
S: 354 Enter mail, end with "." on a line by itself
From: Dave\r\nTo: Test Recipient\r\nSubject: SPAM SPAM SPAM\r\n\r\nThis is message 1 from our test script.\r\n.\r\n
S: 250 2.0.0 k7TKIBYb024731 Message accepted for delivery
C: QUIT
S: 221 2.0.0 example.org closing connection
Connection closed by foreign host.
$


In this session, we have used the five most basic SMTP commands (HELO, MAIL FROM, RCTP TO, DATA, and QUIT) to communicate with a server running Sendmail, one of the most popular mail server programs.  The commands and the numerical response codes are standard for all server programs, but the response text following the standard codes may differ.  Complete details of these and other less common commands are found in [Klensin08].
==== Internet Message Access Protocol (IMAP) ====


Here is a step-by-step explanation of the session above:
==== Webmail and the HyperText Transfer Protocol (HTTP) ====


'''1)''' The telnet program requests a TCP connection to port 25 at the IP address of a server for example.org.  Telnet uses a [[DNS]] query to find this address.


220 is the standard three-digit reply code for an email server to accept a connection request.  If this were an automated process instead of telnet, the Client machine would read the standard code and ignore the rest of the line, which is intended for humans reading a log file.  There is no standard form for the information after a reply code.  The administrator at example.org might decide, for example, that it is not a good idea to advertise exactly what version of Sendmail he is running.  If a vulnerability is discovered in that version, within hours there could be a hundred criminals scanning the Internet for any systems running that version.
=== Reliability Concerns ===


'''2)''' The HELO command
Reliable operation of an email system requires that there be feedback to an Author, either a notice that something has failed along the way, or a confirmation that the message was displayed on the Recipient's machine.  Bounce messages provide the notice of failure.  Return Receipts provide confirmation that the message was displayed, even if not read.
<ref>
An alternative command '''EHLO''', is actually seen more often.  This is a request to use [[ESMTP]], an "extended" version of the original SMTP that supports new functionality.  If the Server doesn't support ESMTP, the command fails, and the ESMTP Client repeats the request using the old HELO command.  This awkward procedure was necessry because the syntax of the original HELO command did not allow for future options.
</ref>
requests a mail session and identifies the Client machine.  The identifier should end in the domain name registered to the organization or individual who is responsible for this machine.


250 is the reply code for OK (the command was run without error).  Following that code Sendmail provides a more complete greeting message with information (the IPname and IP address of the Client machine) that might be useful to the sender if there is a problem.  The IP address is the source address of the TCP connection.  The IPname is found by a [[Reverse DNS]] query on the IP address.
'''Bounce messages''' can be generated by any process that fails to handle the message as expected.  This could be a Relay that cannot make a connection to the destination server, a Recipient whose mailbox is full, or any number of other causes that fall into the "permanent" category.  Temporary failures that last only a few hours are not generally reported, as those are considered normal for email.


Notice that the IPname assigned by the Client's network owner can be different than the HELO name used by the Client, so an IPname is not a good way to identify the Client.  Network owners often assign thousands of these names using a script which generates the name from the IP address.  Savvy mail admins will avoid using these names in their HELO commands, because some anti-spam filters treat these automated names as a sign of spam.
Bounce messages are sent via SMTP to the Return Address provided in step 3 of the procedure above.  This need not be the same as the Author's address, but most often it is, because each Relay simply copies whatever was provided with the incoming message.  Some Agents prefer to take a more active role, however, and be notified of any downstream problems.  This can be done by "re-writing" the Return Address so that bounces are intercepted.


'''3)''' The MAIL FROM command provides a Return Address
Re-writing of Return Addresses is also done to avoid spam disguised as a bounce.  By adding an authentication code to the Return Address, it is possible to automatically discard any bounce message with an invalid code.  See the documents on [[SRS]] and [[BATV]] for details.  Unfortunately, not many Agents do this, preferring instead to report the sender of the bounce as a spammer.  This has led to a practice of simply discarding messages without notice to anyone.  The result has been a terrible degradation in the reliability of email.  Authors can no longer be confident that no bounce means their message was delivered.
<ref>
The '''Return Address''' is also known as the Return Path or the Reverse Path.  "Path" however, is a misnomer, since this is really a destination point, not a complete path.  We prefer Return Address, since this SMTP envelope item has the same function as the return address on a paper envelope.
</ref>
for an error report if there is a problem at any Relay between the sender and the recipient.  The Return Address is usually the same as the From Address in the headers of the message, but it can be different.  It might, for example, be re-written by a [[Forwarder]] so as to intercept any error reports from downstream [[Agents]].  It might also be null, indicating that no error reports should be sent.  This option should always be used for the error messages themselves, avoiding the risk of one error message generating another, ad infinitum.


The 250 reply code is common to four of the commands in this session.  Notice there is an additional "enhanced status code" on some of these replies.  These enhanced codes
Bounce messages are considered a "last resort" to be used when a Relay has accepted a message, and only later discovers it cannot be delivered.  Simple errors, like no-such-recipient, should cause an immediate reject during the SMTP session.  Rejects avoid the problem of fake Return Addresses, but may still leave the Author uninformed, if the reject occurs "downstream" in the chain of Relays.  Rejects cannot be propagated "upstream", because SMTP sessions are terminated at each hop once after the message is accepted.
<ref>
G. Vaudreuil (1996) RFC-1893 Enhanced Mail System Status Codes, http://www.ietf.org/rfc/rfc1893.txt
</ref>
were standardized after years of experience using just the original reply codes.  They provide a more detailed machine-readable classification of the various replies to a command.  Enhanced codes are optional, and all mail servers should work with just the original three-digit codes.


'''4)''' The RCPT TO command specifies the address of one recipient.  This command is repeated for each recipient.  Any or all of the recipients can be rejected, and delivery will be attempted for those recipients whose address was accepted.  If none are accepted, the entire message is rejected.
The move away from bounce messages has led to greater reliance on '''Return Receipts'''.  An Author can request a Return Receipt by adding a line to the [[message headers]]


One of the functions of an email Relay is to group recipient addresses so that only one copy of a message must be sent to each group at a single destination.  This can be a problem if some of those addresses were designated BCC by the message author.  SMTP makes no distinction between TO, CC, and BCC addresses.  Senders should never assume that BCC addresses are truly hidden.
  Disposition-Notification-To: author@example.com
 
Most email programs have a button or menu command to do this, and a pop-up window that appears to the Recipient when a message with this line is displayed.  The Recipient then has an option to ignore the request, or click a button that sends the confirmation.  Return Receipts are sent just like normal messages to the address in the request.


'''5)''' The DATA command initiates the transfer of the message data, which can include headers, plain text, HTML text, and various other blocks of data [[encoded]] so that even binary data can be sent as simple strings of ASCII characters. Raw binary data cannot be sent, since there might be confusion if the end-of-line sequence (\r\n) occurs anywhere in the data.  Headers come first, terminated by a blank line.  The blank line in our example appears as two consecutive end-of-lines.  The end of the entire message appears as a "." on a line by itself (\r\n.\r\n).
=== Email System Terminology ===


'''6)''' The QUIT command terminates the mail session.
{| class="wikitable sortable" width="60%" align="right"
! Term
! Definition
|-
| MUA
| Mail User Agent; a program such as [[Mozilla Thunderbird]], [[Microsoft Outlook]] or [[MUTT]] that can be used by an Author to compose and send, or by a Recipient to retrieve and read email messages.  An MUA can also be a process, such as found in many web browsers or mobile phones.
|-
| MSA
| Message Submission Agent; a machine, program or process that receives messages directly from an MUA, and performs the functions of an Agent responsible for mail submission.
|-
| MTA
| Message Transfer Agent; a Relay; a machine, program or process that receives and stores a message, performs some administrative function, and forwards it to the next Relay or Mailstore.
|-
| MDA
| Message Delivery Agent; the server that accepts mail for a user from a remote MTA and holds it until the user's mail client (their MUA) downloads the message
|-
| DSN
| Delivery Status Notification (aka Bounce); A message generated by any process handling a message in transit, and sent to the Return Address.  DSNs are typically sent when a permanent failure occurs in transit.
|-
| MDN
| Message Disposition Notification (aka Return Receipt); A message generated by the recipient's MUA, providing information on post-delivery handling, such as notice that the message has been displayed.
|-
| SMTP
| Simple Mail Transfer Protocol; the protocol used to transfer mail from one mail system to another. Uses port 25 or 587 for unencrypted message transfer.
|-
| POP
| Post Office Protocol; A protocol where a client connects, downloads mail from the server and then deletes that mail from the server. Mail that is downloaded then "sticks" on the computer the user retrieves their mail from. Contrast with IMAP.
|-
| IMAP
| Internet Message Access Protocol; IMAP differs from POP in that messages are left on the server; this allows a user to "float" between different clients at different locations but still have access to all their mail
|}


'''7)''' The TCP connection is then closed by the mail server.
Table 1 shows some acronyms commonly used in discussions of email systems.  Four of these (MUA, MSA, MTA, and MDA) may be a bit confusing.  The confusion arises over the word agent, which is used to mean a process, a program, or a machine, rather than an individual or organization.  The use of plain English words like actor and agent to mean something else is common in computer science.  This distortion of meaning occurs naturally, as specialists try to be concise in their communications.  Instead of saying "MSA process", meaning the process that performs the functions of an Agent that handles mail submission, it was easier to just use MSA as an acronym.  It works as long as the context is clear.
 
In these articles, we talk about the whole spectrum of "agents", from organizations and individuals to programs and processes, so we can't just use the common jargon without clarification.  We also need to make distinctions based on function, like transmitter vs receiver, instead of just calling every relay an MTA.


== Notes ==
== Notes ==
Line 136: Line 129:


== Bibliography ==
== Bibliography ==
[Klensin08] J. Klensin, ed. (2008) "Simple Mail Transfer Protocol", RFC-5321, http://tools.ietf.org/html/rfc5321.
[Crocker09] D. Crocker (2009) "Internet Mail Architecture", RFC-5598, http://ietf.org/rfc/rfc5598.txt.
 
[Gellens06] R. Gellens, J. Klensin (2006) "Message Submission for Mail", RFC-4409, http://ietf.org/rfc/rfc4409.
 
[Klensin08] J. Klensin, ed. (2008) "Simple Mail Transfer Protocol", RFC-5321, http://ietf.org/rfc/rfc5321.txt.


[PnD07] L. Peterson, B. Davie (2007) "Computer Networks: A Systems Approach" 4th ed. Sect. 9.1.1 "Electronic Mail", ISBN 0-12-370548-7.
[PnD07] L. Peterson, B. Davie (2007) "Computer Networks: A Systems Approach" 4th ed. Sect. 9.1.1 "Electronic Mail", ISBN 0-12-370548-7.


[Resnick08] P. Resnick, ed. (2008) "Internet Message Format", RFC-5322, http://tools.ietf.org/html/rfc5322.
[Resnick08] P. Resnick, ed. (2008) "Internet Message Format", RFC-5322, http://ietf.org/rfc/rfc5322.txt.


[Stevens94] W.R. Stevens (1994) "TCP/IP Illustrated, vol. 1, The Protocols", Chapter 28, "SMTP", ISBN 0-201-63346-9.
[Stevens94] W.R. Stevens (1994) "TCP/IP Illustrated, vol. 1, The Protocols", Chapter 28, "SMTP", ISBN 0-201-63346-9.

Latest revision as of 09:42, 12 August 2009

Email processes and protocols

This article is a subtopic in a cluster of articles under Email system. We assume the reader understands that parent article and its terminology. This article will go into more detail on how email systems work at the machine level, including the processes and protocols used to transfer messages from one machine to the next.

Figure 1 shows the same ideal system as in the parent article, but this time the blocks are machines that handle the message on its way from Author to Recipient. The links between blocks are TCP connections. The labels beside these links are the protocols used to transfer messages over the link. The labels within the blocks designate the processes handling the messages, one per machine in this idealized system (except for the machine associated with the Mailstore, which is usually accessed by two independent processes). This is an application layer view of an email network.

(CC) Diagram: David MacQuigg
Figure 1 Machines, processes and protocols in an ideal email system.

We do not include routers in this figure, because they are part of a lower network layer. Routers are transparent to processes operating on the application layer. That means it doesn't matter how many routers are between one relay and the next. At the level of Figure 1, it is always "one hop" from machine to machine. The unit of transfer is a complete message, not a data packet. Complete messages are stored temporarily at each relay (although only the permanent Mailstore is illustrated).

We also do not show the Border to the open Internet, or any grouping of machines by their administrative role. By separating these concerns, and dealing with them at a higher level, we can reduce the complexity of our machine-level model. All relays between submission and delivery can be treated the same. They receive a message using the SMTP protocol, store it temporarily in a queue, perform some function associated with the administrative level, then send it to the next relay, again using SMTP. The TCP connections may be over the open Internet, or within a private network. At the level of Figure 1, it makes no difference.

As with the parent article, Figure 1 is only one of many posible systems. Most systems have many more relays than shown here, a fact which we illustrate with the dotted line between two of the relays. You can see how many relays handled a message by looking at the lines labeled "Received:" in the message header. There should be one for each relay. Often the purpose of these relays is unclear to anyone outside an organization. It may be something as simple as consolidation of the mail from multiple departmental servers to one corporate server.

In this article, we will briefly explain the processes and protocols shown in Figure 1, and how the system tries to provide reliable delivery and deal with the failure modes known at the time it was designed. Finally, we will introduce some of the acronyms and jargon needed to understand the literature on email systems.

Mail Handling Processes

With the exception of the Author and Recipient processes, all the processes in Figure 1 are relays. They receive, store, and forward messages to the next relay, or to a final destination. In this discussion, however, we will use the term Relay to mean just a simple SMTP relay process.

Relay processes are spawned by programs like Sendmail or Postfix, typically one process for each message. This allows most messages to be processed in seconds, even while there are a few messages taking minutes to process. A busy server might have hundreds of Relays running simultaneously, each Relay receiving a message one packet at a time, waiting for an outgoing connection, or performing the unique administrative function that was assigned to it.

The unique processing at each Relay is typically done by a subprocess or plug-in, written independently of the Relay, and called through a standard interface to the Relay. This allows programmers with expertise in each specific function to focus on their own program and not worry about all the details of message handling that are common to every Relay.

Why not make every process in Figure 1 a simple Relay with a plug-in? It doesn't work when there are requirements not satisfied by SMTP. The Delivery process doesn't use SMTP at all. This is because SMTP is a "push" protocol. The sending relay intitiates the transfer, and the receiving relay is expected to be always on line. This is not the case with most Recipient processes, however. A Recipient may be off line for days. The final transfer from the Mailstore to the Recipient's machine must be done at the Recipient's request. This requires a "pull" protocol (POP, IMAP, or HTTP).

The Submission process could use a standard Relay with SMTP over port 25 (and many do), but there is an advantage to using a distinct process with a unique port (587 instead of 25). Messages arriving on port 587 are all submissions, not a mix of submissions and transfers. Connections on port 587 are always authenticated. The submission process can expect it, and the Author process must always provide it. The parentheses around (SMTP) in Figure 1 show that the protocol on this connection is a modified version of SMTP known as the Submission Protocol.

A Gateway process looks like an SMTP Relay on the SMTP side, but is designed to transparently move messages from an SMTP environment to some other environment. There is less need for gateways, now that almost all email services are using SMTP.

The Mailstore provides "permanent" storage for the Recipient's messages. This is different than the temporary storage at each Relay, where messages wait in a queue only as long as it takes to get an available connection to the next Relay. The final Relay process is shown running on the same machine as the Delivery process. Both must access the same Mailstore.

The Zombie process is not part of the email system, but we include it in Figure 1 for discussions on security. A zombie process typically runs on a home computer, without the knowledge or consent of the computer's owner. It imitates a legitimate Relay, sending exact replicas of legitimate SMTP commands and message headers, faking everything but the IP address of the legitimate Relay. There are millions of these zombies on the Internet, organized into huge "botnets" controlled by hidden "botmasters". The most common defense is to block connections from any IP address known to be a zombie. New zombies keep popping up so fast, however, that spam is a thriving industry, in spite of the IP blacklists.

SMTP and other Mail Transfer Protocols

Message transfer protocols define the interactions at each "hop" in moving a message from Author to Recipient. At one time, every Email Service Provider (ESP) had its own protocols, and could not exchange messages with other ESPs. This forced users to have multiple accounts, one at each ESP having a Recipient to whom they would like to send a message. These proprietary protocols have now been replaced by three universal standards, SMTP, POP, and IMAP. Here is a brief explanation of SMTP, the Simple Mail Transfer Protocol.

Message transfer via SMTP is done by establishing a TCP connection between a Client SMTP process initiating the transfer and a Server SMTP process receiving the message. The initial transfer is done using a Client on the message Author's machine, typically a part of his email program. Intermediate transfers involve Relays having both Server and Client functionality. The final transfer to a Recipient's machine uses the POP or IMAP protocols. SMTP "pushes" messages to the next Relay. POP and IMAP "pull" messages from a mailstore at the destination.

After establishing a TCP connection, the message transfer is guided by a sequence of plain-text commands from the Client and reply codes from the Server. The purpose of these commands is to provide "envelope" information so that the message can be handled without having to read its contents. This separation of function allows the email system to work reliably and efficiently, without putting any constraints on the content or syntax of the message itself. The content may even be encrypted, making it totally unintelligible to the mail handling system.

A good way to understand the SMTP process is to send a small message and issue the commands manually using the 'telnet' program available from the command prompt window on most machines. Telnet is a general-purpose program for communication using TCP. Using telnet, you should be able to connect to your email service provider on port 25, the standard email service port (or port 587 if your provider expects you to authenticate your identity).

Here are the steps in a typical message transfer: 

1) Establish a TCP connection.
2) Establish a mail session.
3) Provide a Return Address for the next message.
4) Provide a Recipient Address for the next message.
4a) Repeat step 4 for additional Recipients.
5) Transfer the message and all its attachments.
5a) Repeat from 3 for additional messages.
6) Terminate the mail session.
7) Close the TCP connection.

For more details on this process, see the related article on SMTP.

Post Office Protocol (POP)

Internet Message Access Protocol (IMAP)

Webmail and the HyperText Transfer Protocol (HTTP)

Reliability Concerns

Reliable operation of an email system requires that there be feedback to an Author, either a notice that something has failed along the way, or a confirmation that the message was displayed on the Recipient's machine. Bounce messages provide the notice of failure. Return Receipts provide confirmation that the message was displayed, even if not read.

Bounce messages can be generated by any process that fails to handle the message as expected. This could be a Relay that cannot make a connection to the destination server, a Recipient whose mailbox is full, or any number of other causes that fall into the "permanent" category. Temporary failures that last only a few hours are not generally reported, as those are considered normal for email.

Bounce messages are sent via SMTP to the Return Address provided in step 3 of the procedure above. This need not be the same as the Author's address, but most often it is, because each Relay simply copies whatever was provided with the incoming message. Some Agents prefer to take a more active role, however, and be notified of any downstream problems. This can be done by "re-writing" the Return Address so that bounces are intercepted.

Re-writing of Return Addresses is also done to avoid spam disguised as a bounce. By adding an authentication code to the Return Address, it is possible to automatically discard any bounce message with an invalid code. See the documents on SRS and BATV for details. Unfortunately, not many Agents do this, preferring instead to report the sender of the bounce as a spammer. This has led to a practice of simply discarding messages without notice to anyone. The result has been a terrible degradation in the reliability of email. Authors can no longer be confident that no bounce means their message was delivered.

Bounce messages are considered a "last resort" to be used when a Relay has accepted a message, and only later discovers it cannot be delivered. Simple errors, like no-such-recipient, should cause an immediate reject during the SMTP session. Rejects avoid the problem of fake Return Addresses, but may still leave the Author uninformed, if the reject occurs "downstream" in the chain of Relays. Rejects cannot be propagated "upstream", because SMTP sessions are terminated at each hop once after the message is accepted.

The move away from bounce messages has led to greater reliance on Return Receipts. An Author can request a Return Receipt by adding a line to the message headers 

  Disposition-Notification-To: author@example.com

Most email programs have a button or menu command to do this, and a pop-up window that appears to the Recipient when a message with this line is displayed. The Recipient then has an option to ignore the request, or click a button that sends the confirmation. Return Receipts are sent just like normal messages to the address in the request.

Email System Terminology

Term Definition
MUA Mail User Agent; a program such as Mozilla Thunderbird, Microsoft Outlook or MUTT that can be used by an Author to compose and send, or by a Recipient to retrieve and read email messages. An MUA can also be a process, such as found in many web browsers or mobile phones.
MSA Message Submission Agent; a machine, program or process that receives messages directly from an MUA, and performs the functions of an Agent responsible for mail submission.
MTA Message Transfer Agent; a Relay; a machine, program or process that receives and stores a message, performs some administrative function, and forwards it to the next Relay or Mailstore.
MDA Message Delivery Agent; the server that accepts mail for a user from a remote MTA and holds it until the user's mail client (their MUA) downloads the message
DSN Delivery Status Notification (aka Bounce); A message generated by any process handling a message in transit, and sent to the Return Address. DSNs are typically sent when a permanent failure occurs in transit.
MDN Message Disposition Notification (aka Return Receipt); A message generated by the recipient's MUA, providing information on post-delivery handling, such as notice that the message has been displayed.
SMTP Simple Mail Transfer Protocol; the protocol used to transfer mail from one mail system to another. Uses port 25 or 587 for unencrypted message transfer.
POP Post Office Protocol; A protocol where a client connects, downloads mail from the server and then deletes that mail from the server. Mail that is downloaded then "sticks" on the computer the user retrieves their mail from. Contrast with IMAP.
IMAP Internet Message Access Protocol; IMAP differs from POP in that messages are left on the server; this allows a user to "float" between different clients at different locations but still have access to all their mail

Table 1 shows some acronyms commonly used in discussions of email systems. Four of these (MUA, MSA, MTA, and MDA) may be a bit confusing. The confusion arises over the word agent, which is used to mean a process, a program, or a machine, rather than an individual or organization. The use of plain English words like actor and agent to mean something else is common in computer science. This distortion of meaning occurs naturally, as specialists try to be concise in their communications. Instead of saying "MSA process", meaning the process that performs the functions of an Agent that handles mail submission, it was easier to just use MSA as an acronym. It works as long as the context is clear.

In these articles, we talk about the whole spectrum of "agents", from organizations and individuals to programs and processes, so we can't just use the common jargon without clarification. We also need to make distinctions based on function, like transmitter vs receiver, instead of just calling every relay an MTA.

Notes

Related Articles

Email System
Message Formats
Email Authentication

Bibliography

[Crocker09] D. Crocker (2009) "Internet Mail Architecture", RFC-5598, http://ietf.org/rfc/rfc5598.txt.

[Gellens06] R. Gellens, J. Klensin (2006) "Message Submission for Mail", RFC-4409, http://ietf.org/rfc/rfc4409.

[Klensin08] J. Klensin, ed. (2008) "Simple Mail Transfer Protocol", RFC-5321, http://ietf.org/rfc/rfc5321.txt.

[PnD07] L. Peterson, B. Davie (2007) "Computer Networks: A Systems Approach" 4th ed. Sect. 9.1.1 "Electronic Mail", ISBN 0-12-370548-7.

[Resnick08] P. Resnick, ed. (2008) "Internet Message Format", RFC-5322, http://ietf.org/rfc/rfc5322.txt.

[Stevens94] W.R. Stevens (1994) "TCP/IP Illustrated, vol. 1, The Protocols", Chapter 28, "SMTP", ISBN 0-201-63346-9.

[Wikipedia08] Simple Mail Transfer Protocol. More details on SMTP, including history, abuse, and related protocols.

Possible Additional Topics

  ESMTP - RFC-5321
  Port 587 - RFC-4409
  Reply Codes
    550, 450 - greylisting
  Options
    MAIL FROM
Retrieved from "https://citizendium.org/wiki/index.php?title=User:David_MacQuigg/Sandbox/MailTransfer03&oldid=665302"