Sending Messages -- Routing and Transport

Windows IT Pro

- Advertise

Sending Messages -- Routing and Transport
View the book table of contents

Author:	Tony Redmond
Published:	October 2000
Copyright:	2001
Publisher:	Digital Press

PROCESSING INCOMING MESSAGES

Messages can arrive into an Exchange 2000 server through a variety of different routes. The initial submission mechanism and processing performed to get the message into the system is different for each route, but they all come together when messages arrive at the categorizer.

SMTP messages
SMTP servers act as clients to other SMTP servers when they transmit messages. Messages are sent to port 25, one of the ports monitored by IIS. A new thread is then created within the SMTP service to process the new message. Protocol events might fire before a message is fully accepted. For example, the sender of the message might be checked against DNS to ensure that the sender comes from a valid server in an attempt to defeat SPAM. If valid, the message can be accepted and the transaction continues.

      A Store driver event is fired when the message is accepted. Currently, Exchange 2000 uses an enhanced version of the standard Windows 2000 NTFS store driver, which is used to access NTFS files, instead of a driver that can access the Store directly using ExIFS. The developers took a deliberate decision not to use ExIFS in Exchange 2000 to avoid creating a dependency on the Store service or more specifically, a database that might not be mounted when a message arrives. NTFS is always available when a server is running, and the drop directory is available if the disk where it is located is mounted. Typically, the mail drop directory is located on the same drive as the Exchange binaries⁸, so if this disk isn’t available Exchange can’t run and an administrator has much more to worry about than messages not being accepted through ExIFS. In addition, directing messages into an NTFS folder rather than the Store avoids problems that might occur when mailbox stores are moved or even deleted.

      The new message is placed on the OnSubmission queue, which fires an event. If an error occurs at this point (for example, a badly formed SMTP message is detected or an unauthorized source has attempted to relay through the server), a failure DSN message (a non delivery notification) is generated and sent back to the originator. If the DSN cannot be generated, perhaps because the sender is unknown, the message is put into the badmail directory (typically \exchsrvr\mailroot\badmail⁹). The message also ends up in the badmail directory if the DSN message is rejected or cannot be delivered.

      The fired event then causes the Transport Core Store driver to create a special object called an ImailMsg and then populate the structure with various properties that are extracted from the incoming message. Similar to an X.400 message, the ImailMsg structure is divided into two main parts. A set of properties such as the sender, subject, and recipients compose the message envelope while a file is created on disk in the Queue directory to hold the message body and any attachments. The Transport Core Store driver returns the pointer to the file created on disk, and this pointer is inserted into the ImailMsg structure. In some respects, the processing so far is very similar to the way that the Exchange 5.5 Internet Mail Service deals with incoming SMTP messages, which are written to files in the IMCDATA\IN directory and placed on a queue managed by the Internet Mail Service.

      After the ImailMsg structure is fully populated, the SMTP service returns it to the Advanced Queuing component, which then places the message in the Pre-Categorizer queue. If an error occurs at any point while an incoming SMTP message is being processed, it is removed from the queue and dumped into the badmail directory (Figure 8).

      If the Queuing engine determines that the message is intended for local delivery, the message must be inserted into the Store. To do this, IIS obtains a file handle to a virtual file and writes the incoming data directly into the file through ExIFS. ESE provides data in 2MB contiguous chunks to ExIFS when it requests space, which then becomes “reserved space” until it is finally used. New messages are written into a set of pages in the reserved space. After each message is received, ExIFS returns the list of pages the message was written to and its properties are promoted into the EDB database. The message transaction is logged into the current transaction log, and after this occurs, the pages in the streaming file are deemed to be committed and are moved out of the reserved set.

      Outgoing messages are handled the same way, being read out by IIS from the Store via ExIFS.

MAPI Messages
MAPI clients do not use IIS and continue to have a direct connection to the Store. MAPI message submission works much as it has since Exchange 4.0 with the notable difference that submission is no longer the point at which non-delivery notifications or delivery receipts are generated. After a message is accepted, the Store puts the message into a special folder called “SendQ” and fires an event to notify the Transport Core Store driver that a new message is waiting. The Transport Core Store driver then processes the MAPI content and builds an ImailMsg structure for submission to the Advanced Queuing component.

      MAPI messages sent between Exchange 2000 servers are rendered in TNEF (Transport Neutral Encapsulated Format). TNEF is a format that can be very quickly transformed back into MAPI properties and content, so this is a low overhead method to transport MAPI messages across SMTP. The messages are formed from an RFC 822 compliant header and an application-specific MS-TNEF MIME bodypart. Unlike TNEF messages sent between Exchange 5.5 servers, no plain text version of the content is included. Exchange 2000 servers communicate using SMTP, which implies that the TNEF would need to be rendered into MIME before it can be transported. The problem is solved by transporting the messages as if they were binary content, so they don’t need to be converted. Messages that are destined outside the organization are converted into MIME using the base64-coding scheme, so they end up between 25-30% larger.

X.400 messages
Similar to Exchange 5.5, inbound X.400 messages are handled by the MTA. The Exchange 2000 MTA does not attempt to perform any routing or assessment of addresses, such as distribution list expansion. After the message is accepted by the MTA, it is simply put into the MTA-Out folder within the Store, from where the message is then placed on the OnSubmission queue before it is fetched by the Transport Core Store Driver and given to the Advanced Queuing component, and on to the Pre-Categorizer queue.

Foreign connectors
Any connector that attaches to Exchange maintains its own MTS-OUT folder in the Store. It is the responsibility of the connector to process incoming messages and ensure that they are validly formed and ready for further processing. When the connector is satisfied, it places the message in the MTS-OUT folder, which causes the Transport Core Store Driver to move the message to the OnSubmission queue and hence for processing by the Advanced Queuing component.¹⁰

WebDAV messages
WebDAV messages are generated by web browsers. Like MAPI, WebDAV uses a direct connection to the Store, so submission of WebDAV messages are handled in a similar manner to MAPI.

CATEGORIZATION AND ROUTING

All inbound messages are eventually processed by the Advanced Queuing component and placed on the Pre-Categorizer queue. The messages are then picked up by a thread running in the Categorizer, which takes responsibility for onward processing. A number of operations are then performed on the message:

The address of the message sender is extracted from the envelope. A search is then conducted against addresses held in the AD (using the proxyAddresses attribute) to determine whether it can be resolved.
Each entry in the recipient list held in the envelope is examined and a lookup performed against the AD to determine whether the recipient can be resolved. If the recipient is a distribution group, the group is expanded to include the members of the group in the recipient list, but only if expansion is allowed on the server. If a recipient cannot be resolved, they are marked as “unknown.”
A check is performed to ensure that the resolved recipients can accept messages, according to any restrictions that might currently apply for the recipients, such as quota exceeded on a mailbox.
Any per-sender and per-recipient limits are checked.
Internet message format settings are checked and properties set to ensure that recipients receive messages in a format they can deal with.
Occasionally multiple copies of a message may need to be created because different recipients must see different versions of the message. A good example of where this happens is if one recipient can be found in the AD and has a mailbox on an Exchange server and another cannot. In this instance, Exchange creates one version of the message for delivery inside the organization (which might, for example, send TNEF content as a binary bodypart) and another for delivery outside (which might have the same bodypart encoded in base64).
Each recipient is then marked as either “gateway” or “local.” Gateway recipients are reached via the MTA, while “local” recipients have mailboxes on an Exchange 2000 server in this organization.

      After categorization, messages are placed in a pre-routing queue and the routing event is called to create a “next hop identifier” for the message. The next hop identifier tells the queuing engine which per-domain queue to handle the message.

Moving Messages off Queues
Recipients on other Exchange 2000 servers can be identified by examining the HomeMDB attribute in their AD entry. If this attribute holds the distinguished name of a mailbox store, then the message can be delivered to the recipient by routing the message to the server that hosts that mailbox store. The routing engine can therefore examine the routing table to determine how the message should be routed and write this information into the next hop identifier. If the mailbox store is located on the same server, then the delivery should be made locally and the message is placed in the local delivery queue.

      When you view a queue, you’re looking at data structures that are maintained in memory. The actual messages that form the contents of the queues remain in the Store or an NTFS directory (\exchsrvr\mailroot\queue — see Figure 9) until they are transmitted.

      Delivery to other Exchange 2000 servers connected in the same Routing Group or the Routing Group Connector is made via SMTP, as are messages going to external SMTP addresses. Messages are placed on the relevant queues and picked up by the connectors for dispatch.

      Figure 10 shows how the Queue Viewer lists messages waiting on queues. All the queues shown here are per-link queues, so they reflect the next step in the routing path for messages. The names used in the queues are the connectors that will transfer the message the next step along its journey. You might expect that the Queue Viewer would list domain queues so that you can see all of the messages going to a particular final destination, but the link queue is actually more interesting, because it shows you queued messages waiting to depart from a specific server, and you know where the messages are destined to go next.

      There is a situation when two queues for the same link might be seen. This occurs when routing information has changed – a new connector might have been added, the properties of an existing connector adjusted, or the Link State Table has been modified because of a network problem. In this situation, Exchange invalidates the old queue and reroutes messages from the old queue to other queues, one of which might have been created for the same link. The old queue is then cleaned up and removed.

      Normally, if all network links are available and messages are flowing freely, very few messages should be seen in the queues. The SMTP transport processes messages very quickly and there should be little delay between a message being placed on the queue and its transmission. Because they don’t handle directory replication traffic, Exchange 2000 queues are much smaller than Exchange 5.5 queues. Some Exchange 5.5 servers, normally those in hub sites, process hundreds of thousands of directory replication messages daily; all of the replication message traffic is now under the control of the AD. This is not an excuse for Exchange administrators to ignore directory replication and close attention must be paid to monitoring the flow of these messages too. Remember, the GAL and configuration data come from the AD, so if it doesn’t replicate correctly, the GAL will be obsolete and routing information may be incorrect. Exchange 2000 ensures that routing information is as accurate as possible by replicating configuration data along with other link state information. If you change the configuration of a connector, the changes are replicated along with other link state updates, so they will get through if the servers can contact each other via SMTP, irrespective whether AD replication works properly.

      Potential still exists to swap message queues with large quantities of public folder replication traffic, especially if Internet newsgroups are replicated around an organization. Keep an eye on the queues, and take action to restrict public folder replication schedules if queues build up and seem to include a lot of public folder traffic.

      Of course, there are reasons why messages build up. The most common cause is that a server is unavailable because it’s down or the network link is saturated and a connection cannot be made. Other less common reasons include the possibility that a server might be underpowered and unable to provide the raw processing power to deal with messages sent to it, or the sheer number of messages has exhausted the capacity of the SMTP transport to process them quickly enough to reduce the queues. The messages might be very large and require a substantial amount of time to transfer to the target server, or the target server might be slow to process incoming traffic. The most common problem is that a network link or server is unavailable when Exchange attempts to make a connection. In Figure 10, we can see that the queue for “qemea-dc1.qemea.qtest.cpqcorp.net” is currently in a retry state, meaning that the SMTP service has failed in an attempt to transfer messages to this destination and is now waiting to make a retry. Viewing the properties of the queue, as seen in Figure 11, reveals more detail about the likely the cause of the problem. In this case, we can see that the remote server did not respond to a request to send a message.

      Any time a domain is unreachable, the SMTP service registers a warning in the system event log. You can use these events to determine what domains are proving troublesome and take steps to try to establish better connectivity. Sometimes problems are purely transitive, and result from servers going offline or network errors. Other problems may point to an issue in your infrastructure, such as DNS updates that are not properly applied or propagated, or perhaps a server that is not routing messages to the correct smart host.

⁸ A Knowledge Base article describes how to move the drop directory away from the default location.

⁹ This is the default “bad mail” directory for the default SMTP virtual server. You can change it through the “Messages” property page for the SMTP virtual server.

¹⁰ A problem with the Lotus Notes connector means that it does not operate correctly in a routing group that contains mailbox servers. You therefore need to create a special routing group to host the Lotus Notes connector, which should run on a server that has no mailboxes. The problem is fixed in Exchange 2000 SP1.

Page: 1, 2, 3, 4, 5, 6, 7, 8

ADS BY GOOGLE