Preferred Practices for Deploying Sun Java System MTA

Wojtek Chemijewski, January 2008

Typically, a Sun Java System Messaging Server deployment consists of front-end and back-end services. The front-end services enable users access to the system, while the back-end services store and provide access to data. In the past, we've heard disputes about how to deploy the MTA on the front end: either as an entrance gateway to the system, or “behind” an anti-spam/anti-virus service that also provides an MTA functionality. This article describes the recommended way to deploy your MTA with respect to anti-spam/anti-virus services.

This article contains the following sections:

Executive Summary

Sun Java System Messaging Server provides a highly reliable, high-performance, full featured SMTP implementation in the form of its Mail Transfer Agent (MTA). This MTA enables you to integrate anti-virus/anti-spam (AV/AS) solutions through the use of plugins. These plugins provide message content to the AV/AS engine for its evaluation and scanning while enabling the MTA to retain full responsibility for the message at the SMTP level. The MTA does not provide the AV/AS engine itself: Messaging Server relies on third-party vendor experts to provide these solutions. Such a best-of-breed architecture, built into Messaging Server, provides you with the greatest flexibility of integration. Thus, you use the Messaging Server MTA for the SMTP message processing and an AV/AS solution of your choice for virus and spam processing.

Messaging Server Architecture Overview

A two-tiered messaging architecture provides the optimum design for scalability and reliability. Instead of having a single host run all the components of a messaging system, a two-tiered architecture separates the components onto different machines. These separate components perform specific specialized functions. As the load for a particular functional component increases, you can add more servers to handle the larger loads. Additional servers can accommodate, for example, more Message Storage, or more outbound relaying.

The two-tiered architecture of Messaging Server consists of an access layer and a data layer.

The access layer is the portion of the architecture that handles delivery, message access, user login, and authentication. The access layer is also called the front-end layer or tier. For a Messaging Server deployment, the access layer is composed of the MTA, Messaging Multiplexor (MMP), and Webmail server. AV/AS services are also part of the access layer.

The data layer is the portion of the architecture that holds all the data. The data layer is also called the back-end layer or tier. This layer includes the LDAP master servers and Messaging Server machines that are configured to store user messages.

For more information about Messaging Server architecture, see

More About the MTA and AV/AS Services

The MTA is one of the Messaging Server access layer services. The MTA receives, routes, transports, and delivers mail messages by using the SMTP protocol. An MTA delivers messages to a local mailbox or to another MTA. The anti-virus/anti-spam check is performed on these mail messages along their entry to the system. The preferred way of integrating AV/AS services with the MTA is by accepting messages from the users and the Internet, providing them to the AV/AS software for a check, and then relaying these messages further to other users or to the Internet as applicable.

AV/AS software vendors usually provide some MTA functionality in their products. Furthermore, AV/AS software vendors tend to recommend that you deploy their software as the entry point to the system, and that their software relay messages to the MTA after they have been checked for spam or viruses. As this article shows, deploying your AV/AS software in such a manner—in front of the MTA—is a suboptimal method. Indeed, if you opt for such a deployment, you cannot even access certain MTA features. Furthermore, some MTA features used in such an architecture can cause a performance degradation.

Deploying AV/AS Services With Messaging Server

To begin with, let's examine the two fundamental ways of developing a Messaging Server architecture for AV/AS so that the MTA is the component to first accept messages:

  • MTA Plugin (Integrated Approach). You deploy the AV/AS services through an integrated MTA plugin, which is dedicated for the specific AV/AS vendor. Such plugins usually work through a proprietary protocol for providing messages to the AV/AS scanner, and that scanner returns its verdict to the MTA. The MTA then uses this verdict for further processing of the message.

  • SMTP Relaying. If no AV/AS plugin is available, you can configure the MTA to accept the message first, and then relay it to the AV/AS service by using SMTP. The AV/AS service should then relay the message back, adding information with the result of the scanning into the message headers. The MTA can then access that information and use it as the verdict for further processing of the message.

Whenever possible, use the integrated approach when developing your AV/AS deployment for Messaging Server. The integrated approach provides more flexibility in the way messages are chosen for scanning. In addition, with the integrated approach, the MTA retains all responsibility for the message while it is being processed.

The integrated approach also provides only the content of messages to the AV/AS service for evaluation and verdict. Thus, no SMTP relaying is involved in providing the message for AV/AS processing, and so the AV/AS scanner does not need to manage queues nor generate non-delivery reports. Finally, with the integrated approach, the set of ESMTP extensions supported by the MTA of the AV/AS does not influence the extensions that the clients can use. When an MTA relays messages to another MTA, it first queries for the protocol extensions that the other MTA supports. In the case of the Messaging Server MTA, when a client first submits a message to the Messaging Server MTA, it queries for the set of supported extensions. Next, the Messaging Server MTA uses the DSN extension to request a successful delivery report. Finally, the Messaging Server MTA relays that message to the AV/AS service and discovers that the AV/AS MTA does not support the DSN extension. Thus, a request for a successful delivery report cannot be relayed any further. In the case of the MTA plugin, the responsibility for message processing remains with the Messaging Server MTA, so no such effect happens.However, the integrated approach is possible only with the AV/AS entities, for which respective plugins exist.

MTA Functionality Lost Due to Lack of Source IP Address Information

When you deploy the AV/AS services “in front” of the MTA, the MTA does not have access to the IP address of the sending client. Instead, it has access only to the IP address of the AV/AS box. The result is that the MTA cannot make make any decisions (that is, possibly reject mail) based on the client IP address. The following situations describe how knowing the source IP address of the sending client is useful:

  • Connection Throttling, Possibly Across Multiple MTAs. The MTA can throttle connections based on the source IP address. Throttling means limiting the number of connections that a particular IP address can open to the MTA within a certain time period. You can configure the MTA to allow only a certain number of connections to be open within one minute, and reject all connections exceeding that limit. You can also penalize an IP address that opens more connections than allowed. The MTA continues to reject such an IP address for some extended time, even if the rate drops below the limit.

    In Messaging Server 6.3, you can also deploy the Metermaid server, which hosts a throttling database that you can use commonly across multiple MTA servers. With Metermaid, you can control how many connections a certain IP address opens, even if the IP address tries to connect to multiple MTAs. These attempts are quite possible, due to the the front-end tier's usual horizontal scalability.

  • Sender Policy Framework (SPF). Spam producers and email scammers often forge email by using false domain names and email addresses or by using legitimate domain names and email addresses in order to lead users to believe that a message is from someone or some company they know. For example, a spammer could send email from an address such as and the user could be led to believe the mail was actually from this address. A forged email might lead users to open the unsolicited message, or worse, provide information to a false authority. Also, spammers prefer to send their email from legitimate domains that are not on a Realtime Blackhole List (RBL).

    Sender Policy Framework (SPF) is a technology that can detect and reject forged email during the SMTP dialogue. Specifically, SPF is a protocol that allows a domain to explicitly authorize the hosts that may use its domain name. In addition, a receiving host might be configured to check this authorization. SPF can thus significantly reduce the instances of forged email. If the IP address is not known, the SPF check cannot be performed. For more information about SPF, see

    The Messaging Server MTA supports SPF in a flexible way, enabling it to influence the action that is performed on the SPF result or to incorporate the SPF result into further (for instance, per-user) processing. Messaging Server provides this capability so that you do not have to depend on the AV/AS, which might or might not be as flexible in handling complex scenarios.

  • Other DNS-Based Checks. The Mail Abuse Protection System's Realtime Blackhole List (MAPS RBL) is a dynamically updated list of known unsolicited bulk email (UBE) sources identified by source IP address. The Messaging Server SMTP server supports use of the RBL and can reject mail coming from sources identified by the RBL as originators of UBE or spam. You an use the result of such a check to reject mail. You can also use this check to influence how the MTA further processes the message. For example, you can increase the likelihood of the message to be spam through a positive RBL check, and you can use that value in a user-based Sieve filter.

  • IP-Based Channel Switch. The MTA can switch the source channel based on the client IP address. Typically, all mail arriving on port 25 of the server is enqueued to the tcp_local channel, and characteristics of that channel are applied to the connection and to the mail submitted on that connection. However, it is possible to switch the connection to another channel based on the client IP address and apply different characteristics if the client's IP address belongs to a certain pool. Examples of these characteristics include the following:

    • Whether encryption is required

    • Whether sender authentication is required

    • Whether the source channel should be further switched based on the fact that connection is encrypted or authenticated

    • Whether ETRN is allowed, and such settings as the maximum message size, and the recipient count limits.

    You can also deploy source-channel-specific rewrite rules so that addresses in messages being switched to a specific channel are rewritten differently than the rest. You can use the source channel in mapping tables, which enable you to deploy regular-expression-based ways of processing messages. The most common example of this functionality is the default behavior of mail coming from clients whose IP address belongs to the value specified in the INTERNAL_IP mapping table: the mail is switched to the tcp_intranet and allowed to be delivered over the Internet.

  • Using IP Address in Mappings. The client IP address can be directly used in mapping tables, which the MTA accesses at various stages of mail processing. You can use the IP address in the following mapping tables:

    • PORT_ACCESS — To make decisions based on IP addresses

    • FROM_ACCESS — To make decisions based on the IP addresses, application information, source channel, envelope From: address, and authentication information

    • MAIL_ACCESS and ORIG_MAIL_ACCESS — To make decisions based on IP addresses, application information, source and destination channels, and envelope From: and To: addresses

  • Logging. You can configure the MTA to log the sending client IP address. Analysis of the MTA behavior and troubleshooting problems is complicated because of the inability to capture and log this IP address because of the difficulty of locating the AV/AS service in front of the MTA.

Functionality Lost Due to Lack of Authentication Information

It is possible to configure an AV/AS entity to provide the means for SMTP authentication (using LDAP bind, for instance), and to relay mail to the MTA without authenticating. In such cases, the MTA does not have access to the authentication information. Therefore, you cannot use MTA functionality that requires users to authenticate or to modify the MTA's behavior based on the authentication information. Examples of lost functionality in this case include the following:

  • Authentication Enforcement. You can enforce authentication to distribute mail to the Internet (the typical setup). You can also enforce authentication on certain source channels. Knowing the sending client's IP address enables switching the source channel based on the client IP. Knowing that the users authenticated enables you to enforce authentication on messages from certain IP address ranges.

  • Per-user Channel Switch. If you provision a user with the mailSMTPSubmitChannel LDAP attribute, the source channel can be switched to a channel specific for that user. For the MTA to do so, the user must authenticate. After the switch is complete, this particular channel's characteristics will be applied to all mail originating from that user.

  • List Submission Authorization. You can provision a mailing list with attributes that specify who can send mail to that list, for example, mgrpDisallowedBroadcaster, mgrpAllowedBroadcaster, mgrpDisallowedDomain, and mgrpDisallowedDomain. The authentication information can be used in these checks.

  • Message Headers. The fact that the user authenticated is put into the Received: header of a message. Also, you can configure the MTA to put the Sender: header line, containing information about who authenticated to send mail, into the message. This behavior can be channel specific, so the ability to switch the channel based on an IP address of the client might be of use here as well.

  • Logging Who Authenticates. The authentication information (information who authenticated) can be recorded in log files.

Other Functionality Lost

When the AV/AS relays messages to the Messaging Server MTA after having accepted them first, the claimed EHLO/HELO name might be lost. You can use the EHLO/HELO name in various mapping tables that the MTA consults during message processing as part of the application information. This usage applies to FROM_ACCESS, MAIL_ACCESS, and ORIG_MAIL_ACCESS.

The behavior of the MTA can be modified based on the fact that the client used ESMTP, or that the encryption (TLS/SSL) was used. This information is also lost if the AV/AS connects to the MTA and uses ESMTP and no encryption persistently.

Support for SMTP Extensions

The Messaging Server MTA supports a rich set of SMTP extensions. Clients can use these extensions to influence the way that messages are processed by the MTA, or to increase their efficiency. If an AV/AS service supports a lower number of ESMTP extensions than the Messaging Server MTA, clients forfeit the ability to access the MTA extensions when the AV/AS service is placed in front of the MTA.

The significance of SMTP authentication (which is one of the SMTP extensions) has already been discussed. Other interesting SMTP extensions include the following:

  • SIZE (RFC 1870). Enables clients to declare the size of a message and give the server the opportunity to reject messages greater than its limit before the message data is even sent. This precaution reduces network bandwidth.

  • DSN (RFC 3461). The server generates standards-compliant notifications. It can also be used by clients to specify conditions in which notifications should be sent back and whether notifications should contain entire original messages.

  • STARTTLS (RFC 3207). Enables more secure transfer of messages.

  • ETRN (RFC 1985). Enables clients to request remote queue starting.

  • CHUNKING (RFC 3030). Enables transferring binary messages, which can reduce the bandwidth use.

  • PIPELINING (RFC 2920). Enables more asynchronous processing, which can improve network throughput.

  • 8BITMIME (RFC 1652). Enables transferring unencoded 8-bit data, hence can improve throughput.

  • NO-SOLICITING (RFC 3865). Reduces unsolicited messages.

Rejecting Mail

According to the SMTP standards, if an MTA is trying to relay mail to another MTA and the destination responds with a rejection, the sender is mandated to perform certain actions, including the following:

  • If the rejection is permanent, sending a Delivery Status Notification (DSN) message to the original sender of the mail.

  • If the rejection is temporary, storing the message in its queue and retrying it later. After a certain number of unsuccessful retries, the MTA is supposed to send the DSN message to the original sender.

The potential problem with this process is that the DSN messages might not be deliverable themselves. Also, retrying to send messages, which are not deliverable, wastes MTA resources: computing power, bandwidth, and outgoing threads. Therefore, the MTA should do all it can to ensure that it does not accept messages that are undeliverable in the first place.

The problem worsens if you have multiple levels of MTAs in your deployment. The MTA that is exposed to the Internet might accept certain messages, which the next relay MTA would reject. Then, if the rejection is permanent, that first MTA is supposed to send the DSN messages to the original senders, which would possibly waste its resources. If the rejection is temporary, the first MTA should retry sending the message for some time, and then return a DSN.

Thus, the preferred practice is for the first MTA (the one exposed to the Internet) to reject more mail than any subsequent MTA in the environment. That is, the first MTA is “less liberal” in accepting mail. If the architecture that places the AV/AS MTA service is exposed to the Internet, and the Messaging Server MTA is the next relay hop, the AV/AS service should be able to reject mail that the Messaging Server MTA would reject.

This type of deployment, in turn, might not be possible. The reason is that the Messaging Server MTA would reject mail in many cases that the AV/AS MTA cannot detect. First, the LDAP support in Messaging Server MTA is quite complex, and therefore flexible. The Messaging Server MTA does not just check the existence of a user, or a domain in LDAP. Instead, the MTA fetches a series of attributes of that user, and makes decisions based on those attributes. Some of these decisions could include, but are not limited to the following situations:

  • Domains and users might have different statuses, some of them resulting with permanent rejections, some with temporary rejections (like inactive, or overquota).

  • The maximum size of messages accepted is computed based on domain, group and user limits. Messages that exceed those dynamically computed limits are rejected.

  • Recipient count limits can be computed dynamically based on domain attributes.

  • Groups might have submission authorization rules imposed. If the sender of a message fails to meet these rules, the message can be rejected.

Also, the Messaging Server MTA can reject messages with temporary errors in case it detects transient problems during message processing.

Flexibility of AV/AS Integration

The Messaging Server MTA provides the means to integrate with various AV/AS vendors in a flexible and efficient way through plugins that are dedicated to AV/AS entities. This flexibility and efficiency is present in various areas:

  • It is possible to scan the same messages multiple times, with various AV/AS entities (up to eight different entities from a single MTA). You can choose to which AV/AS entity the message should be provided on a per-channel, per-domain, and per-user basis. Thus, you can provide premium, commercial-based scanning to some users, and other means of scanning to different users.

  • It is also possible to integrate with various AV/AS entities (for instance from different vendors), and act on verdicts provided by these various entities in complex ways. For example, this feature enables you to rely more on one AV/AS verdict while making use of another.

  • It is possible to provide per-channel, per-domain, or per-user optins to the scanning engine, if supported by the engine. Thus, the engine understands what kind of checks it is supposed to perform for each user.

  • The AV/AS box can return per-recipient verdicts. These verdicts can be later used to construct per-channel, per-domain, or per-user filters acting on messages based on these verdicts. Thus, specific users can define custom actions that should be performed if a message is a spam suspect. Examples of custom actions include the following:

    • Discarding likely spam messages

    • Moving messages to an alternate folder that might be spam but the user would like to read

    • Tagging a message's subject as potential spam but passing along to the user's inbox

Using a Single User Repository

The previous section described how the Messaging Server MTA might reject mail based on the content of LDAP attributes, as well as the ability of providing per-user and per-domain optin values to the AV/AS based on the LDAP directory content. Although the AV/AS entity might support rejecting mail on per-domain and per-user criteria, or providing per-domain and per-user optins, it might not support using LDAP as the repository for that information. In this case, if you want to take advantage of these functionalities while deploying the AV/AS entity in the front, you must provision per-user and per-domain data into the following areas:

  • The configuration repository of that AV/AS service so that it can use the data for providing the previously mentioned functionalities

  • The LDAP so that the MTA can use it for its operation

Note - Having to provision two different types of repositories at the same time adds unnecessary complexity to the provisioning tools and creates the possibility of data inconsistency between the directory and AV/AS service databases.

Enhancing Performance

Positioning an AV/AS box in front of the MTA can have a negative impact on the overall performance of your messaging system. This impact might manifest itself in directory performance and AV/AS performance.

Impact on the Directory Performance

The following two situations can impact the directory performance:

  • If an AV/AS box is facing the Internet, it must check the recipient's existence in LDAP. The way this check is designed can influence the performance of the directory environment:

    1. First, the MTA checks whether the domain is local and caches both negative and positive results.

    2. If the domain is determined to be local, the mail address is searched for in the directory. This type of search saves the directory service from searching nonlocal addresses. The domain-related search is usually not costly, as the number of local domains is typically small, compared to the number of users.

  • If the Messaging Server MTA is facing the Internet, it checks the domain-related and user-related information in LDAP:

    1. This MTA check includes, but is not limited to, verification if the mail address exists in the database. As described previously, the MTA fetches a significant amount of information that influences how to process messages.

    2. If the AV/AS LDAP integration is limited to verification of the user existence in the database, the MTA check ensures that what the AV/AS requires is known. Therefore, if the MTA is performing its LDAP check first, the AV/AS-related LDAP check is no longer needed. If the Messaging Server MTA passes the message to the AV/AS entity, that AV/AS entity can be certain the user exists and can refrain from performing the LDAP check. This overriden check saves directory server cycles.

    3. If, however, the AV/AS service is facing the Internet, and performs its LDAP check to determine if the user is local, the Messaging Server MTA that is deployed further in the processing chain has to fetch the domain-related, group-related, and user-related information from the directory anyway. So it must perform its checks on the directory regardless of whether it is facing the Internet. Thus, the overall number of LDAP searches is always greater if the AV/AS entity is facing the Internet, compared to the other way around.

Impact on the AV/AS Performance

If you deploy the AV/AS entity so that it faces the Internet, it has to check all messages that are determined to be accepted by this entity. If the Messaging Server MTA, which is deployed further in the relay chain rejects a message that has already been checked, because of the group or user-related parameters, the cycles required for the check have been lost. The AV/AS service is utilized better if the MTA rejects such messages before passing them to the AV/AS service. Thus, the architecture where the Messaging Server MTA faces the Internet and rejects messages before passing them to the AV/AS entity may be more efficient from this perspective.

About DKIM

We have refrained from d describing DKIM in this article because it raises no unique issues of its own in regard to deployment design that aren't already in place for other reasons.

How you perform DKIM checks has no bearing on how you implement your AS/AV solution with respect to the MTA. On the other hand, your implementation of AV/AS services in front of the MTA definitely impacts SPF because of SPF's dependency on IP address information.

Note - DKIM is a signature-based mechanism. The signature validation is performed based on information provided through a header. As such, the system can perform DKIM checks whether or not you place an AS/AV appliance in front of the MTA. An AV/AS appliance can alter message content in a way that breaks the signature, but that is highly unlikely.

For More Information

See the Communications Suite BigAdmin hub for more information about managing Messaging Server:

Comments (latest comments first)

Discuss and comment on this resource in the BigAdmin Wiki

Unless otherwise licensed, code in all technical manuals herein (including articles, FAQs, samples) is provided under this License.

Left Curve
Popular Downloads
Right Curve
Untitled Document
Left Curve
More Systems Downloads
Right Curve