High Performance Message Processing with BMT Message-Driven Beans and Spring
Pages: 1, 2, 3, 4, 5

Redelivery and Retry Strategies

The above aspects will properly roll back transaction in case of a processing error (including application errors and/or failures on transactional resources such as JDBC, JMS, or JCA). Usually the messaging server (such as WebLogic Server JMS, MQseries) will redeliver messages, allowing components to retry, which may end up with an endless loop in the case when we get a "poisoned" message that can't be processed for whatever reason.

To cover this case we will have to put a limit on how many times we need to try to process the message. All enterprise messaging servers allow you to configure the number of retries for each queue and also to define an action to take when the number of retries is exceeded (for instance, move messages to the special "dead letter" queue). It could happen that many messages can get into the "dead letter" because of the outage of a database server or other required resource. So, once an outage is resolved, these messages could be moved back to the processing queue for another retry. Several enterprise monitoring applications support such a feature.

If for some reason using a "dead letter" queue is not acceptable, you can implement a trivial guardian advice on the application side. This can also be implemented as an advice at the beginning of the MessageProcessor execution flow.

The actual implementation could vary depending on what you want to do with the failed messages, but you have to take into account that it is quite possible that some resources won't be available during failures—for instance, the database may not be available at this point. The following example will dump failed messages to the log file with fatal severity. It won't be difficult to implement a utility to read such a log and resubmit the messages.

public final class MdbRetryCountAdvice 
                               implements MethodInterceptor {
  private Log log = LogFactory.getLog(getClass());
  private int retryCount;

  public void setMaxRetryCount(int retryCount) {
    this.retryCount = retryCount;
  }
  
  public Object invoke(MethodInvocation methodInvocation) 
                                               throws Throwable {
    Object o = methodInvocation.getArguments()[0];

    try {
      return methodInvocation.proceed();
    } catch (Throwable e) {
       
                         MessageData data = (MessageData) o;       if (data.getDeliveryCount() >= this.retryCount) {         this.log.fatal("Retry count exceeded. Message discarded."                                                       + data, e);         return null;       }       // assume business logic logged exception already        // so no stack trace here.       throw e;
    }
  }
}
                      

As an aside, since you have a mechanism to handle message duplicates you may consider deploying the MDB with a DUPS_OK_ACKNOWLEDGE mode instead of the default AUTO_ACKNOWLEDGE. Theoretically speaking, this should increase performance because the MOM could spend fewer resources on ensuring no duplicates in delivery. The actual performance impact will be different for different MOM providers so we would recommend running some load tests to see real numbers. In the case of IBM WebSphere MQ we didn't notice any measurable performance difference between these two modes, but your mileage may vary.

Business Objects Pooling

Since our pattern relies on throwing a RuntimeException in case of an error during processing of the message, it's important to understand that, according to the EJB specification, the J2EE server should discard the MDB instance in this case. In most cases this is not a problem since in our implementation the concrete MDB is quite lightweight—there is hardly any code at all. You have to be more careful with actual business objects that the MDB is using though. Sometimes you cannot use singleton implementation of business logic for various reasons (one of the most common cases is when you're using XML parsers in your business logic processors that generally are not thread safe). In this case you would have to define the business object as a prototype instead of a singleton in the Spring context, since it could be called from multiple instances of MDBs simultaneously.

Our design of the MessageDataDrivenBean class is already optimized to handle non-singleton beans; as you can see, bean lookup is done only once per MDB instance, in the onEjbCreate() method. Because the EJB specification guarantees single-threaded execution of MDBs, you should be fine using non-singleton lookup and save bean instantiation cost on every invocation.

<!-- Note that this defines a prototype -->  
<bean id="simpleProcessorImpl" singleton="false"
      class="org.javatx.mdb.SimpleMessageProcessor"/>

<bean id="simpleProcessor" 
      class="org.springframework.aop.framework.ProxyFactoryBean">
  <property name="targetSource" ref="simpleProcessorImpl"/>
  <property name="proxyInterfaces" 
            value="org.javatx.mdb.MessageProcessor"/>
  <property name="interceptorNames">
    <list>
      <idref local="mdbRetryCountAdvisor"/>
      <idref local="mdbTransactionAdvisor"/>
      <idref local="mdbDuplicateHandlingAdvisor"/>
      <idref local="messageProcessorPerformanceMonitorAdvisor"/>
      <idref local="messageProcessorTraceAdvisor"/>
    </list>
  </property>
</bean>

If you want to optimize this even further, to avoid excessive creation of business logic bean instances (for example, XML parsers and other objects expensive to create), you can use Spring's support for pooling target sources. Spring can maintain a pool of identical instances with every new method invocation going to free objects in the pool. Again, this can be achieved without any changes in your source code. Simply define SimpleProcessor in the application context as non-singleton, define a pooled target source (you can specify the pool size and other pool parameters there), and then change the simpleProcessor proxy definition to use the pooled target source:

<!-- Note that this defines a prototype -->  
<bean id="simpleProcessorProtImpl" singleton="false"
      class="org.javatx.mdb.SimpleMessageProcessor"/>

<bean id="simpleProcessorPooled" 
   class="org.springframework.aop.target.CommonsPoolTargetSource">
 <property name="targetBeanName" value="simpleProcessorProtImpl"/>
 <property name="maxSize" value="5"/>
</bean>>  

<bean id="simpleProcessor" 
      class="org.springframework.aop.framework.ProxyFactoryBean">
  <property name="targetSource" ref="simpleProcessorImpl"/>
  ...
</bean>

This simple change can help you to squeeze even better performance out of your application, especially if you expect message processing failures to often occur, because the MDB instance is discarded when a runtime exception is thrown from the onMessage() method together with its dependencies. Note that Spring uses the Jakarta Commons Pool for its pooling implementation, so your application must include the commons-pool jar.

Download

You can download the source code referred to in this article: custom-mdb-processing-source.zip

Conclusion

As you can see, we successfully achieved the goal of handling message consumption non-transactionally, yet preserving the once-and-only-once quality of service. We believe that as a result of switching to BMT, the application developer will have much more control over application behavior and transaction handling while at the same time enjoying significant performance improvements. As a side effect of this transition you don't have to worry if the MOM you're using has robust support for XA transactions if you're only going to consume messages.

While at first sight it may look as though the quantity and complexity of code has increased, as this article has shown, the solution can be implemented in a very structured way using Spring's AOP framework, which allows you to compose individual aspects of the non-functional requirements using independent advices. Each advice is responsible for handling specific pieces of the functionality. However, assembled together in a Spring application context, they can manage all non-functional requirements for the message processing including transaction handling, duplicate handling, and handling poisoned messages, as well as common tracing, logging, and performance monitoring.

References

Dmitri Maximovich is an independent consultant specializing in software design, development, and technical training. He has more than twelve years of industry experience and has been involved with J2EE since its inception.

Eugene Kuleshov is an independent consultant. He has over twelve years of experience in software design and development and specializing in application security, enterprise integration (EAI) and message oriented middleware.