by Mihir Kulkarni
The BEA WebLogic Server 8.1 EJB container provides a rich set of parameters for tuning EJB applications. These parameters are exposed in the WebLogic-specific Deployment Descriptors, namely
weblogic-cmp-rdbms-jar.xml. The WebLogic documentation and Deployment Descriptor DTDs provide a comprehensive description of these tuning parameters, the valid values, and how they can be set.
Tuning your EJB applications requires knowledge of these parameters, and setting them appropriately can be a challenging task. From my experience working in the field with J2EE developers, our customers, and support engineers, I believe it's important to explain these parameters from an application point of view. In this article, I will do this with a use case-based approach and describe how to tune them under different scenarios. Some of the internal workings of the EJB container, along with how they are affected by these tuning parameters, are explained. This should provide developers and administrators with valuable information to tune their EJBs, to make more efficient use of server resources, and therefore achieve optimum performance for their EJB applications.
This is the first of a set of two articles and will cover tuning EJB applications specific to stateless and stateful session beans. The second article will focus on the Entity beans, CMP beans in particular. This article has been written with reference to the EJB container in WebLogic Server version 8.1, Service Pack 3.
Stateless session beans are session beans that do not maintain any conversational state on behalf of clients. As there is no state maintained for the caller, every instance of the stateless session bean is equivalent to every other. The instance does not have an identity and any instance can be used to satisfy a large number of clients. These beans are typically used to provide or expose some kind of service.
If you were to design a Bank Account as a stateless session bean, you would expose the deposit, withdraw, and balance operations as services. The state, meaning the account number, would be held in the database and not in the bean instance because any bean instance could be used to process the requests. In such a case, you will have to pass the account number each time you make any deposit or withdrawal operations or check the balance.
WebLogic Server provides various parameters that can be used to tune your stateless session beans. These parameters are available in the
weblogic-ejb-jar.xml descriptor file. The following sections examine the pooling parameters, statistics that can help you tune these parameters, and finally the clustering parameters.
The EJB container inside WebLogic Server uses pooling to manage stateless session bean instances. The container maintains a pool of ready instances. These are instances that are ready to service method calls as soon as they come into the server.
When a stateless session bean method call is made, the EJB container grabs one of the ready instances from the pool and uses this instance to execute the call. The instance is returned back to the pool after the method call is complete. The Pooling strategy helps the container service a large number of clients with a small number of bean instances and also improves the response time of stateless EJBs.
The EJB container maintains a different pool for each stateless session bean deployed. The server provides two parameters to configure the stateless session pool:
This parameter specifies the initial size of the pool. When a stateless session bean is deployed, this number of bean instances is instantiated by the EJB container. The container instantiates the beans, calls
ejbCreate() methods, and places the bean in the pool of ready instances. This allows the EJB container to immediately respond to method calls on the bean without having to initialize the bean.
A typical use case when you would set this parameter would be when your stateless session bean's
setSessionContext() methods initialize some heavy objects like connections to a database or legacy systems. In such a scenario, setting the parameter initializes the beans and keeps them ready to service requests, therefore improving the response time.
This parameter determines the maximum size of the pool. If all the instances in the pool are busy, that is, serving method calls, and a new request arrives, then the container creates another bean instance to serve that request. The server can, however, create new instances only until it reaches the max-beans-in-free-pool limit. The parameter is therefore an upper bound on the number of bean instances that can be created for the stateless EJB.
If the upper limit is reached and all the instances are busy, a new request has to wait for a ready instance to become available again. This wait period is equivalent to the transaction timeout on the request. If a bean does not become available in that period, the transaction would be rolled back, and the request would not be served.
In most cases, EJB developers should not configure this parameter. This would let the EJB container create a bean instance as needed and prevent unnecessary waiting for a bean to be released back to the pool. The number of bean instances available in the pool improves the concurrency of processing the stateless EJB requests, but not completely. The degree of concurrency you can achieve for your stateless EJB depends on the number of execute threads in the WebLogic Server or on the Execute Thread Pool if you have configured one for the EJB.
A typical use case to limit the number of instances in the free pool using this parameter would be when your stateless EJB holds a connection to a backend system and there are a limited number of such connections available. For example, let's assume you have a maximum of 20 connections allowable by your backend/legacy system. If these connections are instantiated in the stateless EJB (
ejbCreate()), then you can configure the
max-beans-in-free-pool parameter to 20. Therefore, if all the bean instances are busy and a 21 st request arrives, the hard limit of 20 beans in the free pool would not allow a new bean instance to be created, thus preventing a new backend connection. This parameter can be used for throttling backend connections when they are limited.
The EJB container provides a number of runtime statistics for monitoring EJB applications. The WebLogic Server Administration Console uses these statistics to calculate specific ratios, which can be used to tune the pooling parameters.
Here, we will look at certain conditions under which you should tune the pooling parameters:
The Pool Miss Ratio is high
The Pool Miss Ratio is calculated as
(Miss Total Count / Access Total count) * 100. This is an indication of how many times a request was made to get a bean from the pool when no beans were available.
Cause: All bean instances are in use due to a heavy request volume. In such a case, there are no ready instances in the free pool to service the new requests. This leads to a pool miss.
Action: Perform a load test to find out the peak load by monitoring the Beans In Use Current Count parameter. Tune the
max-beans-in-free-pool size to the max value reached by the parameter during the load test. Beans in free pool take up memory, so availability of memory should also be taken into consideration while tuning this parameter.
Cause: Beans are getting destroyed. A bean instance is destroyed if a non-application exception is thrown from the bean code. If the bean instance gets destroyed, it would not be released to the free pool when the method completes execution. There will be fewer beans in the free pool, which could cause a pool miss. This will result in extra overhead of creating the bean instance and initializing it and could hurt performance.
Action: Avoid throwing a non-application exception from within the bean code.
Destroyed Bean Ratio is high
The Destroyed Bean Ratio is calculated as
(Destroyed Total Count / Access Total Count) * 100
The cause and action are the same as discussed above.
Pool Timeout Ratio is high
The Pool Timeout Ratio is calculated as
(Pool Timeout Total Count / Access Total Count) * 100
Cause: A high pool timeout ratio means your pool is improperly tuned with respect to the call volume. As the request volume increases, the EJB container creates more instances until it reaches the
maximum-beans-in-free-pool value. If all instances are active and the maximum free pool size has been reached, a new method request has to wait for an instance to be released back to the pool. This wait period is the pool timeout and is equal to the transaction-timeout configured for the stateless EJB.
Action: Increase the transaction timeout in the
weblogic-ejb-jar.xml descriptor. However, this is an application-level setting and could have other ramifications. If the timeout value is increased, it could decrease the throughput, as the response time would increase. Alternatively, use the safer option of increasing the max-beans-in-free-pool.
The stateless programming model makes it possible for a small number of bean instances to serve a large number of clients. This is amplified when the model is applied to a cluster of WebLogic Servers. A WebLogic cluster can take advantage of a larger pool of free instances that comprises the pools on all the servers in the cluster. It does not matter on which server the call is executed because there is no state maintained by the bean on behalf of the client. This also contributes to a scalable architecture for the service offered by the stateless session bean.
WebLogic Server takes advantage of this stateless model to create a smart stub known as a replica-aware stub. The replica-aware stub keeps a list of all the servers available in the cluster. It intercepts any failure and retries the call on another available server in the cluster.
The server provides a few configuration parameters to configure clustered stateless session beans.
This parameter when set to true allows the home stub to load-balance calls on the home interface between servers in the cluster on which the stateless EJB has been deployed. If any server in the cluster is unavailable, the calls will fail over to some other server in the cluster, which hosts the stateless EJB. Setting this parameter to true provides clustering capabilities of load balancing and failover only for the calls on the home interface, that is, the create method.
home-load-algorithm parameter can be used along with the
home-is-clusterable parameter to specify the load-balancing algorithm to be used by the home stub to load-balance between replicas of the stateless EJB home.
This parameter allows the replica-aware stub to load-balance calls on the remote interface between servers in the cluster on which the stateless EJB has been deployed. If a call made by the client through the replica-aware stub fails, the stub intercepts the failure and retries the call on another replica, that is, on a stateless EJB hosted on some other server in the cluster. Setting this parameter to true provides clustering capabilities of load balancing and failover only for the calls on the remote interface. Automatic failover happens only for the methods that are declared idempotent. This is described in detail below.
stateless-bean-load-algorithm parameter can be used along with the
stateless-bean-is-clusterable parameter to specify the load-balancing algorithm to be used by the remote stub to load-balance between replicas of the stateless bean.
A method can be called idempotent if calling it multiple times produces the same result as calling it once. Typically, any methods that do not alter any state on the server side are idempotent, for example, any read-only method. All stateless session home methods are idempotent. For the methods in the remote interface, you have to explicitly declare which methods are idempotent using the above parameter. Making a method idempotent allows a method call to be retried on another server in the cluster if there is failure on the previous server.
The advantage you get from declaring stateless EJB methods idempotent is that the service exposed by the method is more available because the call can be retried if there is a failure. For example, you have a cluster configuration that has three servers in the cluster: serverA, serverB, serverC, and a clustered stateless EJB that is deployed to the cluster. When a client calls a method on the EJB, let's assume that the smart stub makes the first call on serverA. If serverA crashes at this point, the call fails. However, if the method is idempotent, the smart stub intercepts the failure and retries the method on serverB. If the invoke on serverB fails for some reason, the call is retried on serverC.
Using cluster-aware stateless EJBs helps you develop scalable and reliable services.
However, use caution when declaring stateless methods idempotent. Only methods that do not change any state on the server side or produce the same result irrespective of the number of times they are called should be made idempotent.
Stateful session beans are session beans that maintain a conversational state on behalf of the clients. The conversational state has to be retained across methods and transactions. As each bean instance maintains state for a specific client, each bean has it own unique identity. The state is maintained in the member fields of the bean and is initialized by a particular client when the bean is created on the server. These beans are typically used to maintain a server-side state for a particular client.
Stateful EJBs are very similar to their stateless counterparts code-wise; however, you should remember that the programming models of each bean type are quite different.
If you were to design a Bank Account as a stateful session bean, each client would have its state stored in the bean on the server. The state, meaning the account number and balance, would be initialized when the client creates the bean and thereafter the same bean instance would be used for any methods invoked by that client. There would therefore be no need for the client to pass the account number each time as in the stateless case to perform any operations on the account.
WebLogic Server provides various parameters to tune your stateful session beans. These parameters can be set in the
weblogic-ejb-jar.xml descriptor file. Let's take a look at how these can be tuned and the use cases surrounding them.
In the stateless programming model, each bean instance is identical to the other, which makes it possible to maintain a pool of ready instances. Any bean instance from the pool can be used to service a method on behalf of the client.
In the stateful model, however, each bean instance has a unique identity. Each client has its own bean instance on the server side, and the container always routes calls to the specific instance that is associated with the particular client. This model does not allow any kind of pooling strategy. So, the WebLogic Server uses an optimized in-memory cache to improve the performance of stateful EJBs. The stateful EJB cache stores active bean instances and inactive bean instances. Active bean instances are those that are currently executing a method invoked by the client or those that are a part of an ongoing transaction. The inactive beans are the ones that have completed executing methods invoked by their clients and that are not part of an ongoing transaction. Some of the inactive beans that have been idle for
idle-timeout-seconds are called idle beans and are eligible for passivation.
The stateful EJB cache in the WebLogic Server can be configured to control the size, the caching algorithm, and the timeouts for passivation. The following sections examine the parameters for configuring the cache.
This parameter specifies the maximum size of the cache. It is the upper bound on the number of active bean instances that can be held in the in-memory cache and, if not set, defaults to 1000. The EJB container passivates idle beans and the least recently used beans when there is demand for more space in the cache but the cache has reached the
max-beans-in-cache limit. If the cache is full and the container cannot passivate any bean instance (because all are active), a
CacheFullException is thrown.
The memory footprint of the cache is directly proportional to the number of bean instances in the cache and the data held in each bean instance.
It is important to tune the size of the stateful EJB cache appropriately. A smaller than necessary cache size causes excessive passivation and activation to and from the disk, which leads to a drop in performance. The maximum cache size can be tuned depending on server memory, number of active clients at any given time, or data held in each bean instance.
The WebLogic Server EJB container provides monitoring statistics that would help in tuning this parameter. In particular, the Cache-Miss ratio and Passivation/Activation counts play an important role here. This ratio is defined as
(Cache Miss Count / Cache Access Count) * 100. A high cache-miss ratio or a high passivation and activation count indicates that your cache needs some tuning.
Cause: The stateful in-memory cache holds a finite working set of bean instances. When the demand spikes as the number of clients increase, the container passivates the beans that have not been used recently. If a client invokes a method on such an instance that has been already passivated, it leads to a cache miss. The container has to activate the bean from the disk and put it back into the cache to invoke the method. A cache-miss could also be caused because the stateful EJB that has been idle for too long and therefore has been removed from the cache as well as from the disk.
Action: If your application has stateful EJB clients such that some clients are more active than the others, increasing the
max-beans-in-cache will help you to maintain the more frequently invoked beans in the cache. You can minimize the cache-misses for these commonly used beans. If all the clients access the stateful EJB at regular intervals, then the working set of EJBs constantly changes and in such a case, increasing the max-beans-in-cache may not help much.
In general, increasing the
idle-timeout-seconds should reduce the cache-miss ratio. However, this will create a larger memory footprint.
This parameter is used for passivation of the stateful EJBs. An inactive bean that has been idle for
idle-timeout-seconds is considered idle and is eligible for passivation. Whether a bean that is eligible for passivation actually gets passivated to the disk also depends on other factors like the maximum cache size, space available in the cache, and the caching strategy. The caching strategy will be explained shortly.
This parameter should be set to the length of time you would want the beans to remain in the cache. If you have a use case where the clients actively access the stateful EJB over a longer period of time, increase the idle-timeout-seconds. This would let the beans remain in the cache longer and therefore be available to serve method requests. This would reduce passivation and activation and improve the response time.
At the same time, a higher
idle-timeout-seconds value would mean a larger cache and therefore more memory resources.
This parameter controls the lifespan of a passivated bean on the disk. Beans passivated to the disk take up disk resources. To minimize the disk resource usage, the WebLogic EJB container has a cleanup mechanism: a trigger that runs periodically to remove passivated instances from the disk. Bean instances living longer than
session-timeout-seconds are periodically deleted from the disk. By default, the value of this parameter is equal to the value of
idle-timeout-seconds. For certain applications, you may need to tune this parameter to a value different from the idle-timeout-seconds.
Let's consider a use case where a shopping cart is implemented as a stateful EJB. A user typically selects an item and adds it to the shopping cart. Not all users go ahead and buy items stored in the shopping cart. If the user disconnects or closes the browser, the state (items selected) will still be maintained for the user inside the particular bean instance. There are chances that the user logs in after a day or two and accesses his or her shopping cart to purchase an item that had been selected earlier. Rather than making the user browse and select the item again, you would want to display all the items selected by the user earlier. This can be done by keeping the bean instance specific to the user in cache for that period, that is, by configuring a longer idle timeout. But keeping the shopping cart bean for the particular user in memory for as long as a day would unnecessarily consume server resources and degrade performance.
An efficient strategy would be to configure the
session-timeout-seconds parameter to a longer period of time (a couple of days) so that the bean instance is alive on the disk for a longer time. The state can be restored in memory by activating the bean whenever the user logs in again. Setting the
idle-timeout-seconds to a lower value (units of hours) and
session-timeout-seconds to a higher value (units of days) in such a scenario gives you the advantage of maintaining the state of a larger number of clients with a finite amount of memory over a longer period of time.
The WebLogic Server EJB container provides two caching strategies, Not Recently Used (NRU) and Least Recently Used (LRU), that control the passivation of stateful session beans to the disk. Let's look at these in more detail:
Least Recently Used
This eager passivation strategy passivates idle beans to the disk periodically. A bean instance is considered idle if it has been inactive for
idle-timeout-seconds, that is, it has not serviced any method for that period of time. With this strategy passivation takes place for the idle beans periodically, even if there is no pressure on the cache and this period is a function of the value of the
idle-timeout-seconds parameter. However, if there is pressure on the cache, the inactive beans that haven't been idle for
idle-timeout-seconds will also be passivated. The least recently used inactive beans will be chosen for passivation in such a case. Beans that are active, that is, bean instances executing methods or those that are part of a transaction, will never be passivated.
Figure. 1. LRU Caching strategy
Not Recently Used
This lazy passivation strategy passivates beans to the disk only when there is pressure on the cache, meaning only when the cache has reached the
max-beans-in-cache size and there is more demand. When there is no pressure on the cache, avoiding passivation of the idle beans reduces the overhead of serialization.
However, when there is pressure on the cache, the container removes any idle bean instances (beans that have been idle for at least
idle-timeout-seconds) from the cache instead of passivating them. By doing this, an idle instance does not consume memory or disk resources. The container will also passivate any other inactive bean instances that have not been recently used only when there is pressure on the cache.
Figure. 2. NRU Caching strategy
Passivation is the process where a bean's state is serialized and written to the disk. The reverse process, activation, is where a bean's state is deserialized, that is, read from the disk. Passivation and activation are expensive operations and stateful EJBs can perform better if these operations are minimized.
Choosing the right caching strategy for your application can help improve the throughput as well as conserve the server's memory resources. The main factors that you need to consider when you are configuring the caching strategy are:
The LRU strategy is a good choice when server memory is at a premium and the client that invokes the stateful EJB is short lived: the client invokes the EJB in a specific period of time and not thereafter. In such a case, it is acceptable that the bean gets passivated once the bean is idle (after
idle-timeout-seconds) as the EJB would not be (or rarely would be) accessed later. As the bean instance would not be activated again, we can conserve memory resources by having it passivated as soon as it is idle.
When memory resources are scarce, use the LRU strategy so that idle beans are passivated eagerly and the stateful EJB cache uses the server memory conservatively.
The NRU caching strategy is a better choice when throughput is of significance and memory resources are available aplenty. Use this strategy to improve response time by minimizing passivation and activation of the stateful bean instances. The passivation will happen only when there is pressure on the cache, meaning where there is demand for more space and the cache has reached the
max-beans-in-cache. If you have a use case that involves clients that invoke methods on the stateful EJB over a longer period of time, you would want the bean instances to be available in the cache to serve the client requests immediately. In such a scenario, choose the NRU strategy to improve response time of your stateful EJBs.
In general, the NRU strategy is a good choice in situations where method response time is critical and where you have plenty of server memory resources. In these situations, having a large enough cache will keep the beans ready to serve requests immediately.
Avoid using the LRU strategy if the maximum cache size configured is sufficient for handling the number of active clients at any given time, as it would cause unnecessary passivation and activation overhead.
In applications where the stateful EJBs hold a lot of data, passivation and activation may take longer and effect the response time. Avoid using the LRU strategy if you expect higher throughput. However, be aware that less passivation means the beans and referenced objects will be held in memory longer.
In scenarios where server memory resources are scarce, avoid using the NRU strategy.
ejbRemove()on your stateful EJBs
Though not a tuning parameter, this is a best practice for the efficient use of server resources and improving performance, and it warrants a mention here. J2EE developers commonly make the mistake of not calling
remove() on their stateful beans. Consider a common use case where a servlet/JSP in the web tier creates a stateful bean and stores the bean handle in the HTTP session. When the HTTP session times out (due to user inactivity), the bean still stays alive in the stateful cache on the server. The price for this abandoned bean is:
Server memory is not used efficiently - the abandoned bean stays in memory until it is passivated. As the client that created the bean no longer exists, the bean occupies memory unnecessarily. If your stateful EJB is clustered, not calling remove keeps the bean in the cache of the primary server and its replicated counterpart on the secondary server. In addition to the beans, the replication machinery on both the servers also occupies memory.
Performance drops - the abandoned bean will be removed from the memory by passivating it to the disk. This affects performance as passivation is an expensive operation that involves serializing the bean to the disk.
In such a scenario, call
remove() on the bean using the
HttpSessionListener interface. When a HTTP session is about to be invalidated, the class that implements the above interface is notified of the event by calling the
sessionDestroyed() method. You can remove the stateful bean that is associated with the HTTP session in this callback method. This design will help you to use server memory efficiently and also improve performance.
As for stateless beans, the EJB container in WebLogic Server provides configuration parameters to configure clustered stateful session beans too. Load balancing is available for the home stub and failover features are available for the remote stub.
This parameter, when set to true, allows the home stub to load balance calls on the home interface across servers in the cluster on which the stateful EJB has been deployed. It works similarly to the
home-is-clusterable parameter for stateless EJBs: the create method on the stateful EJB's home interface is load balanced.
home-load-algorithm parameter can be used along with this parameter to specify the load-balancing algorithm used by the home stub to load-balance between replicas of the stateful EJB home.
This parameter determines the clustering behavior of the stateful EJBObject. It can take on one of two values: inMemory or None. The stateful EJB is clustered when the parameter is set to InMemory. The replication is similar to that for HTTP sessions as the state is maintained on a pair of servers. When a home interface creates a stateful EJB on one server, another server within the cluster is selected where the state of the EJB is replicated. The state is therefore available on a pair of servers, primary being the server where the EJB was created, and secondary being the one that was chosen for replication. The EJBObject stub returned to the client contains the location of the primary server and secondary server for that stateful EJB. Any changes to the state on the primary server are propagated to the secondary server. WebLogic Server provides optimization in that only the difference in state is propagated to the secondary. For stateful EJBs involved in transaction, replication happens after the transaction commits. For non-transactional EJBs, replication happens at the end of each method call.
For stateful EJBs there is no load balancing for the methods on the remote interface. A client will always be routed to the EJB on the primary server until it is not available. Failover is supported. If the primary server fails, the remote stub automatically retires the call on the secondary server. The secondary server then becomes the primary server, and a new secondary server is chosen. All future calls from the client will be routed to the new primary server. The stub is updated of the new primary-secondary pair.
Note that there are some exceptions:
1. State will be lost if both primary and secondary fail at the same time.
2. State will be stale if the primary fails before replication to the secondary takes place. Remember that replication happens at the end of the transaction.
Set the replication type to
InMemory to take advantage of the reliability offered by clustered stateful EJBs.
The WebLogic Server EJB container provides some parameters that would allow your EJBs to deviate from the behavior described in the EJB Specification. In this section, these parameters are described together with the scenarios in which they can be used.
The EJB Specification does not allow clients to make concurrent calls to a stateful session object. If a stateful session bean instance is actively processing a client invoked call, and another call arrives for the same bean instance from the same or different client, the container is supposed to throw a
Setting this parameter to true allows you to override that behavior. The EJB container will block the concurrent call until the previous call completes and let it proceed only when the previous call has completed. The container therefore serializes access to the Stateful bean instance.
By default, the parameter is set to false to adhere to the specification behavior. However, if you have a multithreaded client that invokes a stateful EJB, it is possible to have a scenario where concurrent calls are made on the same bean instance. In such a case, setting this parameter to true would allow for concurrent calls.
In scenarios where you see a
LockTimeoutException due to concurrent calls being made on the stateful bean, setting this parameter to true may help you to avoid the above exception.
The EJB Specification prohibits a bean developer from calling remove on a stateful session bean participating in a transaction. It results in a
allow-remove-during-transaction parameter lets you override this behavior.
If you have a use case where a stateful EJB is invoked as part of a long-running transaction, it may be useful to remove the stateful bean from within the transaction after its part of the work is done. Waiting to call remove on the bean until the transaction ends will cause the bean to occupy memory for a longer period of time. Calling remove from within the transaction in such a case would help use the server's memory resources efficiently.
However, you need to set this parameter judiciously after verifying that a remove within the transaction would not cause any unnecessary side effects. For example, if your stateful EJB implements the
SessionSynchronization interface and expects the
afterCompletion() callbacks wherein you change the state of your application or release backend connections, you should not set this parameter to true. Setting the parameter to true would allow the EJB to be removed before the transaction commits, and the bean would not get synchronization callbacks from the container. This may lead to undesired effects.
This article presents the various tuning parameters available for session beans in the BEA WebLogic EJB container and advice on how they can be effectively used in different use cases. This should help you tune your EJB applications to run more efficiently. The next article in this series looks at tuning entity beans.
Mihir Kulkarni is a Software Engineer working on the EJB Container for the Customer Centric Engineering team at BEA Systems. He has a Masters degree in Computer Science from Texas A&M University and is a Java Certified Programmer. Previously, he has presented at BEA EWorld Conference at San Diego and has contributed chapters on Performance Tuning and J2EE Deployment to the book titled "BEA WebLogic Platform 7".