by Olaf Heimburger
Oracle Fusion Middleware comes with a great toolset for investigating suspicious application behavior.
Published January 2010
Since Java saw the light of the world in 1995 applications built with it are getting more and more complex. Over time many models or architectures where developed to conquer the increasingly complex requirements. Unfortunately, the solutions will not get lighter and easier but rather more difficult and complex. Every new component increases the latency and makes the solution slower.
When things get slow, the usual suspects are always the technologies in use. Depending on your technical background it might be either the database or the middleware. The network or the browsers are usually the least suspected ones. Of course, this is unfair until proven and the network or the browser can have a tremendous impact.
To find the real cause for the lost time, one needs to investigate the application for a particular usage profile and present bullet-proof evidence. Fortunately, Oracle Fusion Middleware includes various tools for help you doing that.
Poor performance can have many causes. The following are some of the most common issues that can cause poor performance in your application:
Memory Leaks. The term memory leak is a bit misleading. Although the memory of your computer or JVM will not "leak" like water, you will notice over time that the available memory will become less than expected. Why would this occur, as the JVM garbage collector (GC) is responsible for reclaiming unused memory by removing not-used objects from memory? Most likely because the GC is not able to find any objects that can be removed from the memory because they do not appear to be unused.
Most memory leaks derive from over-looked references or poorly implemented homegrown caches. (Hint: If you do not use SoftReferences or WeakReferences, you cannot leverage the GC to automatically shrink your cache.) These leaks will increase the number of GC calls and the time it takes to find small amounts of memory to be freed. The resulting overhead on your application will impact the user experience. Finding and eliminating these memory leaks is one of the primary objectives of performance tuning.
Poor Data Modeling. Data modeling has a tremendous impact on the runtime behavior of an application. In a modern multi-tier application one should remember that data can and will cross process boundaries. Crossing these boundaries takes time as the binary form of the data must be converted to a transport format. However, as this process of serialization and de-serialization is implemented, it is slower than keeping the data inside of the process. The impact of data types and the way to implement the process has a tremendous effect on the overall performance..
For example, a date or a timestamp is usually stored as a 64-bit value (Java Long type). However, nothing prevents you from choosing a Calendar type to store the same information. A Calendar type normally takes 80 bytes. Thus one can easily save 76 bytes by choosing the right type to store a single timestamp.
Second, choosing externalization over serialization can dramatically improve performance. The downside is that externalization comes with the extra cost of highly stringent programming practices during development time.
High Network In/Output. One of the great fallacies of network computing is the "fast" network. Once you need to call something remotely, you definitely lose time. Saving network calls in either numbers or size will save time that the application needs for other purposes.
Saving network time can be quite easy. For one, you can let the remote application do its job. Usually very specific calls that return one correct result are faster than the ones that returns thousands of not needed results.
Poorly Configured JVMs. Every JVM can be configured via a number of command line switches and many of them directly or indirectly impact the runtime behavior of your application. For example, nearly every JVM user knows the switches to define the size of the JVM memory, otherwise known as the heap. Others are less known but setting them correctly can improve performance for a specific usage profile; however, with a different usage profile, the same settings may also decrease performance.
Slow Database. A slow database is a myth of multitier architecture. Rather, the database is almost always the fastest piece in the big picture of your applications architecture. On the other hand, missing indexes can decrease performance dramatically. A well tuned database and correctly used SQL statements play well together in a optimal architecture.
Now that we know the usual suspects, what's the next step? To be able to provide evidence for one of the suspects, the right tool at the right time is important. But before we have a look at the tools, let us have a look at the runtime environments available for Oracle Fusion Middleware.
When installing Oracle WebLogic Server as the base for most configurations, you can decide which JVM you want to use. At the time of this writing, out of the box WebLogic Server offers the Sun JVM 1.6 or Oracle JRockit JVM 1.6. Each JVM has its own idiosyncrasies and will suit a specific application profile. Which JVM is the best fit for your application depends on the application profile and how the JVM can cope with it. To find the right one for your application profile thorough testing and tuning is important.
The Sun JVM is the Java reference implementation and should be used to verify that the application behaves as expected.
The Sun JVM's memory model comprises two major memory areas: the Young Generation and the Old Generation. The Young Generation itself is divided into an Eden Space and two Survivor Spaces. Additionally a Permanent Generation Space (PermGenSpace) is available.
Dynamically created objects start living in Eden Space. During garbage collection they move to the Survivor Space and to the Old Generation until they are un-used and collected by the GC. The PermGenSpace stores static data like classes, constant values, and static variables.
A Sun JVM can very quickly run out of memory if too many classes, static variables, and constant values are loaded in a PermGenSpace that is too small. As the PermGenSpace takes its space from the overall heap size, it must be considerably smaller. When the Old Generation reaches a constant size of 70-80% of the available space, one can assume a memory leak. A mark of 90% is serious and should be investigated immediately.
A highly saturated memory influences the runtime, as GC will be called more often and needs longer cycles to find areas that can be collected. Very often these areas are too small and the GC needs more cycles to fulfill the memory request. When things get worse, the JVM only performs GCs and the application hangs.
The Sun JVM comes with different GCs. Each is perfect for a specific usage profile. The GC and its behavior can be tuned via command line switches.
The Oracle JRockit JVM is a Java-compatible implementation that was designed and implemented as a very fast JVM for Intel hardware. The JRockit memory model is comparable to that of the Sun JVM; it has Young and Old generation areas. New and short-living objects will be found in the Young Generation area while long-living objects can be found in the Old Generation area. The Oracle JRockit memory model has no PermGenSpace and thus will not throw any out-of-memory exception because of a full PermGenSpace.
The Oracle JRockit JVM offers various GCs as well. Especially for better runtime behavior the Most Concurrent Mark or Sweep and Parallel Mark and Sweep GCs deserve a try. As with the Sun JVM, these GCs can be explicitly set through the command line switches and are part of the settings for optimal configuration for the application usage profile.
Now that we you have a little background about the Oracle Fusion Middleware JVMs, to follow is an overview of the basic toolset provided for diagnosis of performance problems. Depending on which JVM you use, a different combination of tools is available.
For finding performance issues you should always start with the Oracle Enterprise Manager Fusion Middleware Control console. This piece of management software is installed with most components and may be already available to you. It shows important information about the runtime situation of your application at a glance.
The Summary screen of the Management Console shows a summary of the current state of your instance. Apart from the Response and Load graph, you should take a close look the Summary portlet. It shows viable information on very little screen estate. The information displayed is very important for any Java EE application as indicators for Servlets/JSPs, JMS, EJBs, and JDBC and JTA usage are shown.
If your application makes heavy use of Servlets and JSPs, the Most Requested portlet (below) is worth a look. Unusually long response times are indicators for poor performance.
Figure 2 Most Requested portlet
Furthermore, never underestimate log files! They are first place to look for indications of issues of any kind. Combined with additional references in time and space you can easily get a better picture of the runtime behavior of your application. The screenshot below shows the Log Messages search window.
The VisualVM and JVisualVM tools are geared toward the Sun JVM. Since JDK 1.6.0_10, JVisualVM is part of the JDK, while VisualVM could be installed separately as a standalone application. Both have the same foundation and can be extended through plugins, while VisualVM has better support for more plugins. VisualVM can be used for JDKs version 1.5.0 and higher but really excels with version 1.6.0.
Every analysis starts with the VisualVM console. This shows all runtime parameter very neatly (without digging through some start scripts). A simple click on the Monitor tab shows the current state of the JVM. Further details are shown in the tabs Threads, Profiler, Memory Sampler or Visual GC.
The Threads tab shows all current threads in use within the JVM. A single click on the Thread Dump button creates a thread dump for later analysis through Thread Dump Analyzer plugin.
VisualGC. When it comes to finding issues related to the Sun JVM memory model, the VisualGC plugin is your choice. Based on the VisualGC tool that comes with jvmstat 3.0 for JDK 1.5.0, this plugin shows the on-line state of the JVM with all its Class Loader Time, Compile Time (for the JIT), and GC Time. If your applications tends to slow down after a while, you should consider an investigation of the JVM memory with VisualGC!
Memory Sampling and Heap Dumps. To analyze how the available memory is used by objects, two tools are available. The Memory Sampler gives you an online view while the Heap Dump Analyzer can be used for off-line view of the memory. You can create the heap dumps at any time by simply pressing a button. VisualVM will automatically read the dump and presents the result as shown below.
When you run Oracle Fusion Middleware with the supplied Oracle JRockit JVM, you can use Oracle JRockit Mission Control without further installations required.
As with VisualVM, analysis starts with the JRockit Console. It displays an overview of the current state of the JVM with gauges for Used Java Heap, Used Physical Memory, CPU Usage, and Live Set as well as graphs for Memory or Processor usage. The tabs Memory, Threads, Runtime, Exception Count, or Memory Profile are available for further insights of the applications profile.
The Oracle JRockit Runtime Analyzer (JRA) is now fully integrated into Mission Control can be used to record the applications runtime behavior on demand. There are no special tricks like memory module replaces, aspect weaving or specific shared libraries necessary. This results in a small overhead of 1-2 %.
Through JRA one gets a snapshot of the application. It includes slow methods, memory usage, all existing objects, possible optimizations, and blocking threads.
Memory Leak Detector. To find possible memory leaks you can use the Memory Leak Detector. It must be started on demand, but instantly shows trends for possible memory leaks. After a while you get to find the real leaks and are able to go through the suspicious classes.
Oracle Fusion Middleware comes with a great toolset for investigating suspicious application behavior. These tools are mainly intended for ad hoc usage and not for continuous monitoring. Furthermore, for an overall picture of all components, a more thorough view is needed - Application Diagnostics for Java (AD4J) or Oracle Realtime User Experience being good options.
If you need to investigate your application you must rely on test cases that mimic the application profile that causes problems. Run them very often to gather a complete picture of your application.
When you tune your application, run the tests after every single step and measure the impact. If it does not work as expect, go back and try a different one. This takes time, but at the end you have an environment that is well tuned for your application profile.