Memory Leaks, Be Gone!
Pages: 1, 2
There are also more specialized tools for doing memory leak detection. The JRockit Memory Leak Detector can be used to watch for memory leaks and can drill down to find the cause of the leak. This powerful tool is tightly integrated into the JRockit JVM to provide the lowest possible overhead as well as easy access to the virtual machine's heap.
Once you know that you really have a memory leak, you will need a more specialized tool to find out why there is a leak. The JVM by itself is not able to tell you that. A number of tools are available. There are essentially two different ways these tools can get information about the memory system from the JVM: JVMTI and byte code instrumentation. The Java Virtual Machine Tools Interface (JVMTI) and its predecessor JVMPI (Profiling Interface) are standardized interfaces for an external tool to communicate with the JVM and gather information from the JVM. Byte code instrumentation refers to the technique of preprocessing the byte code with probes for information that the tool needs.
In the case of memory-leak detection these techniques have two drawbacks that make them less than ideal for use in production-type environments. First, in terms of both memory usage and performance degradation the overhead is not negligible. Information about the heap usage has to be exported somehow from the JVM and gathered and processed in the tool. This means allocating memory. Exporting the information also has a cost in terms of performance of the JVM. For example, the garbage collector will run more slowly when it is collecting the information. The other drawback is the need to always run with the tool attached to the JVM. It's not possible to attach the tool to a previously started JVM, do the analysis, detach the tool, and keep the JVM running.
Since the JRockit Memory Leak Detector is integrated into the JVM, both of these drawbacks no longer apply. First, most of the processing and analysis is done inside the JVM, so there is no need to transfer or recreate any data. The processing can also piggyback on the garbage collector itself, which means increased speed. Second, the Memory Leak Detector can be attached and detached from a running JVM as long as the JVM was started with the
-Xmanagement option (which allows for monitoring and management of the JVM over the remote JMX interface). When the tool is detached, nothing is left of the tool in the JVM; it is back to running code at full speed just like before the tool was attached.
Let's take a deeper look at the tool and how it can be used for tracking down memory leaks. After you know you have a memory leak, the first step is to try to figure out what data you are leaking—what class of objects are causing the leak. The JRockit Memory Leak Detector does this by computing the number of existing objects for each class at every garbage collection. If the number of objects of a certain class increases over time (the "growth rate") you've probably got a leak.
Figure 2. The trend analysis view of the Memory Leak Detector
Because a leak may be just a trickle, the trend analysis must run over a longer period of time. During short periods of time, local increases of some classes may occur that later recede. However, the overhead of this is very small (the largest overhead consists of sending a packet of data from JRockit to the Memory Leak Detector for each garbage collection). The overhead should not be a problem for any system—even one running at full speed in production.
At first the numbers will jump around a lot, but over time they will stabilize and show you which classes are increasing in size.
Knowing which classes of objects are leaking can sometimes be enough to pin down the problem. The class is perhaps only used in a very limited part of the code, and a quick inspection of the code shows where the problem is. Unfortunately, it is likely that this information is not enough. For example, it is very common that the leak is objects of class
java.lang.String, but since strings are used all over the program, this is not very helpful.
What we want to know is which other objects are holding on to the leaking objects, Strings in this case. Why are the leaking objects still around? Who has references to these objects? But a list of every object that has a reference to a String would be way too large to be of any practical use. To limit the amount of data we can group it by class, so that we see which other classes of objects are holding on to the leaking objects (Strings). For example, it is very common that the Strings are in a Hashtable, in which case we would see Hashtable entry objects holding on to Strings. Working backward from the Hashtable entries we would eventually find Hashtable objects holding on to the entries as well as the Strings (see Figure 3 below).
Figure 3. Sample view of the type graph as seen in the tool
Since we are still looking at classes of objects here, not individual objects, we don't know which Hashtable is leaking. If we could find out how large all the Hashtables in the system are, we could assume that the largest Hashtable is the one that is leaking (since over time it will accumulate enough of a leak to have grown to a fair size). Therefore, a list of all the Hashtable objects, together with how much data they are referencing, will help us pinpoint the exact Hashtable that is responsible for the leak.
Figure 4. Screenshot of the list of Hashtable objects and the size of the data they are holding live
Calculating how much data an object is referencing is quite expensive (it requires walking the reference graph with that object as a root) and if we have to do this for many objects it will take some time. Knowing a bit about the internals of how a Hashtable is implemented allows for a shortcut. Internally, a Hashtable has an array of Hashtable entries. This array grows as the number of objects in the Hashtable grows. Therefore, to find the largest Hashtable, we can limit our search to finding the largest arrays that reference Hashtable entries. This is much faster.
Figure 5. Screenshot of the listing of the largest Hashtable entry arrays, as well as their sizes.
When we have found the instance of the Hashtable that is leaking we can see which other instances are referencing this Hashtable and work our way backward to see which Hashtable this is.
Figure 6. This is what an instance graph can look like in the tool.
For example, the Hashtable may be referenced from an object of type MyServer in a field called activeSessions. This will often be enough information to dig into the source code to locate the problem.
Figure 7. Inspecting an object and its references to other objects
When tracking down memory-leak problems it can be very useful to see where objects are allocated. It may not be enough to know how they relate to other objects, that is, which objects refer to them, but information about where they are created can help. Of course, you don't want to create an instrumented build of the application that prints out stack traces for each allocation. Neither do you want to run your application with a profiler attached in your production environment just in case you want to track down a memory leak.
With the JRockit Memory Leak Detector, the code in the application can be dynamically instrumented at the point of allocation to create stack traces. These stack traces can be accumulated and analyzed in the tool. There is zero cost for this feature as long as you do not enable it, which means you can always be ready to go. When allocation traces are requested, code is inserted on the fly by the JRockit compiler to monitor allocations but only for the specific classes requested. Even better, when you are done analyzing the data, the instrumented code is completely removed and there is nothing left in the code that can cause any performance degradation of the application.
Figure 8. The allocation stack traces for String during execution of a sample program
Memory leaks are difficult to find. Some of the best practices for avoiding memory leaks highlighted in this article include always remembering what you put in a data structure, and closely monitoring memory usage for unexpected growth.
We have also seen how the JRockit Memory Leak Detector can be used in a production type system to track down a memory leak. The tool has a three-step approach to finding leaks. First, do a trend analysis to find out which class of objects is leaking. Second, see which other classes are holding on to objects of the leaking class. Third, drill down to the individual objects and see how they are interconnected. It's also possible to get dynamic, on-the-fly stack traces for all object allocations in the system. These features and the fact that the tool is tightly integrated into the JVM allow you to track down and fix memory leaks in a safe, yet powerful way.
Staffan Larsen is a staff engineer working on the JRockit product, which he co-founded back in 1998.