Troubleshooting Guide for Java SE 6 with HotSpot VM

This document is a guide to help troubleshoot problems that might arise with applications that are developed using the release of Java Platform, Standard Edition 6 (JDK 6 release or Java SE 6 release). In particular, this guide addresses possible problems between the applications and the Java HotSpot virtual machine. The document provides a description of the tools, command line options, and other help in analyzing a problem. The document also provides guidance on how to approach some general issues such as a crash, hang, or memory resource issues. Finally, the document provides direction for data collection and bug report preparation.

• Document Information
• Preface
• 1.  Diagnostic Tools and Options
  • 1.1 Introduction
    • 1.1.1 Command-Line Options With -XX
    • 1.1.2 Limitations
    • 1.1.3 Developing New Tools
  • 1.2 Summary of Tools, Options, and Commands
    • 1.2.1 Tools and Options for Post-mortem Diagnostics
    • 1.2.2 Tools and Options for Hung Processes
    • 1.2.3 Tools and Options for Monitoring
    • 1.2.4 Other Tools, Options, Variables, and Properties
• 2.  Detailed Tool Descriptions
  • 2.1 HPROF - Heap Profiler
    • 2.1.1 Heap Allocation Profiles ( heap=sites)
    • 2.1.2 Heap Dump ( heap=dump)
    • 2.1.3 CPU Usage Sampling Profiles ( cpu=samples)
    • 2.1.4 CPU Usage Times Profile ( cpu=times)
  • 2.2 Java VisualVM
  • 2.3 JConsole Utility
  • 2.4 jdb Utility
    • 2.4.1 Attaching to a Process
    • 2.4.2 Attaching to a Core File on the Same Machine
    • 2.4.3 Attaching to a Core File or a Hung Process from a Different Machine
  • 2.5 jhat Utility
    • 2.5.1 Standard Queries
      • 2.5.1.1 All Classes Query
      • 2.5.1.2 Class Query
      • 2.5.1.3 Object Query
      • 2.5.1.4 Instances Query
      • 2.5.1.5 Roots Query
      • 2.5.1.6 Reachable Objects Query
      • 2.5.1.7 Instance Counts for All Classes Query
      • 2.5.1.8 All Roots Query
      • 2.5.1.9 New Instances Query
      • 2.5.1.10 Histogram Queries
    • 2.5.2 Custom Queries
    • 2.5.3 Heap Analysis Hints
      • 2.5.3.1 What is keeping an object alive?
      • 2.5.3.2 Where was this object allocated?
  • 2.6 jinfo Utility
  • 2.7 jmap Utility
    • 2.7.1 Heap Configuration and Usage
    • 2.7.2 Heap Histogram of Running Process
    • 2.7.3 Heap Histogram of Core File
    • 2.7.4 Getting Information on the Permanent Generation
  • 2.8 jps Utility
  • 2.9 jrunscript Utility
  • 2.10 jsadebugd Daemon
  • 2.11 jstack Utility
    • 2.11.1 Forcing a Stack Dump
    • 2.11.2 Printing Stack Trace From Core Dump
    • 2.11.3 Printing a Mixed Stack
  • 2.12 jstat Utility
    • 2.12.1 Example of -gcutil Option
    • 2.12.2 Example of -gcnew Option
    • 2.12.3 Example of -gcoldcapacity Option
  • 2.13 jstatd Daemon
  • 2.14 visualgc Tool
  • 2.15 Ctrl-Break Handler
    • 2.15.1 Thread Dump
    • 2.15.2 Deadlock Detection
    • 2.15.3 Heap Summary
  • 2.16 Operating-System-Specific Tools
    • 2.16.1 Solaris Operating System
    • 2.16.2 Linux Operating System
    • 2.16.3 Windows Operating System
    • 2.16.4 Tools Introduced in Solaris 10 OS
      • 2.16.4.1 Improvements in pmap Tool
      • 2.16.4.2 Improvements in pstack Tool
      • 2.16.4.3 Using the DTrace Tool
        • Probe Providers in Java HotSpot VM
        • Example of Using DTrace
  • 2.17 Developing Diagnostic Tools
    • 2.17.1 java.lang.management Package
    • 2.17.2 java.lang.instrument Package
    • 2.17.3 java.lang.Thread Class
    • 2.17.4 Java Virtual Machine Tools Interface
    • 2.17.5 Java Platform Debugger Architecture
• 3.  Troubleshooting Memory Leaks
  • 3.1 Meaning of OutOfMemoryError
    • 3.1.1 Detail Message: Java heap space
    • 3.1.2 Detail Message: PermGen space
    • 3.1.3 Detail Message: Requested array size exceeds VM limit
    • 3.1.4 Detail Message: request <size> bytes for <reason>. Out of swap space?
    • 3.1.5 Detail Message: <reason> <stack trace> (Native method)
  • 3.2 Crash Instead of OutOfMemoryError
  • 3.3 Diagnosing Leaks in Java Language Code
    • 3.3.1 NetBeans Profiler
    • 3.3.2 Using the jhat Utility
    • 3.3.3 Creating a Heap Dump
      • 3.3.3.1 HPROF Profiler
      • 3.3.3.2 jmap Utility
      • 3.3.3.3 JConsole Utility
      • 3.3.3.4 -XX:+HeapDumpOnOutOfMemoryError Command-line Option
    • 3.3.4 Obtaining a Heap Histogram on a Running Process
    • 3.3.5 Obtaining a Heap Histogram at OutOfMemoryError
    • 3.3.6 Monitoring the Number of Objects Pending Finalization
    • 3.3.7 Third Party Memory Debuggers
  • 3.4 Diagnosing Leaks in Native Code
    • 3.4.1 Tracking All Memory Allocation and Free Calls
    • 3.4.2 Tracking Memory Allocation in a JNI Library
    • 3.4.3 Tracking Memory Allocation With OS Support
    • 3.4.4 Using dbx to Find Leaks
    • 3.4.5 Using libumem to Find Leaks
• 4.  Troubleshooting System Crashes
  • 4.1 Sample Crashes
    • 4.1.1 Determining Where the Crash Occurred
    • 4.1.2 Crash in Native Code
    • 4.1.3 Crash due to Stack Overflow
    • 4.1.4 Crash in the HotSpot Compiler Thread
    • 4.1.5 Crash in Compiled Code
    • 4.1.6 Crash in VMThread
  • 4.2 Finding a Workaround
    • 4.2.1 Crash in HotSpot Compiler Thread or Compiled Code
    • 4.2.2 Crash During Garbage Collection
    • 4.2.3 Class Data Sharing
  • 4.3 Microsoft Visual C++ Version Considerations
• 5.  Troubleshooting Hanging or Looping Processes
  • 5.1 Diagnosing a Looping Process
  • 5.2 Diagnosing a Hung Process
    • 5.2.1 Deadlock Detected
    • 5.2.2 Deadlock Not Detected
    • 5.2.3 No Thread Dump
  • 5.3 Solaris 8 OS Thread Library
• 6.  Integrating Signal and Exception Handling
  • 6.1 Signal Handling on Solaris OS and Linux
    • 6.1.1 Reducing Signal Usage
    • 6.1.2 Alternative Signals
    • 6.1.3 Signal Chaining
  • 6.2 Exception Handling on Windows
    • 6.2.1 Signal Handling in the HotSpot Virtual Machine
    • 6.2.2 Console Handlers
• 7.  Submitting Bug Reports
  • 7.1 Checking for Existing Fixes in Update Releases
  • 7.2 Preparing to Submit a Bug Report
  • 7.3 Collecting Data for a Bug Report
    • 7.3.1 Hardware Details
    • 7.3.2 Operating System
    • 7.3.3 Java SE Version
    • 7.3.4 Command-Line Options
    • 7.3.5 Environment Variables
    • 7.3.6 Fatal Error Log
    • 7.3.7 Core or Crash Dump
    • 7.3.8 Detailed Description of the Problem
    • 7.3.9 Logs and Traces
    • 7.3.10 Results from Troubleshooting Steps
  • 7.4 Collecting Core Dumps
    • 7.4.1 Collecting Core Dumps on Solaris OS
      • 7.4.1.1 Using the ShowMessageBoxOnError Option on Solaris OS
      • 7.4.1.2 Suspending a Process using truss
    • 7.4.2 Collecting Core Dumps on Linux
      • 7.4.2.1 Using the ShowMessageBoxOnError Option on Linux
    • 7.4.3 Reasons for Not Getting a Core File
    • 7.4.4 Collecting Crash Dumps on Windows
      • 7.4.4.1 Configuring Dr. Watson
      • 7.4.4.2 Forcing a Crash Dump
• A.  Environment Variables and System Properties
  • A.1 JAVA_HOME Environment Variable
  • A.2 JAVA_TOOL_OPTIONS Environment Variable
  • A.3 java.security.debug System Property
• B.  Command-Line Options
  • B.1 HotSpot VM Command-Line Options
    • B.1.1 Dynamic Changing of Flag Values
    • B.1.2 -XX:+HeapDumpOnOutOfMemoryError Option
    • B.1.3 -XX:OnError= Option
    • B.1.4 -XX:+ShowMessageBoxOnError Option
    • B.1.5 Other -XX Options
  • B.2 Other Command-Line Options
    • B.2.1 -Xcheck:jni Option
    • B.2.2 -verbose:class Option
    • B.2.3 -verbose:gc Option
    • B.2.4 -verbose:jni Option
• C.  Fatal Error Log
  • C.1 Location of Fatal Error Log
  • C.2 Description of Fatal Error Log
  • C.3 Header Format
  • C.4 Thread Section Format
    • C.4.1 Thread Information
    • C.4.2 Signal Information
    • C.4.3 Register Context
    • C.4.4 Machine Instructions
    • C.4.5 Thread Stack
    • C.4.6 Further Details
  • C.5 Process Section Format
    • C.5.1 Thread List
    • C.5.2 VM State
    • C.5.3 Mutexes and Monitors
    • C.5.4 Heap Summary
    • C.5.5 Memory Map
    • C.5.6 VM Arguments and Environment Variables
    • C.5.7 Signal Handlers
  • C.6 System Section Format
• D.  Summary of Tools in This Release
  • D.1 Troubleshooting Tools Available in Java SE 6
    • D.1.1 Solaris OS and Linux
    • D.1.2 Windows Operating System
  • D.2 Changes to Troubleshooting Tools in Java SE 6