Using DTrace to Profile and Debug A C++ Program

   
By Jay Danielsen, February 17, 2005  

A sample program CCtest was created to demonstrate an error common to C++ applications -- the memory leak. In many cases, a memory leak occurs when an object is created, but never destroyed, and such is the case with the program contained in this article.

The examples in this article demonstrate the use of the DTrace feature in the Solaris 10 Operating System to diagnose C++ application errors. These examples are also used to compare DTrace with other application debugging tools, including Sun Studio 10 software and mdb in the Solaris 10 OS.

C++ Name Mangling

When debugging a C++ program, you may notice that your compiler converts some C++ names into mangled, semi-intelligible strings of characters and digits. This name mangling is an implementation detail required for support of C++ function overloading, to provide valid external names for C++ function names that include special characters, and to distinguish instances of the same name declared in different namespaces and classes.

For example, using nm to extract the symbol table from a sample program named CCtest produces the following output:

# /usr/ccs/bin/nm CCtest
...
[61] | 134549248| 53|FUNC |GLOB |0 |9  |__1cJTestClass2T5B6M_v_
[85] | 134549301| 47|FUNC |GLOB |0 |9  |__1cJTestClass2T6M_v_
[76] | 134549136| 37|FUNC |GLOB |0 |9  |__1cJTestClass2t5B6M_v_
[62] | 134549173| 71|FUNC |GLOB |0 |9  |__1cJTestClass2t5B6Mpc_v_
[64] | 134549136| 37|FUNC |GLOB |0 |9  |__1cJTestClass2t6M_v_
[89] | 134549173| 71|FUNC |GLOB |0 |9  |__1cJTestClass2t6Mpc_v_
[80] | 134616000| 16|OBJT |GLOB |0 |18 |__1cJTestClassG__vtbl_
[91] | 134549348| 16|FUNC |GLOB |0 |9  |__1cJTestClassJClassName6kM_pc_
...
 

(Source code and makefile for CCtest are included at the end of this article.)

From this output, you may correctly assume that a number of these mangled symbols are associated with a class named TestClass, but you can not readily determine whether these symbols are associated with constructors, destructors, or class functions.

The Sun Studio compiler includes three utilities that can be used to translate the mangled symbols to their C++ counterparts: nm -C, dem, and c++filt. (Note: I used Sun Studio 10 software here, but I've tested my examples with both Sun Studio 9 and 10.)

If your C++ application was compiled with gcc/g++, you have an additional choice for demangling your application -- in addition to c++filt (which recognizes both Sun Studio and GNU mangled names), the open source gc++filt found in /usr/sfw/bin can be used to demangle the symbols contained in your g++ application.

Examples:

Sun Studio symbols without c++filt:

# nm CCtest | grep TestClass
[65] | 134549280| 37|FUNC |GLOB |0 |9 |__1cJTestClass2t6M_v_
[56] | 134549352| 54|FUNC |GLOB |0 |9 |__1cJTestClass2t6Mi_v_
[92] | 134549317| 35|FUNC |GLOB |0 |9 |__1cJTestClass2t6Mpc_v_
...
 

Sun Studio symbols with c++filt:

# nm CCtest | grep TestClass | c++filt
[65] | 134549280| 37|FUNC |GLOB |0 |9 |TestClass::TestClass()
[56] | 134549352| 54|FUNC |GLOB |0 |9 |TestClass::TestClass(int)
[92] | 134549317| 35|FUNC |GLOB |0 |9 |TestClass::TestClass(char*)
...
 

g++ symbols without gc++filt:

[86]  | 134550070| 41|FUNC |GLOB |0 |12 |_ZN9TestClassC1EPc
[110] | 134550180| 68|FUNC |GLOB |0 |12 |_ZN9TestClassC1Ei
[114] | 134549984| 43|FUNC |GLOB |0 |12 |_ZN9TestClassC1Ev
        ...
 

g++ symbols with gc++filt:

# nm gCCtest | grep TestClass | gc++filt
[86]  | 134550070| 41|FUNC |GLOB |0 |12 |TestClass::TestClass(char*)
[110] | 134550180| 68|FUNC |GLOB |0 |12 |TestClass::TestClass(int)
[114] | 134549984| 43|FUNC |GLOB |0 |12 |TestClass::TestClass()
        ...
 

And finally, displaying symbols with nm -C:

[64] | 134549344| 71|FUNC |GLOB |0 |9 |TestClass::TestClass()
                                       [__1cJTestClass2t6M_v_]
[87] | 134549424| 70|FUNC |GLOB |0 |9 |TestClass::TestClass(const char*)
                                       [__1cJTestClass2t6Mpkc_v_]
[57] | 134549504| 95|FUNC |GLOB |0 |9 |TestClass::TestClass(int)
                                       [__1cJTestClass2t6Mi_v_]
 

Let's use this information to create a DTrace script to perform an aggregation on the object calls associated with our test program. We can use the DTrace pid provider to enable probes associated with our mangled C++ symbols.

DTrace pid Provider

To test our constructor/destructor theory, let's start by counting the following:

  • The number of objects created -- calls to new()
  • The number of objects destroyed -- calls to delete()

Use the following script to extract the symbols corresponding to the new() and delete() functions from my CCtest program:

# dem `nm CCtest | awk -F\| '{ print $NF; }'` | egrep "new|delete"
__1c2k6Fpv_v_ == void operator delete(void*)
__1c2n6FI_pv_ == void*operator new(unsigned)
 

The corresponding DTrace script is used to enable probes on new() and delete() (saved as CCagg.d):

#!/usr/sbin/dtrace -s

pid$1::__1c2n6FI_pv_:
{
        @n[probefunc] = count();
}

pid$1::__1c2k6Fpv_v_:
{
        @d[probefunc] = count();
}

END
{
        printa(@n);
        printa(@d);
}
 

Start the CCtest program in one window, then execute the script we just created in another window as follows:

# dtrace -s ./CCagg.d `pgrep CCtest` | c++filt
 

The DTrace output is piped through c++filt to demangle the C++ symbols, with one caveat. You can't exit the DTrace script with a ^C as you would do normally: c++filt will be killed along with dtrace and you're left with no output. To display the output of this command, go to another window on your system and type:

# pkill dtrace
 

Use this sequence of steps for the rest of the exercises:

Window 1: # ./CCtest
Window 2: # dtrace -s [scriptname] | c++filt
Window 3: # 'pkill dtrace' window
 

The output of our aggregation script (window 2) should look like this:

void*operator new(unsigned)                          12
void operator delete(void*)                           8
 

So we may be on the right track with our theory: It does appear that we are creating more objects than we are deleting.

Let's check the memory addresses of our objects and attempt to match the instances of new() and delete(). The DTrace arg variables are used to display the addresses associated with our objects. Since a pointer to the object is contained in the return value of new(), we should see the same pointer value as arg0 in the call to delete(). With a slight modification to our initial script, we now have the following script, which I have named CCaddr.d:

#!/usr/sbin/dtrace -s

#pragma D option quiet
/*
__1c2k6Fpv_v_ == void operator delete(void*)
__1c2n6FI_pv_ == void*operator new(unsigned)
*/

/* return from new() */
pid$1::__1c2n6FI_pv_:return
{
        printf("%s: %x\n", probefunc, arg1);
}

/* call to delete() */
pid$1::__1c2k6Fpv_v_:entry
{
        printf("%s: %x\n", probefunc, arg0);
}
 

Execute this script:

# dtrace -s ./CCaddr.d `pgrep CCtest` | c++filt
 

Wait for a bit, then type this in your pkill dtrace window:

# pkill dtrace
 

Our output looks like a repeating pattern of three calls to new() and two calls to delete():

void*operator new(unsigned): 809e480
void*operator new(unsigned): 8068a70
void*operator new(unsigned): 809e4a0
void operator delete(void*): 8068a70
void operator delete(void*): 809e4a0
 

As you inspect the repeating output, a pattern emerges. It seems that the first new() of the repeating pattern does not have a corresponding call to delete(). At this point we have identified the source of the memory leak! Let's continue with dtrace and see what else we can learn from this information. We still do not know what type of class is associated with the object created at address 809e480.

Including a call to ustack() on entry to new() provides a hint. Here's the modification to our previous script, renamed CCstack.d:

#!/usr/sbin/dtrace -s

#pragma D option quiet

/*
__1c2k6Fpv_v_ == void operator delete(void*)
__1c2n6FI_pv_ == void*operator new(unsigned)
*/

pid$1::__1c2n6FI_pv_:entry
{
        ustack();
}

pid$1::__1c2n6FI_pv_:return
{
        printf("%s: %x\n", probefunc, arg1);
}

pid$1::__1c2k6Fpv_v_:entry
{
        printf("%s: %x\n", probefunc, arg0);
}
 

Executing CCstack.d produced the following output (after pkill dtrace):

# dtrace -s ./CCstack.d `pgrep CCtest` | c++filt


             libCrun.so.1`void*operator new(unsigned)
             CCtest`main+0x19
             CCtest`0x8050cda
void*operator new(unsigned): 80a2bd0

             libCrun.so.1`void*operator new(unsigned)
             CCtest`main+0x57
             CCtest`0x8050cda
void*operator new(unsigned): 8068a70

             libCrun.so.1`void*operator new(unsigned)
             CCtest`main+0x9a
             CCtest`0x8050cda
void*operator new(unsigned): 80a2bf0
void operator delete(void*): 8068a70
void operator delete(void*): 80a2bf0
 

The ustack() data tells us that new() is called from main+0x19, main+0x57, and main+0x9a -- we're interested in the object associated with the first call to new(), at main+0x19.

To determine the type of constructor called at main+0x19, we can use mdb as follows:

# gcore `pgrep CCtest`
gcore: core.1478 dumped
# mdb core.1478
Loading modules: [ libc.so.1 ld.so.1 ]
> main::dis
main:          pushl  %ebp
main+1:        movl   %esp,%ebp
main+3:        subl   $0x38,%esp
main+6:        movl   %esp,-0x2c(%ebp)
main+9:        movl   %ebx,-0x30(%ebp)
main+0xc:      movl   %esi,-0x34(%ebp)
main+0xf:      movl   %edi,-0x38(%ebp)
main+0x12:     pushl  $0x8
main+0x14:     call   -0x2e4   <PLT=libCrun.so.1`__1c2n6FI_pv_>
main+0x19:     addl   $0x4,%esp
main+0x1c:     movl   %eax,-0x10(%ebp)
main+0x1f:     movl   -0x10(%ebp),%eax
main+0x22:     pushl  %eax
main+0x23:     call   +0x1d5   <__1cJTestClass2t5B6M_v_>
        ...
 

Our constructor is called after the call to new, at offset main+0x23. So we have identified a call to the constructor __1cJTestClass2t5B6M_v_ that is never destroyed. Using dem to demangle this symbol produces:

# dem __1cJTestClass2t5B6M_v_
__1cJTestClass2t5B6M_v_ == TestClass::TestClass #Nvariant 1()
 

Thus a call to new TestClass() at main+0x19 is the cause of the memory leak. Examining the CCtest.cc source file reveals:

...
t = new TestClass();
cout << t->ClassName();

t = new TestClass((const char *)"Hello.");
cout << t->ClassName();

tt = new TestClass((const char *)"Goodbye.");
cout << tt->ClassName();

delete(t);
delete(tt);
...
 

It's clear that the first use of the variable t = new TestClass(); is overwritten by the second use:

t = new TestClass((const char *)"Hello.");
 

The memory leak has been identified and a fix can be implemented.

Summary

The DTrace pid provider allows you to enable a probe at any instruction associated with a process that is being examined. This example is intended to model the DTrace approach to interactive process debugging. DTrace features used in this example include: aggregations, displaying function arguments and return values, and viewing the user call stack. The dem and c++filt commands in Sun Studio software and the gc++filt in gcc were used to extract the function probes from the program symbol table and display the DTrace output in a source-compatible format.

Source files created for this example:

TestClass.h:

class TestClass
{
        public:
                TestClass();
                TestClass(const char *name);
                TestClass(int i);
                virtual ~TestClass();
                virtual char *ClassName() const;
        private:
                char *str;
};

TestClass.cc:

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <unistd.h>
#include "TestClass.h"

TestClass::TestClass() {
        str=strdup("empty.");
}

TestClass::TestClass(const char *name) {
        str=strdup(name);
}

TestClass::TestClass(int i) {
        str=(char *)malloc(128);
        sprintf(str, "Integer = %d", i);
}

TestClass::~TestClass() {
        if ( str )
                free(str);
}

char *TestClass::ClassName() const {
        return str;
}


CCtest.cc:

#include <iostream.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include "TestClass.h"

int main(int argc, char **argv)
{
        TestClass *t;
        TestClass *tt;

        while (1) {
                t = new TestClass();
                cout << t->ClassName();

                t = new TestClass((const char *)"Hello.");
                cout << t->ClassName();

                tt = new TestClass((const char *)"Goodbye.");
                cout << tt->ClassName();

                delete(t);
                delete(tt);
                sleep(1);
        }
}


Makefile:

OBJS=CCtest.o TestClass.o
PROGS=CCtest

CC=CC

all: $(PROGS)
        echo "Done."

clean:
        rm $(OBJS) $(PROGS)

CCtest: $(OBJS)
        $(CC) -o CCtest $(OBJS)

.cc.o:
        $(CC) $(CFLAGS) -c $<
 
About the Author

Jay Danielsen is a member of Sun's Advanced Development Engineering group and an operating systems ambassador. Based in Detroit, Michigan, Jay works with automotive customers to evaluate and customize Sun technologies. His background in software development includes C, Java, and assembly programming languages.

Rate and Review
Tell us what you think of the content of this page.
Excellent   Good   Fair   Poor  
Comments:
Your email address (no reply is possible without an address):
Sun Privacy Policy

Note: We are not able to respond to all submitted comments.
Left Curve
System Administrator
Right Curve
Left Curve
Developer and ISVs
Right Curve
Left Curve
Related Products
Right Curve
solaris-online-forum-banner