What You See Is What You Get Element

How to Set Up DTrace to Detect PHP Scripting Problems on Oracle Linux

by Christopher Jones

This article shows you how to set up DTrace to use PHP probes to trace scripting problems on Oracle Linux.


Published November 2013 (adapted from Christopher Jones' PHP and Oracle blog)


Want to comment on this article? Post the link on Facebook's OTN Garage page.  Have a similar article to share? Bring it up on Facebook or Twitter and let's discuss.

Introduction

DTrace is a popular, "always available" tracing utility useful for identifying performance and behavior issues in development and production systems. The standard "UEK3" kernel for Oracle Linux includes DTrace support. The PHP core and PHP OCI8 extensions have user probes that can be traced to efficiently identify PHP scripting problems. With DTrace, you have one tool that can trace the interaction of user applications with the operating system.

On Oracle Linux, the DTrace utilities require an Oracle Unbreakable Linux Network (ULN) subscription.

Install Oracle Linux and Oracle's Unbreakable Enterprise Kernel Version 3

The starting point is to install Oracle Linux 6.4 from Oracle E-Delivery.

  1. Once Oracle Linux 6.4 is installed, make sure the "Unbreakable Enterprise Kernel Release 3 for Oracle Linux 6 (x86_64) - Latest" channel is enabled in ULN. Install the UEK3 kernel:
    # yum install kernel-uek
    
  2. Enable the "Oracle Linux 6 Dtrace Userspace Tools (x86_64) - Latest" channel in ULN and install the DTrace utilities:

    # yum install dtrace-utils
    
  3. Reboot to the new UEK3 3.8.13 kernel.

Install PHP

PHP needs to be built with the --enable-dtrace parameter. There are some pre-built, evaluation RPMs you can install that have this enabled; see "DTrace PHP Using Oracle Linux 'playground' Pre-Built Packages."

Alternatively rebuild PHP, as described below:

  1. Download PHP 5.4.20 or PHP 5.5.4 or later from php.net and extract it:

    $ tar -xJf php-5.4.20.tar.bz2
    $ cd php-5.4.20
    
  2. Configure PHP:

    $ ./configure \
    --prefix=$HOME/p54 \
    --enable-dtrace \
    --disable-all --disable-cgi \
    --with-pear --enable-xml --enable-libxml --with-zlib
    

    This builds a minimal command-line PHP with DTrace enabled. All unwanted extensions are disabled. You can include other extensions as needed.

    The --prefix option puts the installation into a local directory, which makes it easy to see the files installed. It is easy to clean up this directory when you are finished with the snapshot.

    The pear, xml, and zlib options allow the use of the pecl command.

  3. Make the PHP binary and install it:

    $ make
    $ make install
    
  4. Copy php.ini-development to $HOME/p54/lib/php.ini and edit it to set the time zone, for example:

    date.timezone = America/Los_Angeles
    

Install PHP OCI8 for Oracle Database

The evaluation PHP RPMs include the OCI8 extension to connect to Oracle Database; see the previously mentioned blog post.

However, if you compiled PHP yourself, then manually add PHP OCI8 as a "shared" extension:

  1. Enable the "Oracle Software for Oracle Linux 6 (x86_64)" channel on ULN.
  2. Install Oracle Instant Client as root:

    # yum install oracle-instantclient12.1-basic oracle-instantclient12.1-devel
    
  3. As a normal user, set PATH so PHP is found:

    $ export PATH=$HOME/p54/bin:$PATH
    
  4. Set a PEAR proxy, if needed for access to http://pecl.php.net:

    $ pear config-set http_proxy http://myproxy.example.com:80/
    
  5. Set environment variable PHP_DTRACE to enable DTrace, and install PHP OCI8:

    $ PHP_DTRACE=yes pecl install oci8-2.0.2
    

    The DTrace probes definitions used later in this article are based on PHP OCI8 2.0.2, so that explicit version is installed. If you install a later version, review the probes and their arguments for the small improvements made in the most recent release.

    When prompted for the ORACLE_HOME directory, press Return without entering text. The installation will autodetect the Oracle Instant Client RPMs. Configuration will continue and the output will contain something like the following:

    [ . . . ]
    checking for Oracle Database OCI8 support... yes, shared
    checking PHP version... 5.4.20, ok
    checking OCI8 DTrace support... yes
    [ . . . ]
    configure: WARNING: OCI8 extension: ORACLE_HOME is not set,
        looking for default Oracle Instant Client instead
    checking Oracle Instant Client directory...
        /usr/lib/oracle/12.1/client64/lib
    checking Oracle Instant Client SDK header directory...
       /usr/include/oracle/12.1/client64
    checking Oracle Instant Client library version compatibility... 12.1
    [ . . . ]
    
  6. Edit php.ini again and add PHP OCI8:

    extension=oci8.so
    
  7. Confirm the installation:

    $ php --ri oci8
    
    oci8
    
    OCI8 Support => enabled
    OCI8 DTrace Support => enabled
    OCI8 Version => 2.0.2-dev
    Revision => $Id: b30fb4bef45d9f5ce8a56b736f1546ea0cff08ef $
    Oracle Run-time Client Library Version => 12.1.0.1.0
    Oracle Compile-time Instant Client Version => 12.1
    
    Directive => Local Value => Master Value
    oci8.max_persistent => -1 => -1
    oci8.persistent_timeout => -1 => -1
    oci8.ping_interval => 60 => 60
    oci8.privileged_connect => Off => Off
    oci8.statement_cache_size => 20 => 20
    oci8.default_prefetch => 100 => 100
    oci8.old_oci_close_semantics => Off => Off
    oci8.connection_class => no value => no value
    oci8.events => Off => Off
    
    Statistics =>  
    Active Persistent Connections => 0
    Active Connections => 0
    

PHP OCI8 Installation Notes

For DTrace support, PHP OCI8 2.0 needs to be installed from PECL because PHP 5.4 and PHP 5.5 have PHP OCI8 1.4, which doesn't have DTrace probes. In the future, when PHP 5.6 is released, you will be able to configure a DTrace-enabled PHP OCI8 while building PHP.

You can, of course, install PHP OCI8 with Oracle Instant Client zip files, or simply use an existing ORACLE_HOME install.

You can DTrace-enable PHP OCI8 on a version of PHP that doesn't have DTrace available or configured. This includes older versions of PHP. You will be able to trace the PHP OCI8 probes but not any core PHP probes. Similarly you can install a DTrace-disabled PHP OCI8 on DTrace-enabled PHP.

If you install PHP OCI8 2.0 from PECL using phpize and configure (instead of pecl), you will still need to set PHP_DTRACE=yes. This is because the --enable-dtrace option will be ignored by the limited configure script of a PECL bundle.

The PHP OCI8 2.0 configuration script is suitable for "real" DTrace use but Linux SystemTap will not trace the extension.

Note that DTrace-optimized binaries might give output that is not quite expected from code observation.

Verify the PHP DTrace Probes

  1. As root, enable DTrace and allow normal users to record trace information:

    # modprobe fasttrap
    # chmod 666 /dev/dtrace/helper
    

    Instead of the chmod command, you could instead use an ACL package rule to limit device access to a specific user.

  2. As a normal user, run php without any options. It will start and wait for input:

    $ php
    
  3. As root, list the DTrace probes that are available. Both PHP core and PHP OCI8 probes are listed:

    # dtrace -l -m php -m oci8.so
     4 php9559     php              dtrace_compile_file compile-file-entry
     5 php9559     php              dtrace_compile_file compile-file-return
     6 php9559     php                       zend_error error
     7 php9559     php ZEND_CATCH_SPEC_CONST_CV_HANDLER exception-caught
     8 php9559     php    zend_throw_exception_internal exception-thrown
     9 php9559     php                dtrace_execute_ex execute-entry
    10 php9559     php          dtrace_execute_internal execute-entry
    11 php9559     php                dtrace_execute_ex execute-return
    12 php9559     php          dtrace_execute_internal execute-return
    13 php9559     php                dtrace_execute_ex function-entry
    14 php9559     php                dtrace_execute_ex function-return
    15 php9559     php             php_request_shutdown request-shutdown
    16 php9559     php              php_request_startup request-startup
    17 php9559 oci8.so  php_oci_dtrace_check_connection oci8-check-connection
    18 php9559 oci8.so               php_oci_do_connect oci8-connect-entry
    19 php9559 oci8.so        php_oci_persistent_helper oci8-connect-expiry
    20 php9559 oci8.so            php_oci_do_connect_ex oci8-connect-lookup
    21 php9559 oci8.so php_oci_pconnection_list_np_dtor oci8-connect-p-dtor-close
    22 php9559 oci8.so php_oci_pconnection_list_np_dtor oci8-connect-p-dtor-release
    23 php9559 oci8.so               php_oci_do_connect oci8-connect-return
    24 php9559 oci8.so            php_oci_do_connect_ex oci8-connect-type
    25 php9559 oci8.so                    php_oci_error oci8-error
    26 php9559 oci8.so        php_oci_statement_execute oci8-execute-mode
    27 php9559 oci8.so             php_oci_create_spool oci8-sesspool-create
    28 php9559 oci8.so           php_oci_create_session oci8-sesspool-stats
    29 php9559 oci8.so           php_oci_create_session oci8-sesspool-type
    30 php9559 oci8.so         php_oci_statement_create oci8-sqltext
    

    The core PHP probes are documented in the PHP Manual. The PHP OCI8 probes are described below.

  4. In your user terminal, stop the php executable with Ctrl-C.

    $ php
    ^C
    $
    

PHP OCI8 2.0 DTrace Probe Overview

The static PHP OCI8 2.0 probes can be categorized as "user" probes and "maintainer" probes. The latter that are more useful for PHP OCI8 maintainers to verify functionality during development of the extension itself. All the probes return data in arguments. The PHP OCI8 DTrace probes are described in the official documentation.

In OCI8 2.0.2, the User probes are:

  • oci8-connect-entry—Initiated by oci_connect(), oci_pconnect(), and oci_new_connect(). Fires before database connection is established.

    • char *username—The connection username.
    • char *dbname—The database connection string.
    • char *charset—The character set specified.
    • long session_mode—A binary "or" of OCI_SYSDBA (0x2), OCI_SYSOPER (0x4), and OCI_CRED_EXT (1<<31) (or -2147483648 on the platform I was using). Set to 0 by default.
    • int persistent—Set to 1 if oci_pconnect() was called; 0 otherwise.
    • int exclusive—Set to 1 if oci_new_connect() was called; 0 otherwise.
  • oci8-connect-return—Fires at the end of connection.

    • void *connection—The address of the connection structure.
  • oci8-check-connection—Initiated if an Oracle error might have caused the connection to become invalid.

    • void *connection—The address of the connection structure.
    • int is_open—Will be 0 if errcode or server_status indicate the connection is invalid and must be re-created.
    • long errcode—The Oracle error number.
    • unsigned long server_status—An indicator from the Oracle library if the connection is considered invalid. If is_open is 0 because errcode indicated the connection was invalid, then server_status will be its default of 1.
  • oci8-sqltext—Initiated when oci_parse() is executed.

    • void *connection—The address of the connection structure.
    • char *sql—Text of the SQL statement executed.
  • oci8-error—Initiated if an Oracle error occurs.

    • int status—The Oracle return status of the failing Oracle library call, such as -1 for Oracle's OCI_ERROR or 1 for Oracle's OCI_SUCCESS_WITH_INFO. See Oracle's oci.h for all definitions.
    • long errcode—The Oracle error number.
  • oci8-execute-mode—Indicates the commit state of an oci_execute() call.

    • void *connection—The address of the connection structure.
    • unsigned int mode—The mode passed to the Oracle library, such as OCI_NO_AUTO_COMMIT (0x00), OCI_DESCRIBE_ONLY (0x10), or OCI_COMMIT_ON_SUCCESS (0x20).

Note: An oci8-connection-close probe is available in the latest OCI8 2.0.6, and several probes now also have a client_id argument and a pointer to the statement structure.

Maintainer probes are below. Refer to the PHP OCI8 source code for the argument descriptions.

  • oci8-connect-p-dtor-close

    • void *connection
  • oci8-connect-p-dtor-release

    • void *connection
  • oci8-connect-lookup

    • void *connection
    • int is_stub
  • oci8-connect-expiry

    • void *connection
    • int is_stub
    • long idle_expiry
    • long timestamp
  • oci8-connect-type

    • int persistent
    • int exclusive
    • void *connection
    • long num_persistent
    • long num_connections
  • oci8-sesspool-create

    • void *session_pool
  • oci8-sesspool-stats

    • unsigned long free
    • unsigned long busy
    • unsigned long open
  • oci8-sesspool-type

    • int type
    • void *session_pool

The probes in PHP OCI8 2.0 replace PHP OCI8 1.4's use of oci_internal_debug() tracing. This function has become a no-op.

Using PHP OCI8 and DTrace

Follow these steps.

  1. Create a simple PHP file, oci8.php, to query the database:

        <?php
    
        error_reporting(0);
        ini_set('display_errors', 'Off');
    
        function do_query($c, $sql)
        {
            $s = oci_parse($c, $sql);
            if (!$s)
                return;
            $r = oci_execute($s);
            if (!$r)
                return;
            while (($row = oci_fetch_row($s)) != false) {
                foreach ($row as $item) {
                    echo $item . " ";
                }
                echo "\n";
            }
        }
    
        $c = oci_new_connect('hr', 'welcome', 'localhost/pdborcl');
    
        do_query($c, "select city from locations where rownum < 5 order by 1");
        do_query($c, "select something from does_not_exist");
    
        ?>
    
  2. Create a D script, user_oci8.d, to probe the execution of oci8.php:

        #!/usr/sbin/dtrace -Zs
    
        # This script is for OCI8 2.0.2
    
        php*:::oci8-connect-entry
        {
            printf("PHP connect-entry\n");
            printf("\t   username      %s\n", arg0 ? copyinstr(arg0) : "");
            printf("\t   dbname        %s\n", arg1 ? copyinstr(arg1) : "");
            printf("\t   charset       %s\n", arg2 ? copyinstr(arg2) : "");
            printf("\t   session_mode  %ld\n", (long)arg3);
            printf("\t   persistent    %d\n", (int)arg4);
            printf("\t   exclusive     %d\n", (int)arg5);
        }
    
        php*:::oci8-connect-return
        {
            printf("PHP oci8-connect-return\n");
            printf("\t   connection    0x%p\n", (void *)arg0);
        }
    
        php*:::oci8-connection-close
        {
            printf("PHP oci8-connect-close\n");
            printf("\t   connection    0x%p\n", (void *)arg0);
        }
    
        php*:::oci8-error
        {
            printf("PHP oci8-error\n");
            printf("\t   status        %d\n", (int)arg0);
            printf("\t   errcode       %ld\n", (long)arg1);
        }
    
        php*:::oci8-check-connection
        {
            printf("PHP oci8-check-connection\n");
            printf("\t   connection    0x%p\n", (void *)arg0);
            printf("\t   is_open       %d\n", arg1);
            printf("\t   errcode       %ld\n", (long)arg2);
            printf("\t   server_status %lu\n", (unsigned long)arg3);
        }
    
        php*:::oci8-sqltext
        {
            printf("PHP oci8-sqltext\n");
            printf("\t   connection    0x%p\n", (void *)arg0);
            printf("\t   sql           %s\n", arg0 ? copyinstr(arg1) : "");
        }
    
        php*:::oci8-execute-mode
        {
            printf("PHP oci8-execute-mode\n");
            printf("\t   connection    0x%p\n", (void *)arg0);
            printf("\t   mode          0x%x\n", arg1);
        }
    

    With OCI8 2.0.6, some of the argument numbers will need adjusting to cater to the extra arguments that are now available.

  3. As root, start the D script. It will pause, waiting for probes to be fired:

        # chmod +x user_oci8.d
        # ./user_oci8.d
    

    (Later, this terminal can be closed using Ctrl-C when you have finished experimenting with PHP.)

  4. Run command-line PHP in another window. The output from the successful query is displayed:

        $ php oci8.php
        Beijing 
        Bern 
        Bombay 
        Geneva 
    
  5. In the root terminal running the D script, the probes firing during execution of PHP will be displayed:

        # ./user_oci8.d
        dtrace: script 'user_oci8.d' matched 0 probes
        CPU  ID                    FUNCTION:NAME
        1    18 php_oci_do_connect:oci8-connect-entry PHP connect-entry
                username      hr
                dbname        localhost/pdborcl
                charset       
                session_mode  0
                persistent    0
                exclusive     0
    
        0    23 php_oci_do_connect:oci8-connect-return PHP oci8-connect-return
                connection    0x7f64e112cff0
    
        0    31 php_oci_statement_create:oci8-sqltext PHP oci8-sqltext
                connection    0x7f64e112cff0
                sql           select city from locations where rownum < 5 order by 1
    
        0    27 php_oci_statement_execute:oci8-execute-mode PHP oci8-execute-mode
                connection    0x7f64e112cff0
                mode          0x20
    
        0    31 php_oci_statement_create:oci8-sqltext PHP oci8-sqltext
                connection    0x7f64e112cff0
                sql           select something from does_not_exist
    
        0    27 php_oci_statement_execute:oci8-execute-mode PHP oci8-execute-mode
                connection    0x7f64e112cff0
                mode          0x20
    
        0    26 php_oci_error:oci8-error PHP oci8-error
                status        -1
                errcode       942
    
        0    17 php_oci_dtrace_check_connection:oci8-check-connection PHP oci8-check-connection
                connection    0x7f64e112cff0
                is_open       1
                errcode       942
                server_status 1
    
        0    25 php_oci_connection_close:oci8-connection-close PHP oci8-connect-close
                 connection    0x7f64e112cff0
    

    (Adding -q to the /usr/sbin/dtrace arguments in user_oci8.d will suppress the CPU and ID details.)

    On multi-CPU machines, the probe ordering might not appear sequential, depending on which CPU was processing the probes. Displaying probe time stamps will help reduce confusion, for example:

        php*:::oci8-connect-entry
        {
            printf("PHP connect-entry at %lld\n", walltimestamp);
        }
    

    From the user_oci8.d DTrace output, you can see the following:

    • The connection being initiated (oci8-connect-entry). The user hr connected to the localhost/pdborcl database. It was an oci_connect() call because both exclusive and persistent were 0. No explicit character set was requested. The default session mode (the optional fifth parameter to oci_connect) was requested.
    • Two SQL statements being parsed (oci8-sqltext) and executed (oci-execute-mode) with mode 0x20 (aka OCI_COMMIT_ON_SUCCESS).
    • An Oracle error—ORA-942 (table or view does not exist)—that was generated (oci8-error).
    • The error causing the connection status to be verified (oci8-check-connection). The value of is_open is 1, indicating that the connection is OK.

    With this information you can trace problematic statement execution and connection issues.

Conclusion

This is just a morsel about using DTrace, which is a very powerful utility. Following on from the example above, you could integrate PHP OCI8 tracing with core PHP tracing. You can time PHP function calls, and count the number of invocations. You can drill down and see what operating systems calls are being made by PHP. Bryan Cantrill posted some examples of core PHP tracing in DTrace and PHP, demonstrated. (Note that blog platform upgrades have caused single backslashes to be displayed as double backslashes in his post. Also, you no longer need the separate PHP DTrace extension.) Brendan Gregg's DTrace Toolkit contains many useful DTrace scripts. There are various blogs, too.

Remember that the intent of DTrace is that its functionality is enabled all the time, suitable for development and ready for when you need it most: in production. The design of DTrace means that the probes have zero overhead when nothing is monitoring them.

See Also

Christopher Jones' PHP and Oracle blog

About the Author

Christopher works in the Linux and Virtualization group at Oracle, focusing on upstream tools. He is a lead maintainer of PHP's OCI8 extension for Oracle Database and is the author of many technical articles on PHP and on Oracle technology. He co-authored the popular book The Underground PHP and Oracle Manual, which is free from OTN.

Revision 1.0, 11/18/2013

Follow us:
Blog | Facebook | Twitter | YouTube