Monitoring VMware with DTrace Analytics and ESXTOP

Best Practices for Oracle ZFS Storage Appliance and VMware vSphere 5.x: Part 7

by Anderson Souza

This article provides information about monitoring the throughput and performance of VMware using the VMware ESXTOP and Oracle ZFS Storage Appliance DTrace Analytics tools.


Published July 2013


This article is Part 7 of a seven-part series that provides best practices and recommendations for configuring VMware vSphere 5.x with Oracle ZFS Storage Appliance to reach optimal I/O performance and throughput. The best practices and recommendations highlight configuration and tuning options for Fibre Channel, NFS, and iSCSI protocols.

Want to comment on this article? Post the link on Facebook's OTN Garage page.  Have a similar article to share? Bring it up on Facebook or Twitter and let's discuss.

The series also includes recommendations for the correct design of network infrastructure for VMware cluster and multi-pool configurations, as well as the recommended data layout for virtual machines. In addition, the series demonstrates the use of VMware linked clone technology with Oracle ZFS Storage Appliance.

All the articles in this series can be found here:

Note: For a white paper on this topic, see the Sun NAS Storage Documentation page.

The Oracle ZFS Storage Appliance product line combines industry-leading Oracle integration, management simplicity, and performance with an innovative storage architecture and unparalleled ease of deployment and use. For more information, see the Oracle ZFS Storage Appliance Website and the resources listed in the "See Also" section at the end of this article.

Note: References to Sun ZFS Storage Appliance, Sun ZFS Storage 7000, and ZFS Storage Appliance all refer to the same family of Oracle ZFS Storage Appliances.

About DTrace Analytics and ESXTOP

DTrace Analytics is a powerful and advanced monitoring tool provided by Oracle ZFS Storage Appliance. DTrace Analytics gives the storage administrator unique visibility into the entire system: monitoring in real time of different statistics for the operating system stack, storage resources, and protocols used as well as I/O throughput and performance of the virtualized environment.

VMware provides a powerful monitoring tool called ESXTOP that is used at the VMware ESXi host level to monitor performance and resource usage of the virtual environment. With this tool, you can identify possible bottlenecks, I/O performance issues, and network degradation as well as throughput levels.

For the most realistic report, VMware ESXTOP and DTrace Analytics should always be used together to validate as well as monitor your entire VMware storage performance and throughput. To ensure that your VMware NFS configuration is properly working, use the following DTrace Analytics and ESXTOP options.

Monitoring Fibre Channel and iSCSI LUN Performance

The following examples show how to use ESXTOP and DTrace Analytics to monitor VMware Fibre Channel and iSCSI LUNs as well as datastore and HBA performance and throughput.

For VMware ESXTOP, open an SSH connection with your ESXi5.x host and run the following commands:

  1. Type esxtop and then press n for monitoring VMware Fibre Channel or iSCSI LUNs.
  2. Press s 2 to alter the update time to every 2 seconds, and press Enter.

Figure 1 shows the VMware ESXTOP output for the n option.

Note: For interpreting VMware ESXTOP statistics, read VMware DOC-9279.

Figure 1

Figure 1. Monitoring Fibre Channel and iSCSI LUNs with VMware ESXTOP

Option d or disk view option (HBA mode) can be used for monitoring virtual HBAs. Figure 2 shows output for this option.

To perform this task, type the esxtop command, and then press f to choose different monitoring options. Press s 2 to alter the update time to every 2 seconds, and then press Enter.

Note: Ensure that the virtual HBAs (vmhbas) are correctly balancing the I/O. Figure 2 highlights vmhba6 and vmhba7. Watch all options available on the screen shown in Figure 2, and then compare them with DTrace Analytics outputs.

Figure 2

Figure 2. Monitoring VMware HBAs with ESXTOP

Figure 3 through Figure 6 show different examples of DTrace Analytics options that can be used in combination with VMware ESXTOP for monitoring Fibre Channel performance and throughput.

Figure 3

Figure 3. Example 1: Monitoring Fibre Channel protocol using DTrace Analytics

Figure 4

Figure 4. Example 2: Monitoring Fibre Channel protocol using DTrace Analytics

Figure 5

Figure 5. Example 3: Monitoring Fibre Channel protocol using DTrace Analytics

Figure 6

Figure 6. Example 4: Monitoring Fibre Channel protocol using DTrace Analytics

Monitoring NFS Performance

Figure 7 through Figure 11 show examples of VMware ESXTOP and DTrace Analytics output used to monitor utilization and performance of NFS datastores as well as the IP network.

Figure 7 through Figure 11 show that the NFSv3 protocol is used for virtual machines' disk datastore. In this approach, DTrace Analytics is monitoring the virtual machines' disk usage in IOPS for each vmdk file.

Figure 7 shows the VMware ESXTOP option for monitoring NFS datastores. To perform this, type esxtop, and then press u. Press s 2 to alter the update time to every 2 seconds, and press Enter.

Figure 7

Figure 7. Monitoring VMware NFS datastores with VMware ESXTOP

Figure 8 shows the VMware ESXTOP option for monitoring virtual machines. To perform this, type esxtop, and then press v. Press s 2 to alter the update time to every 2 seconds, and press Enter.

Figure 8

Figure 8. Monitoring VMware virtual machine with VMware ESXTOP

Figure 9 shows DTrace Analytics options for monitoring network datalink as well as interfaces and TCP bytes.

Figure 9

Figure 9. Monitoring network datalinks, interfaces, and TCP bytes with DTrace Analytics

Figure 10 shows DTrace Analytics options for monitoring the NFS protocol broken down by type of operation and clients and also per file name—in this case, .vmdks files.

Figure 10

Figure 10. Monitoring NFS protocol broken down by type of operation, clients, and file name with DTrace Analytics

Figure 11 shows more DTrace Analytics options for monitoring the NFS protocol broken down by latency and by size, as well as cache ARC broken down by hit/miss.

Figure 11

Figure 11. Monitoring NFS protocol broken down by latency and size, and cache ARC broken down by hit/miss with DTrace Analytics

Monitoring iSCSI Performance

Figure 12 and Figure 13 show examples of VMware ESXTOP and DTrace Analytics monitoring iSCSI protocol utilization and performance.

Note: Some options used to monitor the NFS protocol, such as TCP and interfaces, as well as some options used to monitor the Fibre Channel protocol can be used for monitoring the iSCSI protocol as well.

Figure 12 shows the VMware ESXTOP option for monitoring iSCSI datastores. To perform this, type esxtop, and press u. Press s 2 to alter the update time to every 2 seconds, and press Enter.

Figure 12

Figure 12. Monitoring iSCSI protocol utilization and performance with VMware ESXTOP

Figure 13 shows the VMware ESXTOP option for monitoring iSCSI virtual HBAs. The example shows the virtual HBA vmhba39.

Figure 13

Figure 13. Monitoring iSCSI virtual HBAs with VMware ESXTOP

Figure 14 through Figure 16 show examples of DTrace Analytics options that can be used in combination with VMware ESXTOP for monitoring iSCSI performance and throughput.

Figure 14 shows DTrace Analytics options for monitoring the iSCSI protocol broken down by initiators, targets, and LUNs.

Figure 14

Figure 14. Monitoring iSCSI protocol broken down by initiators, targets, and LUNs with DTrace Analytics

Figure 15 shows more DTrace Analytics options for monitoring the iSCSI protocol broken down by client and LUNs (operations per second) as well as network interfaces broken down by interface (bytes per second). The LUNs display appears at the bottom of Figure 15.

Figure 15

Figure 15. Monitoring iSCSI protocol broken down by clients, network interfaces, and LUNs with DTrace Analytics

Figure 16 shows more DTrace Analytics options for monitoring iSCSI protocol broken down by latency and devices.

Note: Ensure that all your 10GbE NICs' members of the VMware port binding configuration are correctly balancing the I/O traffic. The example in Figure 16 shows two 10GbE NICs (ixgbe0 and ixgbe1) balancing the I/O traffic. That is the expected behavior when you are working with an iSCSI port binding configuration as well as a storage array type VMW_SATP_ALUA and path selection policy VMW_PSP_RR. If you do not see this behavior, review your iSCSI port binding configuration as well as the VMware path policy and storage array type. Also review the network configuration of the Oracle ZFS Storage Appliance and the port-channel configurations of your IP switches.

Figure 16

Figure 16 Monitoring iSCSI protocol broken down by latency and network devices with DTrace Analytics

Also, you can work with VMware ESXTOP in batch mode, which permits you to output data into a CSV format, and then use perfmon on Microsoft Windows or even VMware esxplot to view the data/results. To run ESXTOP in batch mode, type the following:

esxtop -b > esxtop_whatever.csv. 

It is always a best practice to use more DTrace Analytics options, such as ARC, L2ARC access by hit/miss and latency, as well as disk I/O outputs.

Conclusion

Oracle ZFS Storage Appliance offers outstanding performance for virtualized environments. Its architecture's features and intelligent caching technology are designed to deliver thousands of IOPS for your virtualized environment as well as the best throughput and response time for your virtualized applications and databases.

VMware is a robust hypervisor and also provides an easy way to manage your virtualized infrastructure. In combination, the VMware and Oracle ZFS Storage Appliance platforms and technologies are an excellent choice for your virtualized environment.

See Also

Refer to the following websites for further information on testing results for Oracle ZFS Storage Appliance:

Also see the following documentation and websites:

About the Author

Anderson Souza is a virtualization senior software engineer in Oracle's Application Integration Engineering group. He joined Oracle in 2012, bringing more than 14 years of technology industry, systems engineering, and virtualization expertise. Anderson has a Bachelor of Science in Computer Networking, a master's degree in Telecommunication Systems/Network Engineering, and also an MBA with a concentration in project management.

Revision 1.0, 07/09/2013

Follow us:
Blog | Facebook | Twitter | YouTube