by Anderson Souza
Published July 2013
This article is Part 7 of a seven-part series that provides best practices and recommendations for configuring VMware vSphere 5.x with Oracle ZFS Storage Appliance to reach optimal I/O performance and throughput. The best practices and recommendations highlight configuration and tuning options for Fibre Channel, NFS, and iSCSI protocols.
The series also includes recommendations for the correct design of network infrastructure for VMware cluster and multi-pool configurations, as well as the recommended data layout for virtual machines. In addition, the series demonstrates the use of VMware linked clone technology with Oracle ZFS Storage Appliance.
All the articles in this series can be found here:
Note: For a white paper on this topic, see the Sun NAS Storage Documentation page.
The Oracle ZFS Storage Appliance product line combines industry-leading Oracle integration, management simplicity, and performance with an innovative storage architecture and unparalleled ease of deployment and use. For more information, see the Oracle ZFS Storage Appliance Website and the resources listed in the "See Also" section at the end of this article.
Note: References to Sun ZFS Storage Appliance, Sun ZFS Storage 7000, and ZFS Storage Appliance all refer to the same family of Oracle ZFS Storage Appliances.
DTrace Analytics is a powerful and advanced monitoring tool provided by Oracle ZFS Storage Appliance. DTrace Analytics gives the storage administrator unique visibility into the entire system: monitoring in real time of different statistics for the operating system stack, storage resources, and protocols used as well as I/O throughput and performance of the virtualized environment.
VMware provides a powerful monitoring tool called ESXTOP that is used at the VMware ESXi host level to monitor performance and resource usage of the virtual environment. With this tool, you can identify possible bottlenecks, I/O performance issues, and network degradation as well as throughput levels.
For the most realistic report, VMware ESXTOP and DTrace Analytics should always be used together to validate as well as monitor your entire VMware storage performance and throughput. To ensure that your VMware NFS configuration is properly working, use the following DTrace Analytics and ESXTOP options.
The following examples show how to use ESXTOP and DTrace Analytics to monitor VMware Fibre Channel and iSCSI LUNs as well as datastore and HBA performance and throughput.
For VMware ESXTOP, open an SSH connection with your ESXi5.x host and run the following commands:
esxtopand then press n for monitoring VMware Fibre Channel or iSCSI LUNs.
Figure 1 shows the VMware ESXTOP output for the
Note: For interpreting VMware ESXTOP statistics, read VMware DOC-9279.
Figure 1. Monitoring Fibre Channel and iSCSI LUNs with VMware ESXTOP
d or disk view option (HBA mode) can be used for monitoring virtual HBAs. Figure 2 shows output for this option.
To perform this task, type the
esxtop command, and then press f to choose different monitoring options. Press s 2 to alter the update time to every 2 seconds, and then press Enter.
Note: Ensure that the virtual HBAs (vmhbas) are correctly balancing the I/O. Figure 2 highlights
vmhba7. Watch all options available on the screen shown in Figure 2, and then compare them with DTrace Analytics outputs.
Figure 2. Monitoring VMware HBAs with ESXTOP
Figure 3 through Figure 6 show different examples of DTrace Analytics options that can be used in combination with VMware ESXTOP for monitoring Fibre Channel performance and throughput.
Figure 3. Example 1: Monitoring Fibre Channel protocol using DTrace Analytics
Figure 4. Example 2: Monitoring Fibre Channel protocol using DTrace Analytics
Figure 5. Example 3: Monitoring Fibre Channel protocol using DTrace Analytics
Figure 6. Example 4: Monitoring Fibre Channel protocol using DTrace Analytics
Figure 7 through Figure 11 show examples of VMware ESXTOP and DTrace Analytics output used to monitor utilization and performance of NFS datastores as well as the IP network.
Figure 7 through Figure 11 show that the NFSv3 protocol is used for virtual machines' disk datastore. In this approach, DTrace Analytics is monitoring the virtual machines' disk usage in IOPS for each
Figure 7 shows the VMware ESXTOP option for monitoring NFS datastores. To perform this, type
esxtop, and then press u. Press s 2 to alter the update time to every 2 seconds, and press Enter.
Figure 7. Monitoring VMware NFS datastores with VMware ESXTOP
Figure 8 shows the VMware ESXTOP option for monitoring virtual machines. To perform this, type
esxtop, and then press v. Press s 2 to alter the update time to every 2 seconds, and press Enter.
Figure 8. Monitoring VMware virtual machine with VMware ESXTOP
Figure 9 shows DTrace Analytics options for monitoring network datalink as well as interfaces and TCP bytes.
Figure 9. Monitoring network datalinks, interfaces, and TCP bytes with DTrace Analytics
Figure 10 shows DTrace Analytics options for monitoring the NFS protocol broken down by type of operation and clients and also per file name—in this case,
Figure 10. Monitoring NFS protocol broken down by type of operation, clients, and file name with DTrace Analytics
Figure 11 shows more DTrace Analytics options for monitoring the NFS protocol broken down by latency and by size, as well as cache ARC broken down by hit/miss.
Figure 11. Monitoring NFS protocol broken down by latency and size, and cache ARC broken down by hit/miss with DTrace Analytics
Figure 12 and Figure 13 show examples of VMware ESXTOP and DTrace Analytics monitoring iSCSI protocol utilization and performance.
Note: Some options used to monitor the NFS protocol, such as TCP and interfaces, as well as some options used to monitor the Fibre Channel protocol can be used for monitoring the iSCSI protocol as well.
Figure 12 shows the VMware ESXTOP option for monitoring iSCSI datastores. To perform this, type
esxtop, and press u. Press s 2 to alter the update time to every 2 seconds, and press Enter.
Figure 12. Monitoring iSCSI protocol utilization and performance with VMware ESXTOP
Figure 13 shows the VMware ESXTOP option for monitoring iSCSI virtual HBAs. The example shows the virtual HBA
Figure 13. Monitoring iSCSI virtual HBAs with VMware ESXTOP
Figure 14 through Figure 16 show examples of DTrace Analytics options that can be used in combination with VMware ESXTOP for monitoring iSCSI performance and throughput.
Figure 14 shows DTrace Analytics options for monitoring the iSCSI protocol broken down by initiators, targets, and LUNs.
Figure 14. Monitoring iSCSI protocol broken down by initiators, targets, and LUNs with DTrace Analytics
Figure 15 shows more DTrace Analytics options for monitoring the iSCSI protocol broken down by client and LUNs (operations per second) as well as network interfaces broken down by interface (bytes per second). The LUNs display appears at the bottom of Figure 15.
Figure 15. Monitoring iSCSI protocol broken down by clients, network interfaces, and LUNs with DTrace Analytics
Figure 16 shows more DTrace Analytics options for monitoring iSCSI protocol broken down by latency and devices.
Note: Ensure that all your 10GbE NICs' members of the VMware port binding configuration are correctly balancing the I/O traffic. The example in Figure 16 shows two 10GbE NICs (ixgbe0 and ixgbe1) balancing the I/O traffic. That is the expected behavior when you are working with an iSCSI port binding configuration as well as a storage array type
VMW_SATP_ALUA and path selection policy
VMW_PSP_RR. If you do not see this behavior, review your iSCSI port binding configuration as well as the VMware path policy and storage array type. Also review the network configuration of the Oracle ZFS Storage Appliance and the port-channel configurations of your IP switches.
Figure 16 Monitoring iSCSI protocol broken down by latency and network devices with DTrace Analytics
Also, you can work with VMware ESXTOP in batch mode, which permits you to output data into a CSV format, and then use
perfmon on Microsoft Windows or even VMware
esxplot to view the data/results. To run ESXTOP in batch mode, type the following:
esxtop -b > esxtop_whatever.csv.
It is always a best practice to use more DTrace Analytics options, such as ARC, L2ARC access by hit/miss and latency, as well as disk I/O outputs.
Oracle ZFS Storage Appliance offers outstanding performance for virtualized environments. Its architecture's features and intelligent caching technology are designed to deliver thousands of IOPS for your virtualized environment as well as the best throughput and response time for your virtualized applications and databases.
VMware is a robust hypervisor and also provides an easy way to manage your virtualized infrastructure. In combination, the VMware and Oracle ZFS Storage Appliance platforms and technologies are an excellent choice for your virtualized environment.
Refer to the following websites for further information on testing results for Oracle ZFS Storage Appliance:
Also see the following documentation and websites:
Anderson Souza is a virtualization senior software engineer in Oracle's Application Integration Engineering group. He joined Oracle in 2012, bringing more than 14 years of technology industry, systems engineering, and virtualization expertise. Anderson has a Bachelor of Science in Computer Networking, a master's degree in Telecommunication Systems/Network Engineering, and also an MBA with a concentration in project management.
|Revision 1.0, 07/09/2013|