Network Virtualization and Network Resource Management

by Detlef Drewanz

This article, which is Part 5 in a series of virtualization articles, defines internal and external network virtualization and then discusses the features of internal network virtualization and network resource management.

Published January 2013 (reprinted from eStep blog)

Part 8 - Oracle Enterprise Manager Ops Center as a Management Tool for Virtualization
Part 7 - The Role of Oracle Virtual Desktop Infrastructure in a Virtualization Strategy
Part 6 - Oracle VM VirtualBox - Personal Desktop Virtualization
Part 5 - Network Virtualization and Network Resource Management
Part 4 - Resource Management as an Enabling Technology for Virtualization
Part 3 - The Role of Oracle Solaris Zones and Linux Containers in a Virtualization Strategy
Part 2 - The Role of Oracle VM Server for x86 in a Virtualization Strategy
Part 1 - The Role of Oracle VM Server for SPARC in a Virtualization Strategy

After discussing Oracle VM, OS virtualization, and some aspects of resource management in the previous articles of this series, this article will now cover a special area of resource management and virtualization of resources: network virtualization and network resource management.

The network is a special shared resource that glues all the virtual machines (VMs), zones, and systems together and provides a communication channel with the world. Thus, the network is a very important layer of the virtualization stack.

Network virtualization is categorized as external or internal.

• External network virtualization combines many networks, switches, network ports, virtual ports, and virtual network interfaces into virtual units called virtual LANs (VLANs). VLANs are created by using VLAN tags to group different ports, switches, and physical networks together into one common virtual network. A VLAN tag is an identifier that is sent together with network packets to identify which packets belong to a virtual network. A virtual network can also be called a broadcast domain, which is a group of network participants that all receive a network broadcast.
• Internal network virtualization is the virtualization of a network stack, network interfaces, or other networking functionality within one system. This virtualization functionality is provided by the host OS or the hypervisor. Internal network virtualization enables the shared use of a limited number of network ports by many VMs, zones, and containers. All virtualized environments need their own network interfaces, and with internal network virtualization, some physical network interface cards (PNICs) can be divided into many virtual network interface cards (VNICs). This is one of the basic functionalities of internal network virtualization.

Because a shared-resource network is highly used by many consumers—such as processes, VMs, zones, and containers—network resource management is very important for network virtualization. Network resource management helps to deliver powerful, stable network connections to virtualized environments, and the available network bandwidth can be better spread among multiple virtualized environments to meet service level agreements. Extensive use of network virtualization should be considered only with well-implemented network resource management.

Using hypervisor-based virtualization and Oracle Solaris Zones with network virtualization and network resource management enables a whole range of new capabilities for creating network-based architectures. Figure 1 shows one example, in which physical systems and network components have been replaced by Oracle Solaris Zones and virtual switches.

Figure 1. Example Network-Based Architecture

In this article, we concentrate on the functionalities and side effects of internal network virtualization and network resource management in conjunction with hypervisors, containers, and zones in one system.

Features of Internal Network Virtualization

The following base features are common across various type of hypervisors and zone technologies; however, specific implementations differ.

• VNICs are needed to share a small number of PNICs with a larger number of VMs or zones—let's call the VMs and zones consumers. Every consumer requires its own network interface that it can use as if it were a physical port. It is the task of the hypervisor, the host operating system, or the global zone to provide this network interface. The administrator can decide whether this network interface is mapped to a dedicated physical port or it is a VNIC that is assigned to a shared physical port. In the latter case, the physical port is shared by many VNICs, and resource management features are useful for limiting the bandwidth each VNIC can use.
  
  

  Figure 2 shows an example of how VNICs are built in Oracle Solaris on top of physical interfaces and then are used by Oracle Solaris Zones. In this example, we also use bandwidth limitations assigned to VNICs.

  
  

  Figure 2. VNICs Built on Top of PNIC

• Virtual network switches connect multiple VNICs that are created on one physical interface. This makes it possible for VNICs on one physical port to communicate with each other and also to share the physical interface. The name for this feature varies with different products, but the functionality is similar. In Oracle VM Server for x86, this functionality is called a bridge, which is automatically created if a VNIC is created on a physical port.
  
  
  

  Figure 3. Oracle VM Server for x86 and a Network Bridge

  For Oracle VM Server for SPARC, a virtual switch has to be created by the administrator in the service domain to which the network interfaces of the guest domains connect. The logical domain channels (LDC) establish the link between the virtual switch and the guest domains. Oracle Solaris creates a switch above the physical interface when the first VNIC is created. Oracle VM VirtualBox creates virtual PCIe cards and assigns them to VMs as network interfaces. There are different ways these interfaces communicate with the host operating system or the outside world (for example, NAT, bridged networking, internal networking, and host-only networking).

  
  

  Figure 4. Oracle VM Server for SPARC and the Virtual Switch

• A special implementation of a virtual network switch that is available only in Oracle Solaris 11 is an etherstub (see Figure 5). This is a special type of data link that can be used instead of a physical NIC to create VNICs and the virtual switches that connect them. With etherstubs, complex network architectures or just network-in-a-box setups can be created and tested without needing any physical network switches.
  
  
  

  Figure 5. Oracle Solaris Etherstubs

• If Oracle Solaris Zones are used, IP interfaces, VNICs, or physical interfaces are provided by the global zone. An Oracle Solaris Zone can then use a shared-IP instance or an exclusive-IP instance to communicate with the global zone or the outside world, as shown in Figure 6.
  
  

  With a shared-IP instance, the zones share one IP stack infrastructure in the kernel, including its ARP cache, routing table, and IP configuration flags (but not the IP address). A zone with an exclusive-IP instance has its own IP stack. To use an exclusive-IP instance, a dedicated physical interface or VNIC is needed. Using a shared-IP instance does not require a dedicated network interface.

  
  

  Figure 6. Shared-IP and Exclusive-IP Instances

Features of Network Resource Management

The network is always a shared resource, either outside the server chassis—by using central cables, switches, or routers—or inside the chassis—by sharing physical ports, network stacks, or just the CPUs that are handling the traffic by doing check-summing or handling the network adapter interrupts. To meet the different service level agreements of the network consumers in one chassis, network resource management is needed. The requirements can be based on available network bandwidth, the network latency, or the network data loss rate. While network latency and data loss rate are typically based on the network technology that is used and the OS- or hypervisor-specific implementation, the available bandwidth can be controlled by resource management. Various product-specific implementations exist that are related to internal network virtualization:

• Dedication of a network port enables the host or the hypervisor to assign a separate physical port to a consumer. With this, the consumer gets the whole bandwidth of the port, but it might need many network ports and many network adapters, which might be limited by the number of available PCI slots.
• A specific CPU can be assigned to network interfaces or VNICs to handle their device interrupts, doing the data buffer handling and computing network checksums. We can compare these two functions with resource partitioning, which was a resource management feature described in the previous article of this series.
• During the creation of VNICs, an interface-based network bandwidth cap can be assigned. With that, the usable bandwidth is capped on a configured boundary, which enables the sharing of a physical network port by many network consumers by limiting the usable bandwidth for each consumer. This setup is very flexible and can often be changed dynamically. We discussed this resource constraints functionality in the previous article of this series.
• Network bandwidth capping with PNICs and VNICs is interface-based, but there is also a need to control the bandwidth on a logical network-connection basis. A network connection can be described by a source IP address, a destination IP address, and a protocol. In Oracle Solaris, this is called a flow. The configured flows can be used to control network bandwidth—independent of network interfaces—just on a connection basis. Figure 7 shows an example.
  
  

  In Figure 7, a configured flow for the network data type "network backup" can be used to give the green and the blue traffic more available bandwidth in critical load situations. We discussed this functionality as resource scheduling in the previous article, because if "green" and "blue" do not have bandwidth needs, "network backup" can get the maximum available bandwidth.

  
  

  Figure 7. Example of Configured Flow

Conclusion

Virtual network interfaces, virtual bridges, virtual switches, and virtual PCIe cards are basic internal network virtualization features that are part of virtualization products. These networking features "glue" all the VMs, zones, and containers together and enable them to communicate among themselves and with the outside world. To enable stable communication for all of them on a shared-resource network, the use of network resource management features is recommended. We have also seen that for networks, various types of resource management, such as constraints, scheduling, and partitioning, are used.

See Also

• Part 1 of this series, "The Role of Oracle VM Server for SPARC in a Virtualization Strategy," which provides an introduction to the different levels and types of virtualization as well as the functionality and benefits of Oracle VM Server for SPARC
• Part 2 of this series, "The Role of Oracle VM Server for x86 in a Virtualization Strategy," which describes the functionality and benefits of Oracle VM Server for x86
• Part 3 of this series, "The Role of Oracle Solaris Zones and Linux Containers in a Virtualization Strategy," which covers the operating system level of virtualization
• Part 4 of the series, "Resource Management as an Enabling Technology for Virtualization," which covers various resource management strategies that enable virtualization

About the Author

Detlef is a Principal Sales Consultant and is located in Potsdam, Germany. He acts as server and Oracle Solaris specialist on Oracle's Northern Europe Server Architects team. He joined Sun Microsystems in 1998 and is now part of Oracle. Prior to that, Detlef worked at Hitachi Internetworking Frankfurt in network support and as member of scientific staff in the Department of Computer Science of the University of Rostock. Detlef holds a master's degree in computer science.