by Gregory King and Don Smerker
Published June 2013
Oracle VM Server for x86 supports a wide range of options in network design. This article discusses the basic concepts relevant to Oracle VM networking. Understanding these concepts will help you make informed design choices and enable you to build a scalable, robust network architecture that has high degree of channel separation.
This article focuses on understanding a complex Oracle VM network infrastructure that uses bonded interfaces with VLAN tags specifically for Oracle VM 3.1 and 3.2. (For information on other releases, see the release-specific documentation.) Topics include Xen domains, physical hardware choices including network bonding, logical Oracle VM networks and network channels, and VLAN segments and groups.
Note: This article does not specifically step through planning and implementing Oracle VM networks. A future article, dedicated to designing a complex Oracle VM network based on the concepts covered in this article, is planned.
Oracle VM Server for x86 supports a wide range of options in network design, varying in complexity from a single network to configurations that include network bonds, VLANS, bridges, and multiple networks connecting the Oracle VM servers and guests. You can create separate networks to isolate traffic, or you can configure a single network for multiple roles. Network design depends on many factors, including the number and type of network interfaces, reliability and performance goals, the number of Oracle VM servers and guests, and the anticipated workload.
The Oracle VM Manager GUI presents four different ways to create an Oracle VM network (see Figure 1):
Figure 1. Four different ways of creating Oracle VM networks.
This article focuses the second option, designing a complex Oracle VM network infrastructure using only VLANs, and it steps through the concepts needed to create a robust network infrastructure for your Oracle VM servers and guests. Using VLANs allows you to create networking schemes that go far beyond the number of physical network devices installed in your server.
Note: Many networking internals are exposed throughout this article; this information is used simply to help bring clarity to concepts—using any information about internals for the purposes of bypassing Oracle VM Manager or performing work on the command line is not supported or recommended under any circumstance. Using the command line to "save time" will only result in costing more time and will seriously compromise the integrity and reliability of your Oracle VM implementation.
Oracle VM Server incorporates an open source Xen hypervisor component, which has been customized and optimized to integrate into the larger, Oracle-developed virtualization server. For this article, you need a basic understanding of Xen domains, since the terms Dom0 and DomU are used while explaining the relationships among various network components. The following explanation is greatly oversimplified, but sufficient for this article.
There are only two types of domains that are supported by the Xen hypervisor:
eth1, port bonds, VLAN ports, Xen bridges, and netback devices that shuttle network traffic between the Xen bridges on Dom0 and DomU.
Although neither of the above descriptions is technically complete, this explanation will suffice for understanding the networking concepts in this article.
The first step in planning an Oracle VM network infrastructure is to determine the physical network environment:
show int statuscommand is very useful for gathering this information from each switch.
eth1, and so on). You should also know exactly which Ethernet switch and port each server port is connected to—this will save countless, frustrating hours of troubleshooting if something is not working correctly.
Figure 2 shows a screenshot depicting the Oracle VM network objects used as examples throughout this article.
Figure 2. Oracle VM network objects as seen from Oracle VM Manager.
Each line in the screenshot shows a single Oracle VM network that represents two things:
pool3). That is, we have a total of thirty subnets that we are allocating across three server pools to perform various functions for us.
Take a moment to study the descriptions for each network shown in Figure 2 to get an understanding of how each Oracle VM network is used. Notice on the fourth line that the name of the network object helps describe how it is used. The name is simply a text field containing anything you like. Oracle recommends assigning a name to help make the usage clear. For example, "Guest DataCtr1 POOL1 PUB bond1" indicates that this network is used as the primary public interface for accessing any Oracle VM guests in
pool1, and it will be found assigned to bond1 on all servers in the pool.
Continuing along the same row in Figure 2, notice that the assigned virtual machine channel indicates IP addresses can be assigned only on the VM guests (DomU), not on Dom0. Further along on the same row, the description text field is used to tell us that this network should be assigned as eth0 on each guest, and it is associated with VLAN ID 200 on
bond1. Again, both text fields (name and description) can contain anything you want or nothing at all; but the more information you provide, the easier maintenance and troubleshooting will be.
You create Oracle VM network objects by assigning a single VLAN segment from a VLAN group. At this point, you give the network an easy-to-understand name and perhaps a description to help explain further what role the network fulfills. The VLAN segment defines which subnet and bond the network traffic will flow over.
In our example, an Oracle VM network object is a construct using a combination of other network objects as attributes:
eth1. The Ethernet ports can be used to create bonded interfaces or you can leave them unbonded—the choice is up to you.
Bond0is created during installation, and only one Ethernet port is added to bond0 initially.
Bond0can remain with a single port, or you can add up to three additional ports. For our example, we will aggregate six Ethernet ports into three bonded interfaces named
bond2. Bonds are explained in greater detail in the later section "Understanding Bonding."
Figure 3 shows a screenshot of network bond ports in an Oracle VM configuration. Bonding is simply aggregating or associating more than one Ethernet port into a single logical network interface. By aggregating ports into a single interface, you can achieve high availability by eliminating single points of failure for a given subnet: network traffic will still flow across all remaining Ethernet ports if a single port fails in a bonded interface. The effectiveness of your high availability solution relies heavily upon your data center infrastructure team ensuring the Ethernet ports on the Oracle VM servers are physically connected to redundant switch infrastructures.
Figure 3. Bonding as seen from Oracle VM Manager.
Oracle VM 3 provides three different types of bonding modes that can be used to aggregate multiple network devices (ports) into a single bond. Up to four ports are supported per bonded interface in Oracle VM 3.2; Oracle VM 3.1 and previous releases support a maximum of two ports per bond. All network devices/ports that are part of a bond must be the same type and speed.
All of the following modes are highly available: in the event of a network port failure, network traffic will continue to flow across all remaining devices/ports. Two of the three bonding modes also provide aggregated bandwidth, effectively increasing the total bandwidth of a bonded interface by the number of Ethernet ports that are part of a bond.
eth0) is used for network traffic while a second port (such as
eth1) takes over only if the first port fails. Active/backup mode supports VLAN segments with VM traffic.
Figure 4 shows the network device that is created on each Oracle VM server when you create a bond in Oracle VM Manager. Using Oracle VM nomenclature, bonds aggregate one to four Ethernet ports into a bond port.
Figure 4. Bonding as seen from a logical perspective on an Oracle VM server.
Notice that Figure 5 shows
bond0 using active/backup mode. It is important to also note that we do not assign any virtual machine channel roles to
bond0 at any point. Do not use adaptive load balancing if there is any chance you want to assign the virtual machine channel for Oracle VM guest traffic on
bond0 at some point in the future. If you are going to use
bond0 for Oracle VM guest traffic, then use either active/backup or dynamic link aggregation as the bonding mode for
Figure 5. Bonding as seen from a logical perspective.
Configuring LACP on the network infrastructure is normally complex and time consuming for network administrators—make sure you give your network administrators plenty of lead time to configure and validate LACP. If you do not have the ability to work with your network administration team, consider using active/backup or adaptive load balancing for your bonding mode, keeping in mind the limitation that adaptive load balancing has with VLANs and Oracle VM guest bridges.
Figure 6 shows a screenshot of VLAN segments in an Oracle VM configuration.
Figure 6. VLAN segments as seen from Oracle VM Manager.
Oracle VM uses existing VLAN segments; it does not create VLAN segments. The VLAN segments (tags) must be configured on the physical Ethernet switches before they can be used as part of the Oracle VM network infrastructure. Your network administrator will tell you which VLAN IDs are assigned to which ports on a switch. You then need to ensure the network ports on your servers are connected to the correct switch ports that are configured with the VLAN IDs you expect to access across any given network port.
Prior to the VLAN specification, a single network port was able to shuttle traffic across a single subnet. So, if you had four non-bonded network ports on a server, you could have access to a total of four different subnets. The purpose of the VLAN specification is to allow a single network port to access multiple subnets across a single wire connected to a single switch port. Therefore, by using multiple VLANs assigned to a single switch port, you can access multiple subnets on a single network port. This means the same server with four physical network ports can access more than four subnets.
Any number of VLAN segments or trunks can be associated with a single port or bond. However, there is a practical limit to the amount of network traffic based on the potential throughput of the port or bond. For example, if you assign four VLANs to a single 1-gigabit Ethernet (GbE) port and the anticipated network load on each VLAN is 800 megabits, you are most likely going to have a network performance problem because the port might be oversubscribed: you could potentially have 3.2 gigabits (3200 megabits) of traffic on a 1-GbE port.
Each VLAN ID represents a separate subnet (or broadcast domain) with its own IP range, subnet mask, and gateway. A network device is created on each Oracle VM server when a VLAN group containing one or more VLAN segments is assigned to a network port, such as
bond1 shown in Figure 7.
Figure 7. Bonding as seen from a logical device perspective on an Oracle VM server.
VLAN groups are simply containers for any number of VLAN segments that you want associated with a particular port or bond on each server. At the time of this writing, there is a one-to-one relationship between any given bond or port and a VLAN group: once you have assigned a VLAN group to a port or bond, the port or bond cannot be used for anything else.
VLAN groups are a required component when creating Oracle VM networks using VLANs, whether you are implementing a simple or complex network infrastructure using VLANs with ports or bonds. VLAN groups are the key to success for designing complex networks when a variety of server pools are being controlled by a single Oracle VM Manager. The planning of VLAN groups is where you should spend a majority of your time, since you will not be able to create and deploy complex Oracle VM networks if you fail to plan VLAN groups correctly—so spending the time to really think this through can save you days or weeks of frustration down the road.
In Oracle VM, VLAN groups allow you to compartmentalize small and large numbers of subnets into logical groupings that make sense for your scenario. For example, you might have 300 VLAN segments that are used for various reasons, but the guests in server
pool1 need only 6 of the 300 subnets, while guests in server
pool2 need 22 of the 300 subnets. VLAN groups allow you to customize which VLANs the Oracle VM servers in each server pool will have access to, without cluttering up every server with hundreds of extraneous VLANs they might never use.
Although only an example of one way to design VLAN groups, Figure 8 illustrates that you can create VLAN groups specific to particular server pools in different data centers. Just because the servers in each server pool have a
bond2 doesn't mean you are limited to only a single VLAN group for each bond—each server pool can have a completely different set of VLAN segments for each bond, depending on what the servers in each pool need. (Please don't let the above example limit your imagination—there are many ways to approach the problem, so design a scheme that works for the requirements in your data center.)
Figure 8. VLAN groups as seen from Oracle VM Manager.
Create VLANs groups before you attempt to create Oracle VM networks. Although there are several different opportunities to add IP addresses through a variety of network wizards, you should get in the habit of assigning all IP addresses that are relevant to Dom0 at the time you create each VLAN group, not when you create the Oracle VM networks.
Figure 9 briefly shows how VLAN groups and segments are used to create each Oracle VM network. In this example, we are creating an Oracle VM network named "OVM DataCtr1 POOL1 bond2.902" that will be assigned a network role of Live Migration and will be dedicated to live migration of Oracle VM guests between Oracle VM Servers.
Figure 9. VLAN groups and segments are used to create Oracle VM networks.
Figure 10 illustrates the logical association of the VLAN segments in each VLAN group to each bond from the perspective of an Oracle VM server in
pool1. It is important to note that VLAN segments in each VLAN group are available only to the bond that is assigned to a particular VLAN group.
Figure 10. VLAN groups and segments as seen from a logical perspective.
Let's explore what network channels do in the context of Oracle VM network objects. When you create an Oracle VM network using the Oracle VM Manager GUI, you assign one or more of the following channels: server management, cluster heartbeat, live migrate, storage, or virtual machine (see Figure 11).
Figure 11. Network channels as seen from Oracle VM Manager.
Selecting various channels triggers different behaviors from both Oracle VM Manager and the Oracle VM servers. The first three channels (server management, cluster heartbeat, and live migrate) pertain only to server pool management and configure various features on Dom0; these channels have absolutely nothing to do with Oracle VM guests (DomU). The fourth channel (storage) shows storage that currently provides no functionality in the product. The fifth channel (virtual machine) is strictly related to Oracle VM guests and has nothing to do with the Oracle VM servers (Dom0).
/etc/ocfs2/cluster.conf) file with the IP address and host name of each server in the server pool.
A natural inclination for high availability is to assign this role to multiple Oracle VM networks in case one fails, with the assumption that the heartbeat will continue running on any remaining networks within the same server pool. However, this is incorrect since OCFS2 supports only a single IP address/host name for each server in the
cluster.conf file. So, assigning this role to multiple networks within the same server pool doesn't actually accomplish anything; each server pool can have only one network with the cluster heartbeat role.
The network assigned the role for cluster heartbeat should be separate from networks with high load, such as live-migrate or Oracle Real Application Clusters (Oracle RAC) private traffic. Each server pool can share a common network for heartbeat, but this is not scalable for Oracle VM environments with many server pools containing large numbers of Oracle VM servers. Although not a requirement, an Oracle-recommended best practice is to have a completely separate VLAN/subnet dedicated to cluster heartbeat for each server pool. Of course, this might be a moot point if all you have is a single bond0 using the same physical interfaces as all your other servers.
xm migrate -l <DomU> <IP address of destination server>. This network should be separate from networks with high load, such as cluster heartbeat.
Selecting the virtual machine channel in Oracle VM Manager instructs the agent on each Oracle VM Server to create a Xen bridge that is associated with the port, bond, or VLAN segment. A bridge acts as a network switch on each Oracle VM server and shuttles network traffic between DomU and the rest of the world. A bridge will be created for any Oracle VM network you create that has the virtual machine channel selected; each bridge has a unique randomly generated network ID that can be seen in the Networking tab in Oracle VM Manager. Figure 12 shows the relationship between the VLAN interface (
bond1.200) and the netback driver for Oracle VM guests.
Figure 12. The virtual machine channel creates a Xen bridge that acts like a network switch for the Oracle VM guests on a server.
The server management, cluster heartbeat, and live migration network roles are required for each Oracle VM server. The storage network role is not fully implemented at the time of this writing. Each Oracle VM environment will also have one or more virtual machine networks.
Figure 13 shows the network channels used by Oracle VM and illustrates the varying scope for each channel type. The point of the illustration is simply to show which Oracle VM components need to communicate with each other and with the outside world. The management traffic for Oracle VM (server management, cluster heartbeat, and live-migrate) can be restricted to small, non-routed subnets dedicated to each function. For example, if your server pool is contained in a single blade chassis, the cluster heartbeat and live migrate channels can be connected to a Ethernet switch that is configured only for intra-chassis communication—there is no reason to connect all the way to a switch or router outside of the chassis.
Figure 13. The scope of various Oracle VM network channels.
The server management channel includes communication between Oracle VM Manager and the Oracle VM servers in the server pool, but not the guests. Both the cluster heartbeat and live migrate channel carry traffic among only the Oracle VM servers, but not Oracle VM Manager. For these channels, traffic can be on private, non-routed subnets—the traffic doesn't require access to routers. The traffic requires access to only the switches that connect the servers and manager, because communication flows only between the servers in the pool.
In contrast, a virtual machine channel includes traffic between Oracle VM guests as well as communication with external public networks. When you use the command-line interface to explore the networks configured by Oracle VM Manager, commands executed on the Oracle VM server will show network devices for all channel types. In contrast, commands executed on an Oracle VM guest will show only network devices for networks that are assigned the virtual machine channel.
Network channels are assigned when you create an Oracle VM network or you are changing network infrastructure at any point in the future.
Oracle VM Server for x86 supports a wide range of options for network design. Networks can vary in complexity from the simplest single-network configuration to configurations that include network bonds, VLANS, bridges, and multiple networks connecting the Oracle VM servers and guests. Network design depends on many factors, including the number and type of network interfaces, reliability and performance goals, the number of Oracle VM servers and guests, and the anticipated workload.
Understanding the basic concepts of Oracle VM networking is essential for designing a complex Oracle VM network infrastructure with a high degree of channel separation for a more scalable architecture.
The following resources are available for Oracle VM Server for x86:
Gregory King joined the Oracle VM product management team as a senior best practices technical consultant specializing in Oracle VM Server. Greg has a significant amount of hands-on experience with high availability, networking, storage, and storage networking, including SAN boot technology and server automation in production data centers, since 1993. His primary responsibility with the team is to develop and document solutions, including best practices, as well as foster a deeper understanding of the inner workings of Oracle VM with the goal of helping people become more successful with the product.
Don Smerker has been in the IT industry for more than twenty years, joining Oracle during the acquisition of BEA Systems, where he was part of the network operations team designing and managing all aspects of the network infrastructure. After joining Oracle, Don was part of Oracle's GIT networking organization responsible for day-to-day management of the network infrastructure for office spaces and off-production data centers. Don currently works with Oracle's GSS Network Operations team on key initiatives restructuring, reorganizing, and consolidating network infrastructure with some continued day-to-day management aspects of network operations.
|Revision 1.0, 06/07/2013|