Building Hyper-V Hosts: Converged Networking

Previously we looked at the networking basics for building Hyper-V hosts, specifically hosts in a failover cluster.  We saw that having dedicated high speed 10 GbE adapters for some or all networks could be costly and inefficient. In this article we will tackle the concept of converged networking.

With Hyper-V on Windows Server 2012 and newer you can collapse your networks into fewer high speed NIC’s using converged networking.  Traditionally you would have a team of 1 GbE NIC’s for each network (management, cluster, VM, etc.) requiring 6-8 physical 1 GbE NIC’s on the server. But with converged networking you are able to combine all the networks to go across a pair (or more) of 10 GbE NIC’s while ensuring each network gets their fair share of bandwidth using QoS. Converged networking is implemented through PowerShell only, and we will walk through an example script that you can use to get your converged networking up and running.

But first let’s cover the basic concepts of building converged networks.

We take two or more high speed adapters (10 GbE or faster) and put them into a team. Next we create and attach a Hyper-V virtual switch to this team. This switch is then configured to use weighted bandwidth QoS mode (more on that later). Finally we create virtual network adapters that the host OS uses for the networks you need (i.e. Management, cluster, live migration, virtual machines, etc.).

So what is weighted bandwidth QoS?

Weighted bandwidth mode allows you to share the total bandwidth of the underlying physical NIC’s of a Hyper-V virtual switch between the networks you create. The total weight of all the networks plus the default (VM traffic) must equal 100. The QoS is only enforced when the bandwidth is in contention.  This means that a live migration could use nearly all the bandwidth of a 10 GbE physical adapter, and then bandwidth is available for the other networks to use when no live migrations are occurring.

Using the below table as an example, we will guarantee a percentage of bandwidth for each network. Adjust the weights and number of networks to match your environment.

Networks Weight
Management 15
Cluster 5
Live Migration 20
iSCSI A 20
iSCSI B 20
Default(VM) 20
Total 100

The Default(VM) “network”  is set to 20 (referenced as DefaultFlowMinimumBandwidthWeight in PowerShell). This is the reserved bandwidth for anything that doesn’t match the other networks specified. Since we are specifying bandwidth for management, cluster, live migration, and two iSCSI networks, what is “left over” is the virtual machine traffic. This means that 20% of the bandwidth will be reserved for virtual machine traffic.

Now that we have our networks and weights mapped out, it’s time to get into the PowerShell script.

First Things

You should copy this script to the Hyper-V host on which you wish to configure converged networks and modify the parameters to meet the names, IP addresses, VLAN’s, DNS servers, etc. that meet your environment. Note that running this script WILL disrupt your network access to the server, so you will need to run this script locally on the server from a OOB management (DRAC, iLO, CIMC).

Hyper-V Converged Networking PowerShell Script

Here’s a basic overview of what the script does.  Based upon the parameters you specify, the script will build a network team from two pre-defined NIC’s to which a Hyper-V virtual switch will be connected in weighted bandwidth mode. After which the individual virtual network interfaces are created with their appropriate VLAN, IP, subnet mask, weight, et cetera.

To use the script we need to edit the parameters. Although you can specify most of the parameters when you run the script, you may find it simpler to open the script in the ISE and modify the IP address information for each of the networks to match your environment .

If you are not using iSCSI Networks:

    Adjust the weight parameter of each network to what you need
    Comment out (or don’t run) the last two network sections where the iSCSI interfaces are created.

The script assumes you have two 10 GbE network adapters named 10GbE1 and 10 GbE2.  You can adjust this based upon the number and names of your network adapters. Find the line that begins New-NetLBFOTeam.

The script also enables Jumbo frames on all virtual network adapters that are created, but you will need to enable jumbo frames manually on the underlying physical network adapters. (Due to the various names of registry properties for jumbo frames for the multitude of NIC manufactures it is difficult to add that to the template script.) Comment out these lines to skip the jumbo frame configuration.

Once you have made all of the necessary parameter changes and given everything a thorough second look, you’re ready to run the script.

STOP!

Make sure the script is stored locally on the server, AND you’re connected via OOB management just in case something goes wrong. OK. Take a deep breath and run the script. Within a minute or two the script should finish and now you can validate your network connectivity. If something went wrong review the script output.

Troubleshooting Steps

  • Make sure the virtual network adapters have been created and have the correct IP addresses.
  • Verify the network adapters have the correct VLAN ID associated.
  • Verify the network team is healthy (lbfoadmin.exe)
  • Start over again by removing the virtual network adapters (Remove-VMNetworkAdapter), removing the virtual switch, and removing the team.

Once you’re satisfied that your first Hyper-V host is configured and working with converged networking, repeat this process for every node in your Hyper-V cluster, changing the IP addresses and such where appropriate. After each node has been configured with converged networking you should run the cluster validation wizard to verify network connectivity between each node.

TIP

Document your cluster networking and save your converged networking scripts should you need to rebuild or adjust your cluster networks.

Now that you have accomplished converged networking, your next step should be to configure VMQ to ensure the high IO networking bandwidth is spread out properly amongst the CPU cores on your hosts. We’ll save that for another blog post.

Building Hyper-V Hosts: Networking Basics

In this second article on building Hyper-v Hosts series we will be focusing on network design for Hyper-V servers, specifically for failover clusters.

Before determining the number of required NIC’s, we need to talk about how many networks we plan to have. The number of networks you need depend on your existing network infrastructure as well as your existing SAN technology. Hyper-V standalone hosts require less networks since they do not have to deal with cluster traffic or dedicated live migration. However a Hyper-V failover cluster will require a greater number of networks.

Here’s a list of possible networks you may wish to consider:

  • Management
  • Cluster communication
  • Live Migration (vMotion)
  • Virtual Machine Network(s)
  • Storage Networking Fabric A
  • Storage Networking Fabric B
  • Backup

If you plan to use iSCSI or SMB SAN instead of fibre channel (FC), then you’ll need to account for those networks.  Many iSCSI SAN vendors require separate VLAN’s and hardware for each storage fabric path.  Some environments may require you to separate backup traffic from the management traffic. You may also choose to share networks for cluster communication and live migration depending on your needs.

Consider which network(s) should be used for cluster communication. Best practice is to ensure that cluster communication should be excluded from storage networks (i.e. iSCSI).

Hyper-V Networks 001

Specify a cluster network for cluster shared volume traffic. Every cluster shared volume has an owner node in your failover cluster. While each node can access the volumes simultaneously, metadata changes must be sent over the network to the owner nodes (add/remove files, expand VHD, etc.). The cluster network with the lowest metric is used for cluster shared volume traffic.

To configure a specific network to be used, set its metric to be the lowest.

Once you have determined the number of networks you need then you can determine the number and type of network interfaces you require.

A traditional simple design is to have a pair of 1 GbE NIC’s for each network.

  • 1 Team for Management
  • 1 Team for Cluster and Live Migration
  • 1 Team for Virtual Machine traffic
  • 2 separate NIC’s for SMB/iSCSI storage  (if applicable)

This design is great for a smaller cluster since it doesn’t require 10 GbE infrastructure. Unfortunately it will require a server with eight 1 GbE interfaces as well as eight switch ports, multiplied by the number of hosts needed in your cluster. You could see how this model could exhaust your switch ports pretty quickly. In addition you’ll find that it will take a long time to drain hosts of large VM’s when you can only transfer at 1 Gbps.

You could simply upgrade this model by changing your storage NIC’s to 10 GbE or all of your NIC’s to 10 GbE if you can. This model still requires a large number of NIC’s and switch ports which are even more costly at 10 GbE. On top of that you could have expensive 10 GbE ports tied up for resources that are only used sparingly (i.e. cluster, live migration).

If only there was a way to take advantage of these high speed network interfaces and share the high bandwidth between the networks giving each network traffic type their own priority. Good news! Converged networking allows us to do this, and we’ll discuss the implementation in the next post.

As always when building and validating Hyper-V clusters, it is helpful to step through the best practices checklist for each host.

Building Hyper-V Hosts: Hardware and OS

mshv-overview

Hyper-V, Microsoft’s hypervisor platform, has really grown up in the last few years. Introduced with Windows Server 2008, Microsoft has iterated its hypervisor platform reaching version “3.0” with Windows Server 2012. Version 3.0 has often been considered the magic version number for Microsoft as to when the product is not only good enough, but also a contender.  Then they further fine-tuned the product with Windows Server 2012 R2 making it a worthy VMware competitor. Hyper-V does not match feature for feature with VMware, but on most of the features that matter the hypervisors are equals.

To implement Hyper-V host clusters in your environment requires many elements to be integrated properly. Thankfully Microsoft PFE’s and the community maintain a Best Practices Checklist on TechNet that you can use to make your Hyper-V deployment a success.  I recommend you read through it carefully a number of times during your design phase of your Hyper-V deployment. Missing a crucial component could make the difference between success and failure of the project. I’m going to highlight a few key points from the checklist.

Since a hypervisor is actually going to be running on bare metal, we must select hardware that will provide the power, features, and stability that we need. We need to begin by selecting the hardware from a vendor that is listed as being officially supported for running Hyper-V. Start with the Windows Server Catalog to see if your preferred vendor hardware and components are certified. ws2012r2_logo_72x90Verify the CPU model meets the virtualization requirements. Windows Server 2012 R2 is licensed by socket, but with Windows Server 2016 Microsoft will switch to a per-core licensing model similar to SQL. Until Windows Server 2016 is released (and your EA renews) the strategy is to buy as many cores per socket as you can afford to maximize your licensing. The similar strategy should be used for memory: buy as much memory as you can afford. Typically the first resource to be used up is memory. Use the Datacenter edition if you plan to implement many more than two VM’s as it gives you unlimited guest Windows server licenses. The standard edition includes licenses for two VM’s.

Strongly consider using 10 GbE or higher network interface cards, if you can afford it, to allow for fast live migration of large VM’s, and especially if you plan to use SMB or iSCSI SAN. Also consider buying NIC’s that support RDMA (RoCE) for SMB direct for higher performance, especially if implementing SoFS SAN.  Select NIC’s from manufacturers who are known to have reliable drivers. Bad drivers will destroy your Hyper-V environment. We will dive deeper into networking of Hyper-V hosts and guests in a later post.

When installing the OS, aim to use the server core interface to minimize patches and vulnerabilities. Manage the server from an administrative system with the RSAT tools and PowerShell. Ensure rollups and applicable hotfixes are applied to your hosts. And just to state the obvious, keep your Hyper-V clustered hosts at the same patch level. The parent partition or management OS should have the minimal roles and features necessary including Hyper-V, Failover Clustering, MPIO, and SNMP where applicable. The only other installed items should be manufacturer’s drivers and utilities along with backup, operations, and antivirus agents. Be extremely careful about installing antivirus on your Hyper-V hosts. The antivirus exceptions must be setup correctly or you may at best degrade performance and at worst corrupt your virtual machines.  One list tip about the OS of your Hyper-V hosts: protect your hosts by limiting who can log on to them. Using server core will keep the away the server admins “who know enough to be dangerous.”

In later blog posts we will cover network configuration for the host and for guests, including converged networking with iSCSI.

Thanks to Aidan Finn (@joe_elway) and other authors of the Windows Server 2012 Hyper-V: Installation and Configuration Guide.

Read Aidan Finn’s Best Practices for Hyper-V.

More Lessons Learned from Hyper-V Replica

In my previous post on lessons learned from Hyper-V Replica, I introduced the features of Hyper-V Replica introduced in Windows Server 2012 and improved in Windows Server 2012 R2. We saw that while Hyper-V Replica is a great feature, like all other technologies, it requires a workflow along with monitoring and troubleshooting procedures. In this post, I will discuss the workflow of implementing replication on an individual VM.

If there’s one thing that we can all agree on, it is that we should organize our VM configuration and virtual hard disk files in an orderly way (OCD FTW). If there’s one thing we can’t all agree on is how do that. The default way which Microsoft organizes Hyper-V VM’s is by placing the configuration file within one folder and placing the virtual hard disks within a separate folder.

hypervreplica001

hypervreplica002

Having the configuration files and VHD(X) files in separate locations doesn’t seem very tidy to me. I have opted to create a single folder to hold both configuration files and VHD files, naming the folder with the name of the VM. I then named the associated VHD files with the server name and drive letter as appropriate.

hypervreplica003hypervreplica004This organization makes it easier to locate all the files associated with a VM for administration, migration, backups, and other duties to which you might be called.

Tip: If you’re creating a VM from the GUI check the “Store the virtual machine in a different local” box and specify the folder, then all your configuration and VHD files will be stored in that one folder with the folder named as the VM name.

hypervreplica005

We spent all this time organizing our VM files into orderly folders, so the assumption would be that when you enable replication for the VM your orderliness will be mirrored to the replica host. Well you know what assuming does…Here’s what really happens.

When establishing initial replication the replica server creates separate folders for configuration and VHD files using GUID‘s as the folder names.

hypervreplica006

I’m sure there was good reasoning behind this decision but for those of us with CDO (that’s OCD in alphabetical order) we like to keep things organized.  Fortunately a simple storage migration on the replica server will get us back to OCD nirvana ensuring everything is in its place.

When first establishing replication for a VM allow the Hyper-V server to create the replica folders where it wants, but do not initiate replication immediately. Instead set the replication for a future time. hypervreplica007Then create a folder with the name of your choosing and perform a storage migration of the placeholder files. hypervreplica008Then initiate the replication. hypervreplica009hypervreplica010Now your replica VM files will be nice and tidy.

Now that we now have an understanding of the Hyper-V files and folder structure for both primary and replica VM’s, we can implement processes and procedures for deploying Hyper-V VM replicas. With this knowledge we can develop PowerShell scripts to automate these processes to free up your engineers and administrators for other duties.

Lessons Learned from Hyper-V Replica

Hyper-V Replica
Hyper-V Replica

Windows Server 2012 introduced a boatload of new features, but my favorite feature has to be Hyper-V 3.0. Microsoft dramatically revamped its hyper visor since Windows Server 2008 R2 bringing all sorts of reliability, performance, and scalability to the product. Finally it was a hypervisor of which I could be proud. Hyper-V Replica was a feature introduced with Hyper-V 3.0 giving us the ability to replicate selectable VHD(X)’s of VM’s to other Hyper-V hosts or clusters. Hyper-V Replica could now become the basis of a disaster recovery solution adequate for most small to medium size businesses. Windows Server 2012 R2 brought additional replication frequency schedules as well as extended replica allowing you to replicate the VM to a third location.

One thing I found with Hyper-V Replica is that you do have to monitor the replication health of your VMs. Sometimes busy patching windows, extended maintenance, or even certain backup applications can cause the replication to become unhealthy. In some cases simply resuming the replication resolves the issue while other VMs require a resynchronization. (Resynchronization requires additional CPU and network overhead which may be best to be run after business hours.) In some cases we had to move the VM to another node on the cluster to get the replication to resume.

To monitor the Hyper-V Replica I wrote a PowerShell tool to check and monitor the health of VM replication.

Check-HyperVReplica is a PowerShell script designed to work as a Nagios plugin that will get the replication health of all the VM’s on a Hyper-V host and output the status to be consumed by a Nagios server. However the script can be used in a standalone way as shown in the example below. Instructions for using the script along with how to incorporate it into your Nagios/NSClient++ environment are provided in the comment-based help.

Example:

What lessons have you learned from deploying Hyper-V Replica?

Detailed Cluster Shared Volume Information

Cluster Shared Volume DiagramIf you’re running Hyper-V clusters and/or Scale-out File Servers (SoFS) you may have the need to get a quick report of the Cluster Shared Volumes (CSV’s). However with the built-in cmdlet Get-ClusterSharedVolume you only get basic information. I wrote a function called Get-ClusterSharedVolumeUsage to gather additional useful information including Path, Size, FreeSpace, UsedSpace, and PercentFree.

Example:

Get-ClusterSharedVolumeUsage

Hyper-V and Mounted DVD Drives

thinkcentre_dvd_rom_drive_41n5618

From time to time there is a need to mount an ISO to a Hyper-V virtual machine to install an operating system or SQL or other software. Unfortunately not everyone remembers to dismount the ISO from Hyper-V after they completed their task. This can cause errors when live migrating the VM to another host if the ISO is not located in a cluster shared volume. Either way it’s a good habit to keep those DVD drives unmounted if they are not in use. I wrote a script to connect to System Center Virtual Machine Manager to find any VM’s with mounted DVD drives and created a report. The script can also optionally send the report to the address of your choosing. I set it up to run once a week to nag remind our team to clean up after ourselves.

Get-SCVirtualMachinewithMountedDVDDrive