Whats new in VMware vSphere ESXi 5

Following up on today’s announcement from VMware about vSphere 5 the next step towards building the cloud on VMware. I have gathered up some of the more important new and updated features on vSphere 5. There will be other posts written over the next weeks and months diving deeper into these features. But for now this will wet your appetite.

  • Convergence. vSphere 5.0 is the first vSphere release built exclusively on the vSphere ESXi 5.0 hypervisor architecture as the host platform. VMware will not include ESX hypervisor architecture-based releases in this vSphere release or later releases. The vSphere 5.0 management platform, vCenter Server 5.0, provides support for ESXi 5.0 hosts as well as ESX/ESXi 4.x and ESX/ESXi 3.5 hosts.
  • VMware vSphere Auto Deploy. Combining the features of host profiles, Image Builder, and PXE, VMware vSphere Auto Deploy simplifies the task of managing ESXi installation and upgrade for hundreds of machines. New hosts are automatically provisioned based on rules defined by the user. Rebuilding a server to a clean slate is as simple as a reboot. To move between ESXi versions, you update a rule using the Auto Deploy PowerCLI and perform a test compliance and repair operation.
  • Unified CLI Framework. An expanded and enhanced esxcli framework offers a rich set of consistent and extensible commands, including new commands to facilitate on-host troubleshooting and maintenance. The framework allows consistency of authentication, roles, and auditing, using the same methods as other management frameworks such as vCenter Server and PowerCLI. You can use the esxcli framework both remotely as part of vSphere CLI and locally on the ESXi Shell (formerly Tech Support Mode).
  • New Virtual machine capabilities. ESXi 5.0 introduces a new generation of virtual hardware with virtual machine hardware version 8, which includes the following new features:

o    32-way virtual SMP. ESXi 5.0 supports virtual machines with up to 32 virtual CPUs, which lets you run larger CPU-intensive workloads on the VMware ESXi platform.

o    1TB virtual machine RAM. You can assign up to 1TB of RAM to ESXi 5.0 virtual machines.

o    Nonhardware accelerated 3D graphics for Windows Aero support. ESXi 5.0 supports 3D graphics to run Windows Aero and Basic 3D applications in virtual machines.

o    USB 3.0 device support. ESXi 5.0 features support for USB 3.0 devices in virtual machines with Linux guest operating systems. USB 3.0 devices attached to the client computer running the vSphere Web Client or the vSphere Client can be connected to a virtual machine and accessed within it. USB 3.0 devices connected to the ESXi host are not supported at this time.

o    UEFI virtual BIOS. Virtual machines running on ESXi 5.0 can boot from and use the Unified Extended Firmware Interface (UEFI).

  • Graphical User Interface to configure multicore virtual CPUs. You can now configure the number of virtual CPU cores per socket in the Virtual Machine Properties view in the vSphere Web Client and the vSphere client. Previously this feature was only configurable through advanced settings.
  • Client-connected USB devices. USB devices attached to the client computer running the vSphere Web Client or the vSphere Client can be connected to a virtual machine and accessed within it.
  • Smart card reader support for virtual machines. Smart card readers attached to the client computer running the vSphere Web Client or the vSphere Client can be connected to one or more virtual machines and accessed within them. The virtual machine remote console, available in the vSphere Web Client and the vSphere Client, supports connecting smart card readers to multiple virtual machines, which can then be used for smart card authentication to virtual machines.
  • Expanded support for VMware Tools versions. VMware Tools from vSphere 4.x is supported in virtual machines running on vSphere 5.0 hosts. Additionally, the version of VMware Tools supplied with vSphere 5.0 is also compatible with ESX/ESXi 4.x.
  • Apple Mac OS X Server guest operating system support. VMware vSphere 5.0 adds support for the Apple Mac OS X Server 10.6 (“Snow Leopard”) as a guest operating system. Support is restricted to Apple Xserve model Xserve3,1 systems. For additional information, see the vSphere 5.0 RC Release notes.
  • Host UEFI boot support.vSphere 5.0 supports booting ESXi hosts from the Unified Extensible Firmware Interface (UEFI). With UEFI you can boot systems from hard drives, CD-ROM drives, or USB media. Booting over the network requires the legacy BIOS firmware and is not available with UEFI.
  • Support for up to 512 virtual machines. vSphere 5.0 supports up to 512 virtual machines totaling a maximum of 2048 virtual CPUs per host.
  • Support for larger systems. vSphere 5.0 supports systems with up to 160 logical CPUs and up to 2TB RAM.
  • Improved SNMP support. vSphere 5.0 adds the capability to convert CIM indications to SNMP traps. Check with your hardware vendor to see whether their CIM provider supports this functionality. In addition, vSphere 5.0 now supports the Host Resources MIB (RFC 2790) and allows for finer control over the types of traps sent by the SNMP agent.
  • Storage DRS. This feature delivers the DRS benefits of resource aggregation, automated initial placement, and bottleneck avoidance to storage. You can group and manage similar datastores as a single load-balanced storage resource called a datastore cluster. Storage DRS makes VMDK placement and migration recommendations to avoid I/O and space utilization bottlenecks on the datastores in the cluster.
  • Policy-driven storage delivery. This solution allows you to have greater control and insight into characteristics of your storage resources. It also enables virtual machine storage provisioning to become independent of specific storage available in the environment. You can define virtual machine placement rules in terms of storage characteristics and monitor a virtual machine’s storage placement based on these administrator-defined rules. The solution delivers these benefits by taking advantage of the following items:

Storage

o    Integrating with Storage APIs – Storage Awareness to deliver storage characterization supplied by storage vendors.

o    Enabling the vSphere administrator to tag storage based on customer-specific descriptions.

o    Using storage characterizations to create virtual machine placement rules in the form of storage profiles.

o    Providing easy means to check a virtual machine’s compliance against these rules.

As a result, managing storage usage and choice in vSphere deployments has become more efficient and user-friendly.

  • VMFS5. VMFS5 is a new version of vSphere Virtual Machine File System that offers improved scalability and performance.
  • Accelerator. An accelerator has been delivered for specific use with View (VDI) workloads. With this option configured in ESXi, a read cache is constructed in memory that is optimized for recognizing, handling, and deduplicating VDI client images. The cache is managed from within the View Composer and delivers a significant reduction, as high as 90% by early estimates, in IOPS from each ESXi host to the storage platform holding client images. This reduction in IOPS enables large scaling of the number of clients in case multiple I/O storms, typical in large VDI deployments, occur.
  • iSCSI UI support. Usability improvements in this release include the ability to configure dependent hardware iSCSI and software iSCSI adapters along with the network configurations and port binding in a single dialog box using the vSphere Client. Full SDK access is also available for these configurations.
  • Storage I/O Control NFS support. vSphere 5.0 extends Storage I/O Control (SIOC) to provide cluster-wide I/O shares and limits for NFS datastores.
  • Storage APIs – Array Integration: Thin Provisioning. Offers an ability to reclaim blocks of a thin provisioned LUN on the array when a virtual disk is deleted.
  • Swap to SSD. vSphere 5.0 provides new forms of SSD handling and optimization. The VMkernel automatically recognizes and tags SSD devices that are local to ESXi or are on the network. In addition, the VMkernel scheduler is modified to allow ESXi swap to extend to local or network SSD devices, which enables memory overcommitment and minimizes performance impact.
  • 2TB+ LUN support. vSphere 5.0 provides support for 2TB+ VMFS datastores.
  • Storage vMotion snapshot support. Allows Storage vMotion of a virtual machine in snapshot mode with associated snapshots. You can better manage storage capacity and performance by leveraging flexibility of migrating a virtual machine along with its snapshots to a different datastore.
  • Enhanced Network I/O Control. vSphere 5.0 builds on network I/O control to allow user-defined network resource pools, enabling multi-tenancy deployment, and to bridge virtual and physical infrastructure QoS with per resource pool 802.1 tagging.
  • vNetwork Distributed Switch Improvements. vSphere 5.0 provides improved visibility into virtual machine traffic through Netflow and enhances monitoring and troubleshooting capabilities through SPAN and LLDP.
  • ESXi Firewall. The ESXi 5.0 management interface is protected by a service-oriented and stateless firewall, which you can configure using the vSphere Client or at the command line with esxcli interfaces. A new firewall engine eliminates the use of iptables and rule sets define port rules for each service. For remote hosts, you can specify the IP addresses or range of IP addresses that are allowed to access each service.
  • Next-generation browser-based vSphere Client. A browser-based, fully-extensible, platform-independent implementation of the vSphere Client based on Adobe Flex. The vSphere 5.0 release includes both the new browser-based client and the Windows-based client available in prior releases. In this release, the browser-based client includes a subset of the functionality available in the Windows-based client, primarily related to inventory display and virtual machine deployment and configuration.
  • vCenter Server Appliance. A vCenter Server implementation running on a pre-configured Linux-based virtual appliance. This appliance significantly reduces the time required to deploy vCenter Server and associated services and provides a low-cost alternative to the traditional Windows-based vCenter Server.
  • Inventory Extensibility. VMware customers and partners can extend vCenter Server in multiple ways, including the inventory, graphical user interface, and agents. vCenter Server includes a manager to monitor the extensions. By deploying extensions created by VMware partners, you can use vCenter Server as a unified console to manage your virtualized datacenter.
  • Solution Installation and Management. The vCenter Solutions Manager provides a consistent interface to configure and monitor vCenter-integrated solutions developed by VMware and third parties. It provides a simpler installation, configuration, and monitoring interface for managing solutions. Using the new vSphere ESX Agent Manager, you can deploy, update, and monitor vSphere agents on ESXi hosts. vSphere agents inter-operate efficiently with other vSphere features such as maintenance mode and distributed power management.
  • Enhanced logging support. vSphere 5.0 adds several enhancements to system message logging. All log messages are now generated by syslog, and messages can now be logged on either local and/or one or more remote log servers. A given server can log messages from more than one host. Log messages can be remotely logged using either the Secure Sockets Layer (SSL) or TCP connections. The vSphere syslog listener is available as an optional plug-in to vCenter on Windows; in the vCenter Virtual Appliance (VCVA), logging is accomplished using the native syslog-ng facility. With vSphere 5.0, log messages from different sources can be configured to go into different logs for more convenience. Configuration of message logging can also be accomplished using ESXCLI in addition to the vSphere client.
  • Fault Domain Manager — VMware High Availability has been transformed into a cloud-optimized availability platform. With Fault Domain Manager, VMware HA is more reliable in operation, more scalable in its ability to protect virtual machines, and can provide better uptime than before. All hosts in the cluster can now be primary nodes while the cluster also uses shared storage as a channel for host heartbeat detection. This enables VMware HA to react accurately and efficiently to host failures, allowing customers to grow their vSphere cluster.

Networking

  • Enhanced Network I/O Control. vSphere 5.0 builds on network I/O control to allow user-defined network resource pools, enabling multi-tenancy deployment, and to bridge virtual and physical infrastructure QoS with per resource pool 802.1 tagging.
  • vNetwork Distributed Switch Improvements. vSphere 5.0 provides improved visibility into virtual machine traffic through Netflow and enhances monitoring and troubleshooting capabilities through SPAN and LLDP.
  • ESXi Firewall. The ESXi 5.0 management interface is protected by a service-oriented and stateless firewall, which you can configure using the vSphere Client or at the command line with esxcli interfaces. A new firewall engine eliminates the use of iptables and rule sets define port rules for each service. For remote hosts, you can specify the IP addresses or range of IP addresses that are allowed to access each service.

VMware vCenter Server

  • Next-generation browser-based vSphere Client. A browser-based, fully-extensible, platform-independent implementation of the vSphere Client based on Adobe Flex. The vSphere 5.0 release includes both the new browser-based client and the Windows-based client available in prior releases. In this release, the browser-based client includes a subset of the functionality available in the Windows-based client, primarily related to inventory display and virtual machine deployment and configuration.
  • vCenter Server Appliance. A vCenter Server implementation running on a pre-configured Linux-based virtual appliance. This appliance significantly reduces the time required to deploy vCenter Server and associated services and provides a low-cost alternative to the traditional Windows-based vCenter Server.
  • Inventory Extensibility. VMware customers and partners can extend vCenter Server in multiple ways, including the inventory, graphical user interface, and agents. vCenter Server includes a manager to monitor the extensions. By deploying extensions created by VMware partners, you can use vCenter Server as a unified console to manage your virtualized datacenter.
  • Solution Installation and Management. The vCenter Solutions Manager provides a consistent interface to configure and monitor vCenter-integrated solutions developed by VMware and third parties. It provides a simpler installation, configuration, and monitoring interface for managing solutions. Using the new vSphere ESX Agent Manager, you can deploy, update, and monitor vSphere agents on ESXi hosts. vSphere agents inter-operate efficiently with other vSphere features such as maintenance mode and distributed power management.
  • Enhanced logging support. vSphere 5.0 adds several enhancements to system message logging. All log messages are now generated by syslog, and messages can now be logged on either local and/or one or more remote log servers. A given server can log messages from more than one host. Log messages can be remotely logged using either the Secure Sockets Layer (SSL) or TCP connections. The vSphere syslog listener is available as an optional plug-in to vCenter on Windows; in the vCenter Virtual Appliance (VCVA), logging is accomplished using the native syslog-ng facility. With vSphere 5.0, log messages from different sources can be configured to go into different logs for more convenience. Configuration of message logging can also be accomplished using ESXCLI in addition to the vSphere client.

Availability

  • Fault Domain Manager — VMware High Availability has been transformed into a cloud-optimized availability platform. With Fault Domain Manager, VMware HA is more reliable in operation, more scalable in its ability to protect virtual machines, and can provide better uptime than before. All hosts in the cluster can now be primary nodes while the cluster also uses shared storage as a channel for host heartbeat detection. This enables VMware HA to react accurately and efficiently to host failures, allowing customers to grow their vSphere cluster.

 

About Brian Suhr

Brian is a VCDX5-DCV and a Sr. Tech Marketing Engineer at Nutanix and owner of this website. He is active in the VMware community and helps lead the Chicago VMUG group. Specializing in VDI and Cloud project designs. Awarded VMware vExpert status 6 years for 2016 - 2011. VCP3, VCP5, VCP5-Iaas, VCP-Cloud, VCAP-DTD, VCAP5-DCD, VCAP5-DCA, VCA-DT, VCP5-DT, Cisco UCS Design

Read More

Running vSphere 4.0 ESXi embedded Hypervisor on IBM x3690 servers

I’ve been working with a client lately on a datacenter move and they have selected IBM x3690 servers. The 3690’s will be the ESXi hosts for the new site and are running ESXi embedded. I have not had the opportunity to work with many different clients that choose the embedded route, so it was cool to see how IBM setup the servers.

The servers came with ESXi 4.0 installed on a USB stick from the factory and installed in one of the two internal USB ports that the server offers. Upon turning on the servers some of them booted right to VMware and some did not. After some further looking into the boot order in the BIOS I noticed that the Embedded Hypervisor option was not added to the boot order on a couple of the servers. A quick add and they were running just like the rest, guess someone at the factory missed that one.

The servers took a very long time to post and boot up, part of this was due to the 128 GB of RAM installed. We turned off some of the non-essentials and modified the boot order to go right to ESXi and cut the post time down some. You can see from the image below it’s just another x-series server.

I snapped the image below with the cover over showing off all the sticks of memory installed.

The last image below is a close up to the two USB ports that are internal to the server. The lower one as the USB stick from the factory with ESXi embedded on it.

About Brian Suhr

Brian is a VCDX5-DCV and a Sr. Tech Marketing Engineer at Nutanix and owner of this website. He is active in the VMware community and helps lead the Chicago VMUG group. Specializing in VDI and Cloud project designs. Awarded VMware vExpert status 6 years for 2016 - 2011. VCP3, VCP5, VCP5-Iaas, VCP-Cloud, VCAP-DTD, VCAP5-DCD, VCAP5-DCA, VCA-DT, VCP5-DT, Cisco UCS Design

Read More

Installing network card drivers in VMware ESXi after install with vihostupdate

This is not something that I’ve had to do very often. But  on a recent customer engagement I was working with the client on setting up some new hosts that were recently purchased. These hosts were purchased with Embedded ESXi on them and additional PCI NICs were added to the config. The additional NICs did not have drivers available in the base ESXi build. Shortly after bringing the first host online we noticed that only the onboard NICs showed up in the list.

A quick search on Google for the Intel part number for the NIC lead me to the family name for the adapter. Then a search over at VMware lead me to the download page for VMware that provided the .ISO file to load the drivers into ESXi for the family of adapters. The process took only a few minutes and since this is something that does not come up that often I thought a short write up might help someone.

There are a few ways that this could be done, since we happened to be running ESXi the options were to use the vMA or vCLI. Since this was a new install and a vMA was not setup yet I just quickly tossed vCLI on a server. Then a quick download of the driver .ISO from VMware and unzip the package into a folder on the server with vCLI installed on it. If you wanted to use the vMA you could mount the .ISO to the virtual CD-ROM of the VM and issue the command against it.

Since I was using vCLI all I needed to do was point the command to a local folder. Here is a sample of the command used to perform the patch.

vihostupdate –server HOSTNAME –install –bundle c:foldername_of_file.zip

To run this command your host must be in Maintenance mode and it will then take just a couple of minutes to execute. After the update completes a reboot of the host is needed and then the cards should be available for use.

About Brian Suhr

Brian is a VCDX5-DCV and a Sr. Tech Marketing Engineer at Nutanix and owner of this website. He is active in the VMware community and helps lead the Chicago VMUG group. Specializing in VDI and Cloud project designs. Awarded VMware vExpert status 6 years for 2016 - 2011. VCP3, VCP5, VCP5-Iaas, VCP-Cloud, VCAP-DTD, VCAP5-DCD, VCAP5-DCA, VCA-DT, VCP5-DT, Cisco UCS Design

Read More

ESXi management network issues when using EtherChannel and NIC teaming

ESXi behavior with NIC trunking

Sometimes very challenging problems will arise.  Things that make you scratch your head, want to hurl your coffee cup, or just have a nice cold adult beverage.  Customers can change a projects requirements mid-way through, a vendor’s storage array code upgrade can go awry, or a two can creep into the ones and zeros.

In this section, we present examples of those crazy situations with the hopes of helping out our fellow engineers in the field before they become as frustrated as we have!

Recently in working with a customer, the request was for a new cluster comprised of ESXi 4.1 hosts.  They would be using just two onboard NICs for the vmkernel and virtual machine traffic.  These two NICs would feed into a pair of Cisco Nexus 5020’s, using virtual port channel (VPC).

Because of the use of VPC, the virtual switch load balancing needs to be set to IP Hash for the NIC teaming policy.  Okay, no sweat!  After installing ESXi and completing the initial configuration on the hosts, it was time to add the second NIC to vSwitch0 and plug it in.  (Note this configuration is all being done on the hosts directly as no vCenter server has been built yet).  After adding the second adapter to the active section of vSwitch0, and changing the NIC teaming policy to IP hash, we plugged in the second cable.

The host immediately lost connection from our vSphere client, and dropped off entirely from being able to be contacted.  No ping, no nothing!  This was most puzzling indeed:  we unplugged the second cable and the host started to ping again.  We thought maybe there was something wrong with the NIC itself, and so setup a separate NIC to take its place.  This had the same result, and we then thought to look at the switch.  After discussing the current configuration with the network engineer, we felt that his configuration was correct.  The configuration (and more!) can be found in the white paper put out by Cisco and VMware: “Deploying 10 Gigabit Ethernet on VMware vSphere 4.0 with Cisco Nexus 1000V and VMware vNetwork Standard and Distributed Switches – Version 1.0” This doc has been a very helpful during the implementation of this project.

So!  With the network being deemed not the problem and wearing a sheepish smile on my face after the network guy commented “it’s always the network isn’t it?” I returned to the host.  I then tried setting up both NICs on a non-nexus switch that is being used for out of band management, and they worked just fine using virtual port id for NIC teaming.  So at that point, I fired up the googalizer and did some checking.  I came across this KB article from VMware:

VMware KB 1022751:  NIC teaming using EtherChannel leads to intermittent network connectivity in ESXi

Details:

When trying to team NICs using EtherChannel, the network connectivity is disrupted on an ESXi host. This issue occurs because NIC teaming properties do not propagate to the Management Network portgroup in ESXi.
When you configure the ESXi host for NIC teaming by setting the Load Balancing to Route based on IP hash, this configuration is not propagated to Management Network portgroup.

So, based on this very helpful information, I followed the instructions listed in the KB and had great success.  Now my ESXi hosts are talking on both NICs via IP Hash and life is good.

Read More

VMware vSphere designing with VAAI in mind

With the release of vSphere 4.1 last summer VMware customers were provided several new features. Many of these new features were created to lower the burden on the ESX host by being more efficient or offloading the work to something outside of the Virtualization stack. The overall goal of the new features was to continue to improve performance of virtual machines. The one that I am writing about today is VAAI or vStorage API for Array Integration. I wanted to write about how using VAAI in your Architecture Designs is changing the way you are creating environments.

The goal of VAAI is too offload some of the storage focused activities that VMware had previously handled to your storage array. This was accomplished by VMware working closely with the major storage vendors. The idea of VAAI was first announced back at VMworld 2008 and finally came to market when vSphere 4.1 was released. By offloading these storage functions it has reduced the load on the ESX(i) hosts and also increased the performance of these activities by letting the storage array do the work that it was created to do.

In the current offering of VAAI there are 3 functions that have been offloaded. In future product releases it is expected that VMware will continue to work with storage vendors to increase the features of VAAI and the currently available features are explained below.

Full Copy – So you’re probably wondering how this feature is going to help me. I can think to two VMware functions that this VAAI feature provides upwards of 10x speed improvements in. The first would be when you are deploying a VM from a template. We will say for example that you are going to deploy a 50 GB VB. When the VM is deployed vSphere is going to read the entire 50 GB and then write the 50 GB for a total of 100 GB of I/O. With VAAI enabled and a storage array that supports VAAI this process creates very little I/O at the host. The vSphere host will send a command to the storage array that say make a copy of this VM and name it this. The copy will be done locally on the storage array and results in very little I/O between the host and array. Once completed the array will send a notice to the host to let it know the works was completed.

The second VMware feature to benefit from this is a Storage vMotion. I feel that this is where this really pays off because you are most likely moving a larger chunk of data with this command. For example sake let’s say we are going to move a 100 GB Virtual Machine from one disk to another. To do this in the past this would have caused 200 GB of I/O on the host. With VAAI the burden on the host is almost nothing as this work is done on the storage array.

Hardware assisted locking – Too allow multiple hosts in your cluster to talk to the same storage volume VMware would lock the volume when one of the VM’s needed to write to it. This locking is to prevent another host from trying to write to the same blocks. This was not a large issue If you were using smaller volumes with only a handful of virtual machines on them.  Now with VAAI the locking has been offloaded to the storage array and it’s now possible to lock only the blocks that are being written to. This opens up the possibility to use larger volumes and increase the amount of VM’s that can be run on a single volume.

Block Zeroing – This feature is saving vSphere from having to send redundant commands to the array for writes. The host can simple tell the storage array which blocks are zeros and move on. The storage device will handle the work without needing to receive repetitive write commands from the host.

So now that you should have an understanding of what VAAI is and how it should help free up resources. I will now talk about how this changes the way we should be thinking about different design considerations.

The first thing that comes to mind is that I can now think about using larger datastores without the worry of affecting performance due to locking issues. With VAAI the storage device is going to handle the locking and allow me to have far for VM’s per volume than the 5-10 previously held as a guideline to live by in past versions. It’s now possible to have 50+ VM’s on a single volume if you had a valid reason to.

The next thing that came to mind is that I will be able to achieve higher consolidation ratios on vSphere hosts if needed. Due to the savings in CPU, Network and Storage I/O overhead we can use that savings to host more virtual machines on each host. In particular if you are using Blade Chassis you can expect to see a lot of network I/O traffic savings since you can have up to 16 blades in a chassis depending on your vendor. That can equal to a huge decrease in traffic flowing through those shared ports.

Something that I was wondering about and saw little discussion about was what type of extra load does VAAI functions place on the array. I reached out and asked this question to Chad Sakac of EMC and Vaughn Stewart of NetApp. Both contacts replied back via Twitter and stated that currently VAAI adds little to no extra burden to the arrays and in coming firmware updates it’s expected to be even less.

Lastly to sum up what you need to take advantage of VAAI. You will need to have vSphere 4.1 and you need to be licensed for Enterprise or Enterprise Plus. Next you must have a storage array that supports VAAI, this is probably going to be the largest hurdle for most. If you array was purchased within that last 2 years there is a good change that with a firmware upgrade your array may support VAAI. If not you will need to purchase a new one and this is an expensive investment. So it’s conceivable that many smaller shops will never get to reap the benefits of VAAI because of these requirements.

About Brian Suhr

Brian is a VCDX5-DCV and a Sr. Tech Marketing Engineer at Nutanix and owner of this website. He is active in the VMware community and helps lead the Chicago VMUG group. Specializing in VDI and Cloud project designs. Awarded VMware vExpert status 6 years for 2016 - 2011. VCP3, VCP5, VCP5-Iaas, VCP-Cloud, VCAP-DTD, VCAP5-DCD, VCAP5-DCA, VCA-DT, VCP5-DT, Cisco UCS Design

Read More

How to balance VMware ESX hosts paths on HP EVA arrays

Here at 64k, in our smaller cube near the vending machines, we storage-oriented folks like to mull over ideas big and small, 4k at a time.  We also deal in a great number of puns, so consider yourself warned.  Today, in our maiden voyage, I’d like to talk about some of my experience with HP’s line of EVA storage arrays.  As many of our readers know, the EVA line is a middle tier offering from HP.  Though likely to be usurped in the near future by 3PAR’s goodies, I am not here to begin that debate.  Rather, let us delve into a few common gotcha’s that can be overlooked in environments where EVAs live.

ONE]

The tight rope act begins with the storage array, our bright and shiny EVA.  At a fundamental level, an EVA is comprised of two controllers.  The operating environment of the EVA is such that it can, in a semi-intelligent fashion, manage vdisk ownership between the two controllers itself.  By default, vdisks are set to no preference for a failover/mode setting at the time of creation.   This means the EVA will decide which controllers get which vdisks when it (the EVA itself) boots.  Every vdisk is assigned to a controller (and only one controller).  If the non-owning controller is receiving the IO for a server(s) talking to a vdisk, it will after a period of time change the ownership of the vdisk.  This will reduce the load crossing the mirror ports.   While the EVA can run in this fashion, it is sub-optimal.

The other side of the tight rope of this balancing act is the hosts.  IO can walk many paths from host to array, some optimal and others not.  The start of such begins at the host’s adapter.  If it is a dual port (or multiple single port) host, then you have all the more paths to choose from.  Even in the case of a single port host, you can still cover multiple paths to arrive at the vdisk.  The handling of the proper path comes in the form of multipathing software.  From HP for Microsoft operating systems, we have Device Specific Module (DSM), which uses MS’s MPIO stack as its basis.  HP makes specific DSM’s for each of its line of arrays.  Without the MPIO stack, the host will see a drive presented once for each host port.  In an 8×00 series array, that is 8!  So clearly the MPIO software and HP’s DSM is needed for correct operation.  The default install does not enable Adaptive Load Balance (ALB).  This hampers read operations by not passing through the correct controller for a vdisk.  Note that non-MS based operating systems (like VMware) have their own multipathing stacks.  In the case of VMware ESX(i) 3.x, the options are fixed and mru.  In the case of vSphere, we get round robin added to the mix.  In pre-vSphere environments, the fixed path does not by default balance load across the host ports.  You can end up with all your VM traffic running over one host port!  Yikes!

TWO]

Now, to balance things out, let me start with the array.  A good habit to get into involves understanding your environment from an IO perspective.  You need to understand the profile, or workload, of your IO, so that you can balance between the controllers (among other things!).  Make sure to capture your performance data using evaperf (or other tools) to allow you the view of your controller’s current load.  As you add new vdisks, you can balance them by setting the failover/mode setting to the controller with failover + failback.  This will allow the balancing to remain should you lose and regain a controller.  Further, this specifies the controller for the vdisk in terms of mastership.  This helps from the host side as the controller it needs to talk through is clearly defined.  One thing to keep in mind also is the need to accept all load on one controller should failure occur.  This should be something you are aware of via your performance data.  A good rule of thumb is a controller should be no more than 30% ideally (at least in my experience).   And as always, have the latest Command View and XCS code.  One other thing to check for balance is to make sure the host ports are set to their top speed (4GB, except the very old EVA models) as well as properly balanced on the fabric (equal ports on both sides).  One customer I came across had all ports from controller A on fabric A and all ports of controller B on fabric B!  Definitely a big problem there!

For the host side, there is a bit more that can be done.  There is some work to be done on the array as well, which I will address.  The hosts should have the latest firmware, drivers, and software for their HBAs.  Additionally, make sure you have the latest HP DSM software.   Within the DSM software, you will want to enable Automatic Load Balancing.  As I stated before, this is not enabled by default.  To enable, just right click on each LUN (listed by WWN) that is listed and choose Enable ALB.

So, as a quick explanation:  write requests from hosts will hit the controller that owns the vdisk in question, but that write will propagate over the mirror link into both controllers’ cache.  This is in case a controller is lost, the write can still be committed.  Read requests will hit whichever controller, and if it is the wrong controller, will have to travel over the mirror ports to the correct controller.  This is sub-optimal, but is alleviated by enabling ALB.  ALB communicates with the array and will always communicate its read requests through the owning controller.  Very handy!

Now, from a VMware standpoint, let’s talk about fixed and then round robin (two most common multipathing situations found today).  For Fixed, you will need to balance IO to your datastores over the host ports of the controllers.  Also keep in mind which controller you selected at the array.  As an example, if I have 8 datastores of average IO (no virtualized heavy apps) then I would want 4 datastores on each controller.  To further balance, I would have each datastore talking over one of the host ports for each of the controllers (4 ports per controller x 2 controllers).  The IO is evenly balanced.  To set this, simply go into each datastore properties (via the VI Client) and pick the WWN for the corresponding host port).  Under heavy IO circumstances, you may not be able to move your traffic to a different host port.  Just try again at a later date.  When it comes to round robin, the IO works a bit differently.  Round Robin will send IO to each host port in turn after a certain amount of IOPS.   In the HP best practices for vSphere on the EVA, it states to change this value to 1 (and thus pushing even IOPS over every host port visible).  There was a bug which would, after a reboot of the ESX(i) host, reset this to a very high number.  I have found in my experience that leaving it as-is seems to work fairly well.  I would guess there is good reason that HP came up with that figure, and so at this point, with vSphere 4.1, I would suspect you could set this without issue.

Summary

Presented here are some of the findings I have come across in working with different customers.  I figure that having these kinds of storage discussions can help to make for a very engaging conversation.  Let me know what you think (and if I make any errors, which being human, am prone to!

Read More
Page 3 of 1012345...10...Last »