How to balance VMware ESX hosts paths on HP EVA arrays

Here at 64k, in our smaller cube near the vending machines, we storage-oriented folks like to mull over ideas big and small, 4k at a time.  We also deal in a great number of puns, so consider yourself warned.  Today, in our maiden voyage, I’d like to talk about some of my experience with HP’s line of EVA storage arrays.  As many of our readers know, the EVA line is a middle tier offering from HP.  Though likely to be usurped in the near future by 3PAR’s goodies, I am not here to begin that debate.  Rather, let us delve into a few common gotcha’s that can be overlooked in environments where EVAs live.


The tight rope act begins with the storage array, our bright and shiny EVA.  At a fundamental level, an EVA is comprised of two controllers.  The operating environment of the EVA is such that it can, in a semi-intelligent fashion, manage vdisk ownership between the two controllers itself.  By default, vdisks are set to no preference for a failover/mode setting at the time of creation.   This means the EVA will decide which controllers get which vdisks when it (the EVA itself) boots.  Every vdisk is assigned to a controller (and only one controller).  If the non-owning controller is receiving the IO for a server(s) talking to a vdisk, it will after a period of time change the ownership of the vdisk.  This will reduce the load crossing the mirror ports.   While the EVA can run in this fashion, it is sub-optimal.

The other side of the tight rope of this balancing act is the hosts.  IO can walk many paths from host to array, some optimal and others not.  The start of such begins at the host’s adapter.  If it is a dual port (or multiple single port) host, then you have all the more paths to choose from.  Even in the case of a single port host, you can still cover multiple paths to arrive at the vdisk.  The handling of the proper path comes in the form of multipathing software.  From HP for Microsoft operating systems, we have Device Specific Module (DSM), which uses MS’s MPIO stack as its basis.  HP makes specific DSM’s for each of its line of arrays.  Without the MPIO stack, the host will see a drive presented once for each host port.  In an 8×00 series array, that is 8!  So clearly the MPIO software and HP’s DSM is needed for correct operation.  The default install does not enable Adaptive Load Balance (ALB).  This hampers read operations by not passing through the correct controller for a vdisk.  Note that non-MS based operating systems (like VMware) have their own multipathing stacks.  In the case of VMware ESX(i) 3.x, the options are fixed and mru.  In the case of vSphere, we get round robin added to the mix.  In pre-vSphere environments, the fixed path does not by default balance load across the host ports.  You can end up with all your VM traffic running over one host port!  Yikes!


Now, to balance things out, let me start with the array.  A good habit to get into involves understanding your environment from an IO perspective.  You need to understand the profile, or workload, of your IO, so that you can balance between the controllers (among other things!).  Make sure to capture your performance data using evaperf (or other tools) to allow you the view of your controller’s current load.  As you add new vdisks, you can balance them by setting the failover/mode setting to the controller with failover + failback.  This will allow the balancing to remain should you lose and regain a controller.  Further, this specifies the controller for the vdisk in terms of mastership.  This helps from the host side as the controller it needs to talk through is clearly defined.  One thing to keep in mind also is the need to accept all load on one controller should failure occur.  This should be something you are aware of via your performance data.  A good rule of thumb is a controller should be no more than 30% ideally (at least in my experience).   And as always, have the latest Command View and XCS code.  One other thing to check for balance is to make sure the host ports are set to their top speed (4GB, except the very old EVA models) as well as properly balanced on the fabric (equal ports on both sides).  One customer I came across had all ports from controller A on fabric A and all ports of controller B on fabric B!  Definitely a big problem there!

For the host side, there is a bit more that can be done.  There is some work to be done on the array as well, which I will address.  The hosts should have the latest firmware, drivers, and software for their HBAs.  Additionally, make sure you have the latest HP DSM software.   Within the DSM software, you will want to enable Automatic Load Balancing.  As I stated before, this is not enabled by default.  To enable, just right click on each LUN (listed by WWN) that is listed and choose Enable ALB.

So, as a quick explanation:  write requests from hosts will hit the controller that owns the vdisk in question, but that write will propagate over the mirror link into both controllers’ cache.  This is in case a controller is lost, the write can still be committed.  Read requests will hit whichever controller, and if it is the wrong controller, will have to travel over the mirror ports to the correct controller.  This is sub-optimal, but is alleviated by enabling ALB.  ALB communicates with the array and will always communicate its read requests through the owning controller.  Very handy!

Now, from a VMware standpoint, let’s talk about fixed and then round robin (two most common multipathing situations found today).  For Fixed, you will need to balance IO to your datastores over the host ports of the controllers.  Also keep in mind which controller you selected at the array.  As an example, if I have 8 datastores of average IO (no virtualized heavy apps) then I would want 4 datastores on each controller.  To further balance, I would have each datastore talking over one of the host ports for each of the controllers (4 ports per controller x 2 controllers).  The IO is evenly balanced.  To set this, simply go into each datastore properties (via the VI Client) and pick the WWN for the corresponding host port).  Under heavy IO circumstances, you may not be able to move your traffic to a different host port.  Just try again at a later date.  When it comes to round robin, the IO works a bit differently.  Round Robin will send IO to each host port in turn after a certain amount of IOPS.   In the HP best practices for vSphere on the EVA, it states to change this value to 1 (and thus pushing even IOPS over every host port visible).  There was a bug which would, after a reboot of the ESX(i) host, reset this to a very high number.  I have found in my experience that leaving it as-is seems to work fairly well.  I would guess there is good reason that HP came up with that figure, and so at this point, with vSphere 4.1, I would suspect you could set this without issue.


Presented here are some of the findings I have come across in working with different customers.  I figure that having these kinds of storage discussions can help to make for a very engaging conversation.  Let me know what you think (and if I make any errors, which being human, am prone to!

Read More

Everything you wanted to know about HP BladeSystem Matrix

With all the talk about converged infrastructure and stacks these days especially in the virtualization space I was really glad that I got to do this interview. There has been a lot written about its competitors but the HP BladeSystem Matrix was still kind of a mystery to anyone that had not had HP in to talk about it. I was lucky enough to spend some time talking with a couple of members from the HP BladeSystem Matrix team. These guys were very helpful in explaining what Matrix is and answered all of my crazy questions.

What I hope everyone gets from this is a better understanding of what BladeSystem Matrix has to offer if you’re looking at these types of converged offerings. Also highlight some of the features that are unique to the HP stack. In interest of being totally open I am also an employee of HP but my current work responsibilities have nothing to do with BladeSystem Matrix. Now that all that is out of the way let’s get started with the good stuff.

VT: Can you give me your elevator pitch?
: Matrix is the foundation for a private cloud solution managing both physical and virtual infrastructure. Matrix allows you to rapidly provision infrastructure via a self service portal. In addition, it offers the ongoing life-cycle management including capacity planning and disaster recovery. You can buy Matrix with a single SKU that includes hardware, software and services. The solution is all tested and certified by HP to work together.

VT: Who benefits from this solution?
: Customers who need to be able to address fast change and achieve a competitive advantage through time to market. Typical customers for Matrix are large Enterprises and Service Providers who have invested already in virtualization and shared infrastructure and want to take the next step to cloud computing. I think that these target customers are common to all converged infrastructure offerings.

VT: What hardware makes up a BladeSystem Matrix?
: BladeSystem Matrix all begins with something called a starter kit. This kit includes the following items, Central Management Server on a ProLiant DL360, HP C7000 Blade Chassis w/Virtual Connect networking and Insight Management software for managing Matrix. For the storage you have multiple options – you can use your existing Fiber Channel SAN storage if it’s supported or you can use HP storage, e.g. 3PAR or HP EVA 4400 array. iSCSI storage is supported as well for VM data stores. There is also something called an Expansion kit which is a C7000 Blade chassis, Insight Management software licenses and HP Services needed to integrate the expansion kit into your existing Matrix environment. It should be noted that Matrix supports both ProLiant and Integrity blades.

VT: What are HP Cloud Maps and how do they relate to BladeSystem Matrix?
: These Cloud Maps help customers to get started quickly with Matrix – they jump start the creation of a customized self-service portal.  Cloud Maps include white papers and templates for hardware or software configurations that can be imported into BladeSystem Matrix that can save days or weeks of design time. A Cloud Map can also provide workflows and scripts designed to expedite the installation.

VT: What does the CMS or Central Management Server do?
: The CMS server is a physical server that is running the management software that controls, automates and monitors your BladeSystem Matrix. If you have a DR site with a Matrix you would need a CMS server there to control the environment. It’s also possible to setup the CMS in a HA or Highly Available configuration to prevent a single failure point for Matrix management. Lastly for large environments that exceed the maximums of a single CMS you can now stand up secondary CMS servers that will still allow you to manage everything from one admin console.

VT: Can I use existing HP gear with a Matrix install?
: If you purchase a new HP BladeSystem Matrix you can use it to also manage any qualifying HP hardware that you already own. HP has created something called the Matrix Conversion Services to assist with integrating your existing HP infrastructure with BladeSystem Matrix. This program is new and will evolve to allow customers to accomplish these integrations.

VT: Can I use arrays from other vendors?
: You can use Storage Arrays from other vendors as long as they are able to meet a list of criteria – for example the storage vendor needs to be certified with Virtual Connect.  More details can be found in the Matrix compatibility chart.

VT: What software is used for Matrix?
: The software for Matrix is called the Matrix Operating Environment, which includes the whole Insight Management stack including Insight Foundation and Insight Control. With Insight Foundation you get the controls to install, configure, and monitor physical servers. With Insight Control you get all the essential server management including server deployment and power management. The real magic happens with the additional Matrix Operating Environment software (aka Insight Dynamics). It provides a service design tool, infrastructure provisioning with a self-service portal, capacity planning, and recovery management

VT: Does it come configured and who does the setup work?
: Some factory configuration is done then remaining work is done onsite by HP Services. The install and configure period can take from a few days to 2 weeks depending on the level of complexity.

VT: Explain how it is managed?
: There are two separate consoles that control a BladeSystem Matrix. The first would be the admin console used by your support team to configure and control the environment. The second would be the Self Service portal site. This allows for IT consumers to request and provision resources from the Matrix environment.

VT: What types of automation and provisioning can Matrix do?
: One example would be in the creation of templates. You can create templates in the Matrix software or use ones already created, for example on your VMware vCenter server. If you use an existing template that might be created with only one OS partition you can use the Matrix template process to provision the VM and add on additional disks and features not present in the base template.

VT: How is support handled for Matrix customers?
: There is a dedicated team to contact for Matrix support issues. Matrix is treated as a single solution, with all calls coming in through a central team. This team is cross trained in the various aspects that make up Matrix and they will escalate to product specific engineers if needed.

VT: Can you explain fail over P2V and then back to V2P for DR?
: This feature allows for a physical server to be recovered at the DR site on a physical or virtual machine. To make this HP spoke about creating what is known as a “portable image” What this meant was that the logical server was created in a way that it would be able to be deployed on either another physical blade, or as a VM within a virtual machine host. . I asked about if there was any type of conversion process that takes place and there is not. The engineer talked about the creation of the portable image which to me meant that you need to include both OS drivers for the physical hardware and the virtual hardware. This way when the imaged was moved to the other platform the physical OS or the hypervisor-based OS would find all of the devices. The last piece would be the network settings and these are preserved with an application called PINT so that when new network cards are installed your settings will remain.

VT: How does it integrate with VMware?
: The HP tool set for BladeSystem Matrix offers many integration points with VMware vSphere. A short list of the functions would include provisioning VM’s, change in power state, activate/deactivate, add servers to group, and add disks to a VM or group of VM’s. Along with those features Matrix monitors status and performance, capacity & workload analysis and Disaster Recovery integration.

VT: What separates Matrix from other converged stacks?
: A big selling point is that HP BladeSystem Matrix is integrated and engineered holistically by one company, while still allowing for heterogeneous components in areas such as networking and storage. Also at this time BladeSystem Matrix is the only solution that is capable of managing both physical and virtual servers with the same tools and allow movement between physical and virtual resources. Something that Matrix offers that others do not is integrated automated Disaster Recovery. Lastly Matrix supports both VMware and Microsoft Hyper-V, as well as Integrity Blades, for virtualization.

VT: What SAN protocols are supported today?
: As of today BladeSystem Matrix supports Fiber Channel as the preferred method of connecting to storage. In addition, Matrix does support FCOE and iSCSI for VM data stores.

VT: What is storage provisioning manager?
: This was explained as enhanced volume provisioning management, allowing more proactive maintenance of the pools of storage available for provisioning of an environment. Where this seem to tie for me was using it to publish or tag which volumes are available for provisioning. For example you could label a volume as boot disk and others as data disks. Then when creating your templates for provisioning the system will know which volumes are available for boot, as well as which volumes are available as data volumes during OS install, so that you provide better management of the storage you’ll utilize during provisioning.

VT: How many customers or units sold so far?
: I had to try but was only told that HP does not release any numbers or revenues for products. BladeSystem Matrix is made up of components that have been offered for many years by HP, and includes multi-million unit sales of components such as BladeSystem servers and Virtual Connect.

VT: How will software and firmware updates be handled?
: There are update bundles that are created for BladeSystem Matrix. At this time these updates must be performed by an HP Services person. These updates can be done in person or remotely.

VT: How does the SAN fabric interact with BladeSystem Matrix?
: In the current version of Matrix you will need to pre-create volumes and your server zoning ahead of any provisioning.

VT: What is Insight Virtualization Manager?
: Also known as VSE Virtualization Manager that is part of Insight Dynamics. With VSE you can move a logical server from the existing blade it’s running on to another blade. The VSE application will move the server profile to the new blade and restart the server once the move is complete and your operating system will start up. The VSE interface will offer recommendations for target blades that match your requirements. There are a few reasons for such a move that would include upgrades and maintenance. Video demo of moving a blade server to another blade. Video Link

About Brian Suhr

Brian is a VCDX5-DCV and a Sr. Tech Marketing Engineer at Nutanix and owner of this website. He is active in the VMware community and helps lead the Chicago VMUG group. Specializing in VDI and Cloud project designs. Awarded VMware vExpert status 6 years for 2016 - 2011. VCP3, VCP5, VCP5-Iaas, VCP-Cloud, VCAP-DTD, VCAP5-DCD, VCAP5-DCA, VCA-DT, VCP5-DT, Cisco UCS Design

Read More

Issues with Controller replacement and Firmware upgrade on HP MSA 2312fc

Recently I was pulled into an issue involving an HP MSA 2312fc array. The device had a failed Controller B and for some reason it was also affecting the A Controller. This was causing the OS on the servers receiving storage from the device to loose connection to the vdisks. The way things are suppose to work with the MSA as in any array is for the other controller to take ownership and continue to provide connections to the vdisks. For some reason this did not happen.

When looking at the management console on the MSA 2312 you could clearly see that controller B was failed and that A had taken ownership over of any vdisks. But something with the fail over did not fully complete or corrupted something because it did not work as expected. Even with removing the failed controller from the device the vdisks were still not usable by the servers.

Once the failed part was replaced the configuration was copied to the replaced controller and service was restored. Then it was time to make sure the firmware levels matched on both controllers and update accordingly.

In the past I had been warned to proceed with caution about updating firmware levels on MSA arrays. I had read several horror stories about them getting stuck in loops during the update process. We did encounter this when the controllers tried to sync them selves to match the firmware levels. I was told by an HP Support Engineer that it’s best to disable this auto sync feature when it comes to updating firmware to prevent these loops. It can then be enabled after the updates are done on both controllers.

Below are a few steps to prevent the automatic firmware update from the partner controller. The first image below shows you how to access the Firmware options from the Advanced Settings area of the Configuration menu.

The next image is showing the option to turn on or off the Partner Firmware Update option. This determines if the firmware on a new controller is auto updated when it is inserted. The HP support engineer recommended that we disable this option prior to installing the new controller. If you did not follow this you can always come back and change and they wait for the current update to fail. This is what we had to do that ended up stopping the continuous loop that was happening.

The last image just shows what you will see if you log into the management controller that is being updated.

About Brian Suhr

Brian is a VCDX5-DCV and a Sr. Tech Marketing Engineer at Nutanix and owner of this website. He is active in the VMware community and helps lead the Chicago VMUG group. Specializing in VDI and Cloud project designs. Awarded VMware vExpert status 6 years for 2016 - 2011. VCP3, VCP5, VCP5-Iaas, VCP-Cloud, VCAP-DTD, VCAP5-DCD, VCAP5-DCA, VCA-DT, VCP5-DT, Cisco UCS Design

Read More

How to initialize a new HP EVA storage array

I’ve been thinking about writing this post for a while and finally got around to doing it. This is something that needs to be done to a new EVA array when they are first installed. I have found that sometimes this work is done by the installing Engineer or you can do it yourself if your setup was not ready when the device was installed.

Before you can start you need to have a few things setup and ready before you can initialize the array. The first thing that must be done is to install HP Command View EVA on a server that is SAN connected and has been zoned in to see the EVA array. This is essential because Command View EVA is used to manage the array.

You can see from the image below that I have logged into Command View EVA and it’s showing my new EVA and from the lower part of the screen in Red that the System is Uninitialized. So the first thing that I like to do is look over the EVA and make sure things match up to what was ordered. Once I am comfortable that everything is as expected I am ready to Initialize the array. To get this started just click on the Initialize button located at the top of the main screen.

After clicking on the Initialize button you will be presented with the following confirmation screen. The system is warning you that any data on drives will be lost. This is not a concern in this situation since this is a new Storage Array.

After clicking OK on the warning screen you will be presented with the following screen that allows you to set several features that will affect how your Array will function. I will not cover them all but the main ones. The first and most obvious is setting the System Name this will be what you want the EVA system name to be. Next is to select the Disk Failure Protection settings, this allows you to select the number of Hot Spare disks that will protect your Disk Groups. The options for this setting are None, Single or Double. This is a setting that should be selected carefully based on your needs. The next setting is the Disk group type setting, depending on your choice here will determine which Raid selections are available for use. With the Standard option you can use Raid 0, 1 and 5, and by selecting the Enhanced option you get the same Raid levels with the addition of Raid 6 to use.

In the section just below the name field you can set the number of disks that will be placed in the Default Disk group that will be created in this process. In the image below it’s showing us there are 74 disks online. In the field I have selected to have 8 place in the group, the remaining 62 disks would be place in the ungrouped disks section. If you know all of the disks are of the same size and class you could just dump them all into the initial group and save time.

The lower part of the screen from above is allowing you to set how the system time will be managed. I have selected to use the Management server which is the Command View server. Once all of your settings are done click the Initialize button at the top of the screen.

You will be presented with the warning shown below that confirms the disk group settings selected in the previous stage.

After confirming the warning listed above the system will start to initialize. You will seen a screen like the one below. This task can take from a few minutes to more than an hour.

Once the Initialize process completes you will get a confirmation screen like the one below and from the tree view on the left the array now reflects the system name that was assigned to it.

You can see from the image below that I have now expanded the tree to show the disk groups. In the default disk group that was created you can see the 8 disks that I choose to have placed in there. The remaining disks in the array were placed in the Ungrouped Disks section. I will change the name of the Default Disk group and continue to add more disks to the group.

From the image below you can now see the new disk group name. I will next start to add the Ungrouped disks into the group that I created. I won’t go into HP’s strategy to managing EVA arrays I’ll leave that for another post. But the short version is HP recommends creating large disk groups with like sized disks and add them in multiples of 8 disks. Also from the image below when clicking on a disk you can see the firmware version on the disk and its capacity size. You can use a Code Load when you finish grouping the disks to make sure they are all up to date on Firmware.

To add the disk to the group I created you just need to click on the disk and then click the Group button. You will then be asked to confirm the selection.

In this image you are asked to confirm the selection to group the disk. You must type YES and it does have to be in capital letters.

After passing the confirmation screen it will ask you what Disk Group you would like to add the disk to. You can see from the image below that you just choose the correct Group from the drop down list and then click Add Disk button.

And the final step shows that the Operation succeeded and the disk is now in the Disk Group that we selected. Now click OK and continue adding more disks to the group. Unfortunately using the Command View EVA console the disks can only be grouped one by one, so the process  can take awhile. If your a scripting person you may be able to use the SSSU command line that HP provides to add disks to a group.

About Brian Suhr

Brian is a VCDX5-DCV and a Sr. Tech Marketing Engineer at Nutanix and owner of this website. He is active in the VMware community and helps lead the Chicago VMUG group. Specializing in VDI and Cloud project designs. Awarded VMware vExpert status 6 years for 2016 - 2011. VCP3, VCP5, VCP5-Iaas, VCP-Cloud, VCAP-DTD, VCAP5-DCD, VCAP5-DCA, VCA-DT, VCP5-DT, Cisco UCS Design

Read More