With the release of vSphere 4.1 last summer VMware customers were provided several new features. Many of these new features were created to lower the burden on the ESX host by being more efficient or offloading the work to something outside of the Virtualization stack. The overall goal of the new features was to continue to improve performance of virtual machines. The one that I am writing about today is VAAI or vStorage API for Array Integration. I wanted to write about how using VAAI in your Architecture Designs is changing the way you are creating environments.
The goal of VAAI is too offload some of the storage focused activities that VMware had previously handled to your storage array. This was accomplished by VMware working closely with the major storage vendors. The idea of VAAI was first announced back at VMworld 2008 and finally came to market when vSphere 4.1 was released. By offloading these storage functions it has reduced the load on the ESX(i) hosts and also increased the performance of these activities by letting the storage array do the work that it was created to do.
In the current offering of VAAI there are 3 functions that have been offloaded. In future product releases it is expected that VMware will continue to work with storage vendors to increase the features of VAAI and the currently available features are explained below.
Full Copy – So you’re probably wondering how this feature is going to help me. I can think to two VMware functions that this VAAI feature provides upwards of 10x speed improvements in. The first would be when you are deploying a VM from a template. We will say for example that you are going to deploy a 50 GB VB. When the VM is deployed vSphere is going to read the entire 50 GB and then write the 50 GB for a total of 100 GB of I/O. With VAAI enabled and a storage array that supports VAAI this process creates very little I/O at the host. The vSphere host will send a command to the storage array that say make a copy of this VM and name it this. The copy will be done locally on the storage array and results in very little I/O between the host and array. Once completed the array will send a notice to the host to let it know the works was completed.
The second VMware feature to benefit from this is a Storage vMotion. I feel that this is where this really pays off because you are most likely moving a larger chunk of data with this command. For example sake let’s say we are going to move a 100 GB Virtual Machine from one disk to another. To do this in the past this would have caused 200 GB of I/O on the host. With VAAI the burden on the host is almost nothing as this work is done on the storage array.
Hardware assisted locking – Too allow multiple hosts in your cluster to talk to the same storage volume VMware would lock the volume when one of the VM’s needed to write to it. This locking is to prevent another host from trying to write to the same blocks. This was not a large issue If you were using smaller volumes with only a handful of virtual machines on them. Now with VAAI the locking has been offloaded to the storage array and it’s now possible to lock only the blocks that are being written to. This opens up the possibility to use larger volumes and increase the amount of VM’s that can be run on a single volume.
Block Zeroing – This feature is saving vSphere from having to send redundant commands to the array for writes. The host can simple tell the storage array which blocks are zeros and move on. The storage device will handle the work without needing to receive repetitive write commands from the host.
So now that you should have an understanding of what VAAI is and how it should help free up resources. I will now talk about how this changes the way we should be thinking about different design considerations.
The first thing that comes to mind is that I can now think about using larger datastores without the worry of affecting performance due to locking issues. With VAAI the storage device is going to handle the locking and allow me to have far for VM’s per volume than the 5-10 previously held as a guideline to live by in past versions. It’s now possible to have 50+ VM’s on a single volume if you had a valid reason to.
The next thing that came to mind is that I will be able to achieve higher consolidation ratios on vSphere hosts if needed. Due to the savings in CPU, Network and Storage I/O overhead we can use that savings to host more virtual machines on each host. In particular if you are using Blade Chassis you can expect to see a lot of network I/O traffic savings since you can have up to 16 blades in a chassis depending on your vendor. That can equal to a huge decrease in traffic flowing through those shared ports.
Something that I was wondering about and saw little discussion about was what type of extra load does VAAI functions place on the array. I reached out and asked this question to Chad Sakac of EMC and Vaughn Stewart of NetApp. Both contacts replied back via Twitter and stated that currently VAAI adds little to no extra burden to the arrays and in coming firmware updates it’s expected to be even less.
Lastly to sum up what you need to take advantage of VAAI. You will need to have vSphere 4.1 and you need to be licensed for Enterprise or Enterprise Plus. Next you must have a storage array that supports VAAI, this is probably going to be the largest hurdle for most. If you array was purchased within that last 2 years there is a good change that with a firmware upgrade your array may support VAAI. If not you will need to purchase a new one and this is an expensive investment. So it’s conceivable that many smaller shops will never get to reap the benefits of VAAI because of these requirements.
Brian is a Technical Architect for a VMware partner and owner of this website. He is active in the VMware community and is helps lead the Chicago VMUG group. This blog Virtualize Tips was started to document and remember things that I come across while working with tech.
Here at 64k, in our smaller cube near the vending machines, we storage-oriented folks like to mull over ideas big and small, 4k at a time. We also deal in a great number of puns, so consider yourself warned. Today, in our maiden voyage, I’d like to talk about some of my experience with HP’s line of EVA storage arrays. As many of our readers know, the EVA line is a middle tier offering from HP. Though likely to be usurped in the near future by 3PAR’s goodies, I am not here to begin that debate. Rather, let us delve into a few common gotcha’s that can be overlooked in environments where EVAs live.
ONE]
The tight rope act begins with the storage array, our bright and shiny EVA. At a fundamental level, an EVA is comprised of two controllers. The operating environment of the EVA is such that it can, in a semi-intelligent fashion, manage vdisk ownership between the two controllers itself. By default, vdisks are set to no preference for a failover/mode setting at the time of creation. This means the EVA will decide which controllers get which vdisks when it (the EVA itself) boots. Every vdisk is assigned to a controller (and only one controller). If the non-owning controller is receiving the IO for a server(s) talking to a vdisk, it will after a period of time change the ownership of the vdisk. This will reduce the load crossing the mirror ports. While the EVA can run in this fashion, it is sub-optimal.
The other side of the tight rope of this balancing act is the hosts. IO can walk many paths from host to array, some optimal and others not. The start of such begins at the host’s adapter. If it is a dual port (or multiple single port) host, then you have all the more paths to choose from. Even in the case of a single port host, you can still cover multiple paths to arrive at the vdisk. The handling of the proper path comes in the form of multipathing software. From HP for Microsoft operating systems, we have Device Specific Module (DSM), which uses MS’s MPIO stack as its basis. HP makes specific DSM’s for each of its line of arrays. Without the MPIO stack, the host will see a drive presented once for each host port. In an 8×00 series array, that is 8! So clearly the MPIO software and HP’s DSM is needed for correct operation. The default install does not enable Adaptive Load Balance (ALB). This hampers read operations by not passing through the correct controller for a vdisk. Note that non-MS based operating systems (like VMware) have their own multipathing stacks. In the case of VMware ESX(i) 3.x, the options are fixed and mru. In the case of vSphere, we get round robin added to the mix. In pre-vSphere environments, the fixed path does not by default balance load across the host ports. You can end up with all your VM traffic running over one host port! Yikes!
TWO]
Now, to balance things out, let me start with the array. A good habit to get into involves understanding your environment from an IO perspective. You need to understand the profile, or workload, of your IO, so that you can balance between the controllers (among other things!). Make sure to capture your performance data using evaperf (or other tools) to allow you the view of your controller’s current load. As you add new vdisks, you can balance them by setting the failover/mode setting to the controller with failover + failback. This will allow the balancing to remain should you lose and regain a controller. Further, this specifies the controller for the vdisk in terms of mastership. This helps from the host side as the controller it needs to talk through is clearly defined. One thing to keep in mind also is the need to accept all load on one controller should failure occur. This should be something you are aware of via your performance data. A good rule of thumb is a controller should be no more than 30% ideally (at least in my experience). And as always, have the latest Command View and XCS code. One other thing to check for balance is to make sure the host ports are set to their top speed (4GB, except the very old EVA models) as well as properly balanced on the fabric (equal ports on both sides). One customer I came across had all ports from controller A on fabric A and all ports of controller B on fabric B! Definitely a big problem there!
For the host side, there is a bit more that can be done. There is some work to be done on the array as well, which I will address. The hosts should have the latest firmware, drivers, and software for their HBAs. Additionally, make sure you have the latest HP DSM software. Within the DSM software, you will want to enable Automatic Load Balancing. As I stated before, this is not enabled by default. To enable, just right click on each LUN (listed by WWN) that is listed and choose Enable ALB.
So, as a quick explanation: write requests from hosts will hit the controller that owns the vdisk in question, but that write will propagate over the mirror link into both controllers’ cache. This is in case a controller is lost, the write can still be committed. Read requests will hit whichever controller, and if it is the wrong controller, will have to travel over the mirror ports to the correct controller. This is sub-optimal, but is alleviated by enabling ALB. ALB communicates with the array and will always communicate its read requests through the owning controller. Very handy!
Now, from a VMware standpoint, let’s talk about fixed and then round robin (two most common multipathing situations found today). For Fixed, you will need to balance IO to your datastores over the host ports of the controllers. Also keep in mind which controller you selected at the array. As an example, if I have 8 datastores of average IO (no virtualized heavy apps) then I would want 4 datastores on each controller. To further balance, I would have each datastore talking over one of the host ports for each of the controllers (4 ports per controller x 2 controllers). The IO is evenly balanced. To set this, simply go into each datastore properties (via the VI Client) and pick the WWN for the corresponding host port). Under heavy IO circumstances, you may not be able to move your traffic to a different host port. Just try again at a later date. When it comes to round robin, the IO works a bit differently. Round Robin will send IO to each host port in turn after a certain amount of IOPS. In the HP best practices for vSphere on the EVA, it states to change this value to 1 (and thus pushing even IOPS over every host port visible). There was a bug which would, after a reboot of the ESX(i) host, reset this to a very high number. I have found in my experience that leaving it as-is seems to work fairly well. I would guess there is good reason that HP came up with that figure, and so at this point, with vSphere 4.1, I would suspect you could set this without issue.
Summary
Presented here are some of the findings I have come across in working with different customers. I figure that having these kinds of storage discussions can help to make for a very engaging conversation. Let me know what you think (and if I make any errors, which being human, am prone to!
I’ve been waiting for someone to put something like this together for awhile now. If you wanted to know which Storage vendors are supporting VMware VAAI today or when they will be this list should help out. For myself it was clear that EMC and Equalogic had plugins out but only heard that others were coming soon. I would like to give big thanks to Tomi Hakala for pulling this list together. Watch his blog v-reality.info for updates.
I have to say that I’m a bit skeptical of the HP release dates for VAAI support. I have inquired about this internally with Product Managers and all I could get was sometime in calendar year 2011. That makes me think that it wont be Q1. I guess time will tell.
Update 12/22/2010 – I have been hearing the VAAI support for HP EVA and XP series will be in second half of 2011. Looks like they are saying that next major release of vSphere will be supporting a SCSI T10 standards based support for VAAI functions and they are waiting on this to release support. They will develop based on the standards support vs proprietary support.
Brian is a Technical Architect for a VMware partner and owner of this website. He is active in the VMware community and is helps lead the Chicago VMUG group. This blog Virtualize Tips was started to document and remember things that I come across while working with tech.
If you are familiar with VMware Lab Manager then this won’t be breaking news for you, but if your just getting started or looking for more details. Then I hope this post will help clear up how Network Templates and Virtual Networks work inside of Lab Manager. The ability to create Virtual Networks can be very helpful if you are looking to keep your Lab Manager Workspace separate from others.
There are of course other ways to separate virtual machines within Lab Manager. The most talked about method would be to use network Fencing that is a feature on Lab Manager Physical networks. When using fencing you can hide your configuration behind a virtual router and have the option to allow no in/out bound connection. This works great but can confuse some of your lesser experienced users.
If you don’t have the need to communicate with any servers outside of your configuration then using a Lab Manager Network Template might be easier. If setup properly the Virtual Networks within VMware Lab Manager can make your life easier and allow the IT customers to perform their work without any hand holding. By definition the Network Templates are separate Virtual Networks that can only talk with Virtual Machines within the same configuration. It does not matter if you choose the same Network Template with matching IP scheme on a different configuration the VMs will not communicate to the other workspaces. This is by design and is a good and simple way to create a Sand Box to test in.
If you choose to use IP Static – Pool selection for your Network Template you will assign a block of IP’s and Lab Manager will hand them out to the virtual machines as they are deployed. The IP pool will start over for each different configuration that you deploy so if you have 10 workspaces with 5 VMs each you don’t need 50 IP’s all you really need is 5. Since each Workspace or Configuration starts over at the first address in the pool. This is normal since they are not able to communicate with the other workspaces.
Here is a sample definition of the Lab Managers User guide to sum up Virtual Networks.
Virtual networks are configuration local networks. They exist inside configurations and do not span configurations. Virtual machines connected to a virtual network cannot communicate with virtual machines connected to a different virtual network. Each virtual network that Lab Manager creates is unique and isolated from other virtual networks, even if the networks are based on the same network template.
Brian is a Technical Architect for a VMware partner and owner of this website. He is active in the VMware community and is helps lead the Chicago VMUG group. This blog Virtualize Tips was started to document and remember things that I come across while working with tech.
So I have to admit this is not something that I paid much attention to in the past. But I’m currently working on creating documentation for a customer and putting together a Run Book for them to use with Lab Manager 4. During this process of running through things in the end user mind set I noticed this limit and just wanted to write something down for future reference.
Hopefully you are fairly familiar with how VMware Lab Manager works and I don’t loose you. The limit that I will cover here is when you are creating a New Configuration in a Workspace within Lab Manager when you select a VM Template to use by default the Datastore that it will be created on is locked. The Datastore that it is locked onto is the one that the VM Template is sitting on. This is done for a reason, since by default when creating these new configurations the new VMs created form the Template will be a Linked Clone of the original. For a Linked Clone to work it must reside on the same Datastore as the master VM image.
As you are probably aware that use Linked Clones will save you a lot of disk space on your Datastores so if you do not have a valid reason to not use them I would suggest to leave it this way. Now the only thing that I see is depending on the size of your VMs and your Datastore you could eventually run out of space for this Template on a particular Datastore. To solve that issue you could expand the Datastore or Clone your Template to another Datastore and start creating Linked Clones from that one. If anyone else has run into this issue and has a better idea drop something in the comments.
I have pasted a couple of screen shots below showing what the screens will look like with the default Linked Clone method. And then the option to do a Full Clone which will copy the entire template to the new VM and end up using all of the space.
Here is a note that I found in a VMware KB article to help explain this behavior also.
In vCenter Lab Manager, the configuration files will only be accessible from the same host or datastore as the base template. For example, if you have a Virtual Machine 1 on DataStore1, all the configurations you create from Virtual Machine 1 will be created on DataStore1.
Brian is a Technical Architect for a VMware partner and owner of this website. He is active in the VMware community and is helps lead the Chicago VMUG group. This blog Virtualize Tips was started to document and remember things that I come across while working with tech.
Some people were hoping to hear that the VMware iPad app for vSphere Management would be released at VMworld Copenhagen but that was not the case. We did get another look at the applications and a glimpse at a new VMware product that works nicely with the iPad also.
In the first section of the video we get another peek at the VMware vSphere management iPad app. There are some improvements that were made since we first say it at VMworld San Francisco. In the video he mentions that they hope to release a Beta version in a few weeks and then the final version late 2010 as a VMware Labs Fling. We get to see how you can now power cycle or restart a VM, get a look at home much CPU and Memory it’s consuming and some other details.
The next part of the video shows us the VMware View Client for iPad. We see a demo of it logging into the View portal at VMware corporate and connecting to a Windows 7 VDI machine. He then showcases the virtual Touch Pad that we had heard about in the last video from San Fran. The improvements that were made to the virtual keyboard are pretty cool. I like how they have added the Windows function keys and other standard keys that will make using a Windows VDI on an Apple iPad easier. VMware is definitely stepping up the level for portable VDI with this application.
In the final part of the video we get to see the web interface for the newly announced vCloud Request Manager from the iPad. This allows for requests to be made inside of vCloud Director that will require a managers approval. You are able to view all of your prior requests and see what is pending, approved or denied. This looks really cool and will go a long way for improving the mobility for people working with VMware and vCloud.
Brian is a Technical Architect for a VMware partner and owner of this website. He is active in the VMware community and is helps lead the Chicago VMUG group. This blog Virtualize Tips was started to document and remember things that I come across while working with tech.
