Posted by Brian on Apr 11, 2011 in CLI, vMA, VMware, vSphere | 0 comments
This is not something that I’ve had to do very often. But on a recent customer engagement I was working with the client on setting up some new hosts that were recently purchased. These hosts were purchased with Embedded ESXi on them and additional PCI NICs were added to the config. The additional NICs did not have drivers available in the base ESXi build. Shortly after bringing the first host online we noticed that only the onboard NICs showed up in the list.
A quick search on Google for the Intel part number for the NIC lead me to the family name for the adapter. Then a search over at VMware lead me to the download page for VMware that provided the .ISO file to load the drivers into ESXi for the family of adapters. The process took only a few minutes and since this is something that does not come up that often I thought a short write up might help someone.
There are a few ways that this could be done, since we happened to be running ESXi the options were to use the vMA or vCLI. Since this was a new install and a vMA was not setup yet I just quickly tossed vCLI on a server. Then a quick download of the driver .ISO from VMware and unzip the package into a folder on the server with vCLI installed on it. If you wanted to use the vMA you could mount the .ISO to the virtual CD-ROM of the VM and issue the command against it.
Since I was using vCLI all I needed to do was point the command to a local folder. Here is a sample of the command used to perform the patch.
vihostupdate –server HOSTNAME –install –bundle c:\folder\name_of_file.zip
To run this command your host must be in Maintenance mode and it will then take just a couple of minutes to execute. After the update completes a reboot of the host is needed and then the cards should be available for use.
Brian is a Technical Architect for a VMware partner and owner of this website. He is active in the VMware community and is helps lead the Chicago VMUG group. This blog Virtualize Tips was started to document and remember things that I come across while working with tech.
Mail | Web | Twitter | LinkedIn | More Posts (169)
Posted by Brian on Mar 9, 2011 in View, VMware | 0 comments
After months of vGeeks waiting for some View love for their lonely iPads VMware has finally delivered the almighty View Client. Only time will tell how well it will work and how much people will use it. But based upon the number of iPads that Apple has sold and just the amount that I see in offices these days it should be very successful.
The VMware View iPad client is available now from the iTunes app store here.
The VMware View client for iPad supports the native Apple iPad gestures as well as some new VMware created ones. You can see from the images below the virtual trackpad that is available and some of the gestures. Also the VMware communities has posted a document to cover install, setup and troubleshooting of the iPad App here.
Brian is a Technical Architect for a VMware partner and owner of this website. He is active in the VMware community and is helps lead the Chicago VMUG group. This blog Virtualize Tips was started to document and remember things that I come across while working with tech.
Mail | Web | Twitter | LinkedIn | More Posts (169)
Posted by Brian on Mar 8, 2011 in VMware | 2 comments
VMware Forum 2011 is a free, interactive, full-day event where you will learn about accelerating IT, so that your business can respond more effectively to markets, competitors and customers. Hear VMware, industry analysts and IT professionals discuss virtualization and how it helps organizations to reduce capital and operating expenses, improve agility, ensure business continuity, strengthen security and go green. You will also hear how to enable “your cloud” to meet your company’s specific business needs, while dramatically lowering costs and enabling a flexible, agile IT service delivery model. Engage with speakers from companies of all sizes as they share best practices for preserving existing investments as they move to virtualization and develop a cohesive, secure and compliant cloud strategy in three core areas: infrastructure, applications and end-user computing.
VMware Forum 2011 Cities
Washington, DC – May 3
New York – May 11
Anaheim – May 19
Atlanta – June 2
Houston – June 8
Chicago – June 15
Toronto – June 23
Minneapolis – July 21
Check out this LINK to register and get further details.
Sample of the Agenda for the day.
Brian is a Technical Architect for a VMware partner and owner of this website. He is active in the VMware community and is helps lead the Chicago VMUG group. This blog Virtualize Tips was started to document and remember things that I come across while working with tech.
Mail | Web | Twitter | LinkedIn | More Posts (169)
Posted by mike on Mar 5, 2011 in vSphere | 1 comment
ESXi behavior with NIC trunking
Sometimes very challenging problems will arise. Things that make you scratch your head, want to hurl your coffee cup, or just have a nice cold adult beverage. Customers can change a projects requirements mid-way through, a vendor’s storage array code upgrade can go awry, or a two can creep into the ones and zeros.
In this section, we present examples of those crazy situations with the hopes of helping out our fellow engineers in the field before they become as frustrated as we have!
Recently in working with a customer, the request was for a new cluster comprised of ESXi 4.1 hosts. They would be using just two onboard NICs for the vmkernel and virtual machine traffic. These two NICs would feed into a pair of Cisco Nexus 5020’s, using virtual port channel (VPC).
Because of the use of VPC, the virtual switch load balancing needs to be set to IP Hash for the NIC teaming policy. Okay, no sweat! After installing ESXi and completing the initial configuration on the hosts, it was time to add the second NIC to vSwitch0 and plug it in. (Note this configuration is all being done on the hosts directly as no vCenter server has been built yet). After adding the second adapter to the active section of vSwitch0, and changing the NIC teaming policy to IP hash, we plugged in the second cable.
The host immediately lost connection from our vSphere client, and dropped off entirely from being able to be contacted. No ping, no nothing! This was most puzzling indeed: we unplugged the second cable and the host started to ping again. We thought maybe there was something wrong with the NIC itself, and so setup a separate NIC to take its place. This had the same result, and we then thought to look at the switch. After discussing the current configuration with the network engineer, we felt that his configuration was correct. The configuration (and more!) can be found in the white paper put out by Cisco and VMware: “Deploying 10 Gigabit Ethernet on VMware vSphere 4.0 with Cisco Nexus 1000V and VMware vNetwork Standard and Distributed Switches – Version 1.0” This doc has been a very helpful during the implementation of this project.
So! With the network being deemed not the problem and wearing a sheepish smile on my face after the network guy commented “it’s always the network isn’t it?” I returned to the host. I then tried setting up both NICs on a non-nexus switch that is being used for out of band management, and they worked just fine using virtual port id for NIC teaming. So at that point, I fired up the googalizer and did some checking. I came across this KB article from VMware:
VMware KB 1022751: NIC teaming using EtherChannel leads to intermittent network connectivity in ESXi
Details:
When trying to team NICs using EtherChannel, the network connectivity is disrupted on an ESXi host. This issue occurs because NIC teaming properties do not propagate to the Management Network portgroup in ESXi.
When you configure the ESXi host for NIC teaming by setting the Load Balancing to Route based on IP hash, this configuration is not propagated to Management Network portgroup.
So, based on this very helpful information, I followed the instructions listed in the KB and had great success. Now my ESXi hosts are talking on both NICs via IP Hash and life is good.
read more
Posted by Brian on Mar 2, 2011 in VMware, vSphere, vStorage API | 0 comments
With the release of vSphere 4.1 last summer VMware customers were provided several new features. Many of these new features were created to lower the burden on the ESX host by being more efficient or offloading the work to something outside of the Virtualization stack. The overall goal of the new features was to continue to improve performance of virtual machines. The one that I am writing about today is VAAI or vStorage API for Array Integration. I wanted to write about how using VAAI in your Architecture Designs is changing the way you are creating environments.
The goal of VAAI is too offload some of the storage focused activities that VMware had previously handled to your storage array. This was accomplished by VMware working closely with the major storage vendors. The idea of VAAI was first announced back at VMworld 2008 and finally came to market when vSphere 4.1 was released. By offloading these storage functions it has reduced the load on the ESX(i) hosts and also increased the performance of these activities by letting the storage array do the work that it was created to do.
In the current offering of VAAI there are 3 functions that have been offloaded. In future product releases it is expected that VMware will continue to work with storage vendors to increase the features of VAAI and the currently available features are explained below.
Full Copy – So you’re probably wondering how this feature is going to help me. I can think to two VMware functions that this VAAI feature provides upwards of 10x speed improvements in. The first would be when you are deploying a VM from a template. We will say for example that you are going to deploy a 50 GB VB. When the VM is deployed vSphere is going to read the entire 50 GB and then write the 50 GB for a total of 100 GB of I/O. With VAAI enabled and a storage array that supports VAAI this process creates very little I/O at the host. The vSphere host will send a command to the storage array that say make a copy of this VM and name it this. The copy will be done locally on the storage array and results in very little I/O between the host and array. Once completed the array will send a notice to the host to let it know the works was completed.
The second VMware feature to benefit from this is a Storage vMotion. I feel that this is where this really pays off because you are most likely moving a larger chunk of data with this command. For example sake let’s say we are going to move a 100 GB Virtual Machine from one disk to another. To do this in the past this would have caused 200 GB of I/O on the host. With VAAI the burden on the host is almost nothing as this work is done on the storage array.
Hardware assisted locking – Too allow multiple hosts in your cluster to talk to the same storage volume VMware would lock the volume when one of the VM’s needed to write to it. This locking is to prevent another host from trying to write to the same blocks. This was not a large issue If you were using smaller volumes with only a handful of virtual machines on them. Now with VAAI the locking has been offloaded to the storage array and it’s now possible to lock only the blocks that are being written to. This opens up the possibility to use larger volumes and increase the amount of VM’s that can be run on a single volume.
Block Zeroing – This feature is saving vSphere from having to send redundant commands to the array for writes. The host can simple tell the storage array which blocks are zeros and move on. The storage device will handle the work without needing to receive repetitive write commands from the host.
So now that you should have an understanding of what VAAI is and how it should help free up resources. I will now talk about how this changes the way we should be thinking about different design considerations.
The first thing that comes to mind is that I can now think about using larger datastores without the worry of affecting performance due to locking issues. With VAAI the storage device is going to handle the locking and allow me to have far for VM’s per volume than the 5-10 previously held as a guideline to live by in past versions. It’s now possible to have 50+ VM’s on a single volume if you had a valid reason to.
The next thing that came to mind is that I will be able to achieve higher consolidation ratios on vSphere hosts if needed. Due to the savings in CPU, Network and Storage I/O overhead we can use that savings to host more virtual machines on each host. In particular if you are using Blade Chassis you can expect to see a lot of network I/O traffic savings since you can have up to 16 blades in a chassis depending on your vendor. That can equal to a huge decrease in traffic flowing through those shared ports.
Something that I was wondering about and saw little discussion about was what type of extra load does VAAI functions place on the array. I reached out and asked this question to Chad Sakac of EMC and Vaughn Stewart of NetApp. Both contacts replied back via Twitter and stated that currently VAAI adds little to no extra burden to the arrays and in coming firmware updates it’s expected to be even less.
Lastly to sum up what you need to take advantage of VAAI. You will need to have vSphere 4.1 and you need to be licensed for Enterprise or Enterprise Plus. Next you must have a storage array that supports VAAI, this is probably going to be the largest hurdle for most. If you array was purchased within that last 2 years there is a good change that with a firmware upgrade your array may support VAAI. If not you will need to purchase a new one and this is an expensive investment. So it’s conceivable that many smaller shops will never get to reap the benefits of VAAI because of these requirements.
Brian is a Technical Architect for a VMware partner and owner of this website. He is active in the VMware community and is helps lead the Chicago VMUG group. This blog Virtualize Tips was started to document and remember things that I come across while working with tech.
Mail | Web | Twitter | LinkedIn | More Posts (169)
Posted by mike on Mar 1, 2011 in HP, vCenter Server, VMware, vSphere | 1 comment
Here at 64k, in our smaller cube near the vending machines, we storage-oriented folks like to mull over ideas big and small, 4k at a time. We also deal in a great number of puns, so consider yourself warned. Today, in our maiden voyage, I’d like to talk about some of my experience with HP’s line of EVA storage arrays. As many of our readers know, the EVA line is a middle tier offering from HP. Though likely to be usurped in the near future by 3PAR’s goodies, I am not here to begin that debate. Rather, let us delve into a few common gotcha’s that can be overlooked in environments where EVAs live.
ONE]
The tight rope act begins with the storage array, our bright and shiny EVA. At a fundamental level, an EVA is comprised of two controllers. The operating environment of the EVA is such that it can, in a semi-intelligent fashion, manage vdisk ownership between the two controllers itself. By default, vdisks are set to no preference for a failover/mode setting at the time of creation. This means the EVA will decide which controllers get which vdisks when it (the EVA itself) boots. Every vdisk is assigned to a controller (and only one controller). If the non-owning controller is receiving the IO for a server(s) talking to a vdisk, it will after a period of time change the ownership of the vdisk. This will reduce the load crossing the mirror ports. While the EVA can run in this fashion, it is sub-optimal.
The other side of the tight rope of this balancing act is the hosts. IO can walk many paths from host to array, some optimal and others not. The start of such begins at the host’s adapter. If it is a dual port (or multiple single port) host, then you have all the more paths to choose from. Even in the case of a single port host, you can still cover multiple paths to arrive at the vdisk. The handling of the proper path comes in the form of multipathing software. From HP for Microsoft operating systems, we have Device Specific Module (DSM), which uses MS’s MPIO stack as its basis. HP makes specific DSM’s for each of its line of arrays. Without the MPIO stack, the host will see a drive presented once for each host port. In an 8×00 series array, that is 8! So clearly the MPIO software and HP’s DSM is needed for correct operation. The default install does not enable Adaptive Load Balance (ALB). This hampers read operations by not passing through the correct controller for a vdisk. Note that non-MS based operating systems (like VMware) have their own multipathing stacks. In the case of VMware ESX(i) 3.x, the options are fixed and mru. In the case of vSphere, we get round robin added to the mix. In pre-vSphere environments, the fixed path does not by default balance load across the host ports. You can end up with all your VM traffic running over one host port! Yikes!
TWO]
Now, to balance things out, let me start with the array. A good habit to get into involves understanding your environment from an IO perspective. You need to understand the profile, or workload, of your IO, so that you can balance between the controllers (among other things!). Make sure to capture your performance data using evaperf (or other tools) to allow you the view of your controller’s current load. As you add new vdisks, you can balance them by setting the failover/mode setting to the controller with failover + failback. This will allow the balancing to remain should you lose and regain a controller. Further, this specifies the controller for the vdisk in terms of mastership. This helps from the host side as the controller it needs to talk through is clearly defined. One thing to keep in mind also is the need to accept all load on one controller should failure occur. This should be something you are aware of via your performance data. A good rule of thumb is a controller should be no more than 30% ideally (at least in my experience). And as always, have the latest Command View and XCS code. One other thing to check for balance is to make sure the host ports are set to their top speed (4GB, except the very old EVA models) as well as properly balanced on the fabric (equal ports on both sides). One customer I came across had all ports from controller A on fabric A and all ports of controller B on fabric B! Definitely a big problem there!
For the host side, there is a bit more that can be done. There is some work to be done on the array as well, which I will address. The hosts should have the latest firmware, drivers, and software for their HBAs. Additionally, make sure you have the latest HP DSM software. Within the DSM software, you will want to enable Automatic Load Balancing. As I stated before, this is not enabled by default. To enable, just right click on each LUN (listed by WWN) that is listed and choose Enable ALB.
So, as a quick explanation: write requests from hosts will hit the controller that owns the vdisk in question, but that write will propagate over the mirror link into both controllers’ cache. This is in case a controller is lost, the write can still be committed. Read requests will hit whichever controller, and if it is the wrong controller, will have to travel over the mirror ports to the correct controller. This is sub-optimal, but is alleviated by enabling ALB. ALB communicates with the array and will always communicate its read requests through the owning controller. Very handy!
Now, from a VMware standpoint, let’s talk about fixed and then round robin (two most common multipathing situations found today). For Fixed, you will need to balance IO to your datastores over the host ports of the controllers. Also keep in mind which controller you selected at the array. As an example, if I have 8 datastores of average IO (no virtualized heavy apps) then I would want 4 datastores on each controller. To further balance, I would have each datastore talking over one of the host ports for each of the controllers (4 ports per controller x 2 controllers). The IO is evenly balanced. To set this, simply go into each datastore properties (via the VI Client) and pick the WWN for the corresponding host port). Under heavy IO circumstances, you may not be able to move your traffic to a different host port. Just try again at a later date. When it comes to round robin, the IO works a bit differently. Round Robin will send IO to each host port in turn after a certain amount of IOPS. In the HP best practices for vSphere on the EVA, it states to change this value to 1 (and thus pushing even IOPS over every host port visible). There was a bug which would, after a reboot of the ESX(i) host, reset this to a very high number. I have found in my experience that leaving it as-is seems to work fairly well. I would guess there is good reason that HP came up with that figure, and so at this point, with vSphere 4.1, I would suspect you could set this without issue.
Summary
Presented here are some of the findings I have come across in working with different customers. I figure that having these kinds of storage discussions can help to make for a very engaging conversation. Let me know what you think (and if I make any errors, which being human, am prone to!
read more
