Virtualization Tips » Troubleshooting

Error while installing VMware license server on Windows 2008

Brian — Fri, 04 Jun 2010 03:00:16 +0000

While I was upgrading my vCenter server to Server 2008 I came across the following issue. I was trying to install VMware License Server on a Window Server 2008 32bit server and the install would fail saying it cannot start system services. After checking to make sure that I had not made some dumb mistake with my account I hit the intertubes and found that others had the same issue.

I was able to get pass this issue by downloading the latest copy of Virtual Center server 2.5 update 6 and then extracting the License server install file from the VPX folder. Seems that the copy from the install .iso file is newer than the file available for direct downloading. Maybe VMware can fix this soon. I did not try it but according to some other posts the version in VC 2.5 U5 might also work. Once I installed this version it worked the first time.

Error 1920.Service VMware License Server failed to start. Verify that you have sufficient privileges to start system services.

Location of VMware log files for ESX, ESXi, SRM and vCenter

Brian — Wed, 02 Jun 2010 14:20:11 +0000

Whether your tracking down an issue on your own or collection data to submit a support request to VMware your gonna need to round up the necessary log files. I have collected and listed some of the main log locations from VMware and also linked to KB articles covering a full list of log file locations.

VMware ESX log files

You can see ESX logs: (KB Link)

From the Service Console
From the vSphere Client connected directly to the ESX host (click Home > Administration > System Logs)
From the VMware Infrastructure Client connected directly to the ESX host (click Administration > System Logs)

The vmkernel logs (which log everything related to the kernel/core of the ESX) are located at /var/log/vmkernel.

The vmkwarning logs (which log warnings from the vmkernel) are located at /var/log/vmkwarning.

The vmksummary logs (which provide a summary of system activities such as uptime, downtime, reasons for downtime) are located at /var/log/vmksummary.

The hostd log (which is the log of the ESX management service of the ESX) are located at /var/log/vmware/hostd.log.

The messages log (which log activity on the Service Console operating system) is located at /var/log/messages.

The VirtualCenter Agent log is located at /var/log/vmware/vmware/vpx/vpxa.log.

The Automatic Availability Manager (AAM) logs are located at /var/log/vmware/aam/vmware_-xxx.log.

The SW iSCSI logs are located at /var/log/vmkiscsid.log.

The System boot log is located at /var/log/boot-logs/sysboot.log.

VMware ESXi log files – (KB Article)

The vmkernel, vmkwarning, and hostd logs are located at /var/log/messages.

The Host Management service (hostd = Host daemon) log is located at /var/log/vmware/hostd.log\.

The VirtualCenter Agent log is located at /var/log/vmware/vmware/vpx/vpxa.log.

The System boot log is located at /var/log/sysboot.log.

The Automatic Availability Manager (AAM) logs are located at /var/log/vmware/aam/vmware_-xxx.log.

vCenter log files – (KB Article)

SRM log files – (KB Article)

The SRM configuration files are located at:

C:\Program Files\VMware\VMware Site Recovery Manager\config\extension.xml
C:\Program Files\VMware\VMware Site Recovery Manager\config\vmware-dr.xmlOr
C:\Program Files\VMware\VMware vCenter Site Recovery Manager\config\extension.xml
C:\Program Files\VMware\VMware vCenter Site Recovery Manager\config\vmware-dr.xml

The SRM Logs (on vCenter Server for connection with SRM and on SRM for SRM workflow) are located at:

%ALLUSERSPROFILE%\VMware\VMware Site Recovery Manager\Logs, which translates by default to C:\Documents and Settings\All Users\Application Data\VMware\VMware Site Recovery Manager\LogsOr
%ALLUSERSPROFILE%\VMware\VMware vCenter Site Recovery Manager\Logs, which translates by default to C:\Documents and Settings\All Users\Application Data\VMware\VMware vCenter Site Recovery Manager\Logs

The SRM Installation Logs (on the SRM Server, which may not be the vCenter Server) are located at C:\Documents and Settings\Administrator\Local Settings\Temp\1.

The location of the SRA Logs (on the SRM server) depends on the SRA type and vendor. They may be located in:

C:\Program Files\VMware\VMware vCenter Site Recovery Manager\scripts\SAN\*\logOr
C:\Program Files\\

Having trouble with VMware Fault Tolerance

Brian — Fri, 28 May 2010 18:41:32 +0000

I found a recent VMware KB article about FT and some common reasons that it might fail to start. The article goes on to explain some reasons why FT options might be greyed out, disabled or you cannot turn on Fault Tolerance. It also talks about requirements that must be met to enable FT. See the full article here.

VMware article that explains Fault Tolerance and what it can do. VMware KB

Find the Service tag or Serial number of VMware ESX host

Brian — Mon, 24 May 2010 23:11:06 +0000

There might be a few reasons that you would need to do this. But if you need to locate the Serial Number of server or Service Tag of your Dell server you can do this from the service console command line. In the past I have needed this to schedule service and also to confirm the identity of a server for the Vendor that was on site. In case you do not have a database to reference or maybe someone mistyped the digits you can always fall back to this method.

Type the following command from the command line on the service console and you will get some Vendor details and serial number information.

[root@host name]# /usr/sbin/dmidecode |grep -A4 “System Information”

Failed HA configuration when adding vSphere host to cluster

Brian — Tue, 11 May 2010 18:54:28 +0000

Recently while setting up a new cluster on vSphere I had an issue with adding one of the hosts to the cluster. It would fail the HA configuration piece each time I would try. The host would join the cluster but HA would have a Red alarm for its failure. Nothing seemed to be wrong with the host, hardware or configuration. I would get the error listed below on each attempt.

cmd addnode failed for primary node: Internal AAM Error agent could not start

I found the following VMware KB article to help troubleshoot these types of errors. My issue ended up being an issue with the cluster that was created. I created a new cluster and moved my ESX Hosts and Virtual Machines over to it and the issue was gone. Before trying this route I had examined several of the options listed in the KB article.

How to run EMC grabs on a VMware ESX host

Brian — Sat, 10 Apr 2010 01:29:58 +0000

This is something that I have to do on Windows boxes all the time, but less on our ESX boxes. In the past we used to just run the vm-support to collect the support logs and turn those over to the storage team. They no longer are happy with those and since EMC now has a grab that supports both vSphere and VI3.5 it’s hard to deny them now. In case you do not know what a grab is, it’s a log collection utility that will provide the storage admin with all of the details about WWN’s, paths and which LUN’s a host can see. They can use this for planning upgrades and troubleshooting issues.

First thing you will need to do is to download the proper EMC grab version to support your hosts. At the time of writing this its a version 1.2.1 and is supports both ESX 3 and 4. Proceed on over to http://powerlink.emc.com and download it from the support programs area.

Once you have the file you will need to upload it to your host with something like WinSCP. I always upload it into the /tmp folder and unzip it there. It will create a folder called “emcgrab”. You can use the following command to do the unzip in case your not familiar with what to do.

tar -xvf emcgrab_ESX_vSphere_v.1.2.1.tar

Next thing is to move into the emcgrab directory that was created. From within there you will need to execute the following command. If you read the help file included in the directory it will explain some options to supress some annoying confirmation screens about the licensing and such.

./emcgrab.sh -nomsg

Once the program starts to run depending on the options you used it will prompt you to confirm and read the licensing. After you pass that part it will ask you a string of questions about your contact details and some questions about your environment. These are not necessary to complete it you are using these grabs in house. If you plan on sending these to EMC then I would advise to fill them out.

Once the script completes it will ask you if you want to run vm-support to collect the VMware support logs along with the EMCgrabs. This is up to you, if you have a need for them go ahead. Once the script finished it will place the zipped up file in the Output folder and you can pull off the file with WinSCP.

In closing it’s not necessary to be running Powerpath on your hosts to collect these grabs.

VMware vSphere Health Check Report updated to v4.0

Brian — Fri, 19 Mar 2010 14:23:56 +0000

I hope that most of you have been using or at least tried this script at some point. The script was created by William Lam and is now more user customizable. There is a big list of updates and improvments to this version a few are listed below. To see an example of what the report looks like head over here. The report gives you a ton of information on the status of your Hosts and Virtual Machine status. Be sure to thank William for such a great product and working hard to update it, you can follow William on twitter at @lamw. To download the script head over to VMware communities here.

New Report is now completely modular in which categories to display via a configuration file
New Ability to specify specific ESX/ESXi host to query
New Ability to specify specific Virtual Mchines to query
New vCenter HA Advanced Runtime information
New vCenter HA Configuration (primary/secondary and node states)
New vCenter HA Advanced Configurations
New vCenter DRS Advanced Runtime information
New ESX/ESXi IP/HOSTNAME of vCenter Management IP
New ESX/ESXi Newly improved Hardware and System Health Stuats information
New ESX/ESXi Advanced Configurations
New ESX/ESXi NUMA information
New VM UUID,Bootime,Resource Statistics, Fault Tolerance, Thin provisioned and NPIV information

How to use ESXplot to read your ESXTop reports

Brian — Thu, 04 Mar 2010 16:38:59 +0000

By now everyone should know that esxtop is a great tool that you can use to find out what might be causing performance issues on your ESX host. I don’t plan on writing a post on how to use esxtop. You can refer to Duncan’s ESXtop post for details on thresholds or to VMwares esxtop bible for good base information.

This post is more of a starting point on what ESXplot is and how to install and use. By now most of you should know that ESXplot was born out of the mind of Geoff White of VMware. Geoff is a coworker of Scott Drummonds the well known ESX performance expert. Esxplot is a GUI application that lets you explore the data collected by esxtop in batch mode. The program takes a single command line argument which is the esxtop batch mode output file.

Why would you want to use esxplot? Well in the past your options to examine these files were to use Excel or Perfmon. Both options were very slow, sometimes taking hours to import the file before viewing. With esxplot your dump file will open within seconds usually. For more details you can read Scott’s post on version 1.0.

If you are using Windows to view your files you have 2 options. You can use the Windows binary file or run the python script. If you wish to run the python script you will need the following applications installed.
Python 2.6 or higher but not Python 3.x – Download
wxPython – Download
NumPy is also needed – Download

Once installed the application is very easy to use. First thing to do in capture some data from esxtop in batch mode using the command below. Then copy the file over to your workstation that you will run esxplot on. Open the app and the File ->Import->Dataset option from the menu to view your file. Then from the lower left section that will list your Hosts name open the tree to view all the captured values.

esxtop -b -d 2 -n 100> esxtopcapturefile.csv

You can download esxplot from here.

VMware KB video – Restarting ESX management agents

admin — Thu, 15 Oct 2009 15:15:13 +0000

The knowledge base team over at VMware have created a short video to show you how to restart the mgmt agenst on ESX and ESXi hosts. These commands are good for ESX 3, 3.5 and 4. So if you are someone that likes to see things in a video rather than reading this is a good thing for you. Swing by VMwares site and see the video here.

How to decode VMware PSOD Purple Screen of Death crashes

admin — Wed, 23 Sep 2009 04:12:40 +0000

If you have ever lost a host to a PSOD (Purple screen of Death) you know that they may as well be written in Greek. Sure you used to think that reading a Windows BSOD was a pain, but the VMware PSOD takes this to a new level.

In the past trying to break down what caused your host to crash on your own was a big task. Most folks would require a call to VMware support if the host did not come back up afterwards. But now VMware has since written a nice document on breaking down the many sections of the PSOD screen. This will give you some insight on what might be going on with your ESX host.

You can have a look at the VMware document here.