Posted by Brian on Jun 3, 2010 in Troubleshooting, VMware, vCenter Server | View Comments
While I was upgrading my vCenter server to Server 2008 I came across the following issue. I was trying to install VMware License Server on a Window Server 2008 32bit server and the install would fail saying it cannot start system services. After checking to make sure that I had not made some dumb mistake with my account I hit the intertubes and found that others had the same issue.
I was able to get pass this issue by downloading the latest copy of Virtual Center server 2.5 update 6 and then extracting the License server install file from the VPX folder. Seems that the copy from the install .iso file is newer than the file available for direct downloading. Maybe VMware can fix this soon. I did not try it but according to some other posts the version in VC 2.5 U5 might also work. Once I installed this version it worked the first time.
Error 1920.Service VMware License Server failed to start. Verify that you have sufficient privileges to start system services.
read more
Posted by Brian on Jun 2, 2010 in Troubleshooting, VMware | View Comments
Whether your tracking down an issue on your own or collection data to submit a support request to VMware your gonna need to round up the necessary log files. I have collected and listed some of the main log locations from VMware and also linked to KB articles covering a full list of log file locations.
VMware ESX log files
- From the Service Console
- From the vSphere Client connected directly to the ESX host (click Home > Administration > System Logs)
- From the VMware Infrastructure Client connected directly to the ESX host (click Administration > System Logs)
The vmkernel logs (which log everything related to the kernel/core of the ESX) are located at /var/log/vmkernel.
The vmkwarning logs (which log warnings from the vmkernel) are located at /var/log/vmkwarning.
The vmksummary logs (which provide a summary of system activities such as uptime, downtime, reasons for downtime) are located at /var/log/vmksummary.
The hostd log (which is the log of the ESX management service of the ESX) are located at /var/log/vmware/hostd.log.
The messages log (which log activity on the Service Console operating system) is located at /var/log/messages.
The VirtualCenter Agent log is located at /var/log/vmware/vmware/vpx/vpxa.log.
The Automatic Availability Manager (AAM) logs are located at /var/log/vmware/aam/vmware_<hostname>-xxx.log.
The SW iSCSI logs are located at /var/log/vmkiscsid.log.
The System boot log is located at /var/log/boot-logs/sysboot.log.
The vmkernel, vmkwarning, and hostd logs are located at /var/log/messages.
The Host Management service (hostd = Host daemon) log is located at /var/log/vmware/hostd.log\.
The VirtualCenter Agent log is located at /var/log/vmware/vmware/vpx/vpxa.log.
The System boot log is located at /var/log/sysboot.log.
The Automatic Availability Manager (AAM) logs are located at /var/log/vmware/aam/vmware_<hostname>-xxx.log.
vCenter log files – (KB Article)
The SRM configuration files are located at:
- C:\Program Files\VMware\VMware Site Recovery Manager\config\extension.xml
- C:\Program Files\VMware\VMware Site Recovery Manager\config\vmware-dr.xmlOr
- C:\Program Files\VMware\VMware vCenter Site Recovery Manager\config\extension.xml
- C:\Program Files\VMware\VMware vCenter Site Recovery Manager\config\vmware-dr.xml
The SRM Logs (on vCenter Server for connection with SRM and on SRM for SRM workflow) are located at:
- %ALLUSERSPROFILE%\VMware\VMware Site Recovery Manager\Logs, which translates by default to C:\Documents and Settings\All Users\Application Data\VMware\VMware Site Recovery Manager\LogsOr
- %ALLUSERSPROFILE%\VMware\VMware vCenter Site Recovery Manager\Logs, which translates by default to C:\Documents and Settings\All Users\Application Data\VMware\VMware vCenter Site Recovery Manager\Logs
The SRM Installation Logs (on the SRM Server, which may not be the vCenter Server) are located at C:\Documents and Settings\Administrator\Local Settings\Temp\1.
The location of the SRA Logs (on the SRM server) depends on the SRA type and vendor. They may be located in:
- C:\Program Files\VMware\VMware vCenter Site Recovery Manager\scripts\SAN\*\logOr
- C:\Program Files\<SRA Vendor or Name>\
read more
Posted by Brian on May 28, 2010 in Troubleshooting, VMware | View Comments
I found a recent VMware KB article about FT and some common reasons that it might fail to start. The article goes on to explain some reasons why FT options might be greyed out, disabled or you cannot turn on Fault Tolerance. It also talks about requirements that must be met to enable FT. See the full article here.
VMware article that explains Fault Tolerance and what it can do. VMware KB
read more
Posted by Brian on May 24, 2010 in Troubleshooting, VMware | View Comments
There might be a few reasons that you would need to do this. But if you need to locate the Serial Number of server or Service Tag of your Dell server you can do this from the service console command line. In the past I have needed this to schedule service and also to confirm the identity of a server for the Vendor that was on site. In case you do not have a database to reference or maybe someone mistyped the digits you can always fall back to this method.
Type the following command from the command line on the service console and you will get some Vendor details and serial number information.
[root@host name]# /usr/sbin/dmidecode |grep -A4 “System Information”
read more
Posted by Brian on May 11, 2010 in Knowledge Base, Troubleshooting, VMware, vCenter Server, vSphere | View Comments
Recently while setting up a new cluster on vSphere I had an issue with adding one of the hosts to the cluster. It would fail the HA configuration piece each time I would try. The host would join the cluster but HA would have a Red alarm for its failure. Nothing seemed to be wrong with the host, hardware or configuration. I would get the error listed below on each attempt.
cmd addnode failed for primary node: Internal AAM Error agent could not start
I found the following VMware KB article to help troubleshoot these types of errors. My issue ended up being an issue with the cluster that was created. I created a new cluster and moved my ESX Hosts and Virtual Machines over to it and the issue was gone. Before trying this route I had examined several of the options listed in the KB article.
read more
Posted by Brian on Apr 9, 2010 in Troubleshooting, VMware | View Comments
This is something that I have to do on Windows boxes all the time, but less on our ESX boxes. In the past we used to just run the vm-support to collect the support logs and turn those over to the storage team. They no longer are happy with those and since EMC now has a grab that supports both vSphere and VI3.5 it’s hard to deny them now. In case you do not know what a grab is, it’s a log collection utility that will provide the storage admin with all of the details about WWN’s, paths and which LUN’s a host can see. They can use this for planning upgrades and troubleshooting issues.
First thing you will need to do is to download the proper EMC grab version to support your hosts. At the time of writing this its a version 1.2.1 and is supports both ESX 3 and 4. Proceed on over to http://powerlink.emc.com and download it from the support programs area.
Once you have the file you will need to upload it to your host with something like WinSCP. I always upload it into the /tmp folder and unzip it there. It will create a folder called “emcgrab”. You can use the following command to do the unzip in case your not familiar with what to do.
tar -xvf emcgrab_ESX_vSphere_v.1.2.1.tar
Next thing is to move into the emcgrab directory that was created. From within there you will need to execute the following command. If you read the help file included in the directory it will explain some options to supress some annoying confirmation screens about the licensing and such.
./emcgrab.sh -nomsg
Once the program starts to run depending on the options you used it will prompt you to confirm and read the licensing. After you pass that part it will ask you a string of questions about your contact details and some questions about your environment. These are not necessary to complete it you are using these grabs in house. If you plan on sending these to EMC then I would advise to fill them out.
Once the script completes it will ask you if you want to run vm-support to collect the VMware support logs along with the EMCgrabs. This is up to you, if you have a need for them go ahead. Once the script finished it will place the zipped up file in the Output folder and you can pull off the file with WinSCP.
In closing it’s not necessary to be running Powerpath on your hosts to collect these grabs.
read more
