Virtualization Tips » ESX

VMware vSphere and security scan false positives on OpenSSH vulnerabilities

Brian — Thu, 09 Sep 2010 00:40:04 +0000

This is something that we get on a regular basis from the security team. When doing their regular security scans for compliance and vulnerabilities I always get a long list of ESX hosts. The scans normally come back and complain about an OpenSSH x11 vulnerability or an OpenSSH Memory and Buffer Overflow.

These seem to be False positives from the tool being used to scan the hosts. We always make sure that we have installed the necessary updates related to OpenSSH as VMware releases them. But the tool always comes back with these issues. It seems to stem from the fact that the tool looks at OpenSSH in generic terms and assumes that all vendors implement it in the same way. From the documents listed below VMware indicates that since ESX 3.x VMware no longer included the x11 packages with their products. I would recommend that you make sure you are up to date on your patches and if the scans still come back dirty that you should discuss this results with the Application vendor that created the scanning tool. You might find out that this is common and they are just false positives.

Links:

VMware ESX Server and Security Issues in OpenSSH

Security Response: SSH Version Installed with ESX Server May Be Vulnerable

About Brian

Brian is a Technical Architect for a VMware partner and owner of this website. He is active in the VMware community and is helps lead the Chicago VMUG group. This blog Virtualize Tips was started to document and remember things that I come across while working with tech.

Mail | Web | Twitter | LinkedIn | More Posts (169)

What is all included in the VMware vShield Family of products

Brian — Tue, 31 Aug 2010 18:37:01 +0000

Today at VMworld 2010 VMware announces the new family of vShield products. The new products to this family are vShield Endpoint, vShield App and vShield Edge. Each product has been designed for a few core functions that are helping to facilitate and secure the IT as a Service model that VMware is promoting with its new vCloud Director solution. These security related products are going to secure, make management easier and help move down the patch to a cloud infrastructure.

I will try and provide some more details about each product below gathering any details that are available as of today. In the image below are some of the concerns that VMware is address based on what Enterprises have been telling them.

vShield Endpoint – vShield Endpoint provides on-host antivirus and malware protection that reduces performance latency and eliminates the need to maintain individual security agents in each and every virtual machine, helping to simplify security administration while minimizing the risk of malware infections. Datasheet

vShield App -VMware vShield App protects applications in the virtual datacenter from network-based threats. vShield App gives organizations the ability to create and manage business-relevant policies that adapt to dynamic cloud environments. It also provides deep visibility into network communications between virtual machines and granular enforcement through security groups. Datasheet

vShield Edge – vShield Edge is a network gateway solution that protects the edges of the virtual datacenter with DCHP, network address translation (NAT), firewalling, load balancing, site-to-site VPN, port group isolation and other capabilities that help organizations maintain proper segmentation between different organizational units. Datasheet

vShield Manager – Included with all vShield products, vShield Manager provides a central point of control for managing, deploying, reporting, logging and integrating third-party security services. Working in conjunction with vCenter Server, vShield Manager also enables role-based access control and administrative delegation as part of a unified framework for managing virtualization security.

vShield Zones – VMware vShield Zones, included with vSphere, provides basic protection from network-based threats in virtual datacenters, with application firewalling and policy management based on administrator-defined zones, using basic traffic information such as the source IP address, the destination port, and so on.

Here is a quote from a VMware product release.

Enterprise Partner Extranets – vShield lets enterprises extend their networks and application resources to branch offices, home offices and business partner sites through site-to-site VPN services that offer simplified provisioning, streamline administrative tasks and improve scalability. All traffic between sites is encrypted using IPsec to maintain the confidentiality and integrity of all site-to-site communications.

vShield Product Family Brochure

About Brian

Mail | Web | Twitter | LinkedIn | More Posts (169)

VMware best practices for virtual machine snapshots

Brian — Tue, 03 Aug 2010 15:32:54 +0000

I will start this post off with the standard snapshot warning. Just a reminder that Snapshots are not backups, they are only a change log of the original virtual disk. You should not count on them as a backup. There are a number of different reasons that you might use a snapshot for. One of my most used reasons would be for a software upgrade I would use the snapshot to allow for an easy rollback to the machine state prior to the upgrade. If you have some other reasons leave a comment to share with others.

The maximum supported amount in a chain is 32. However, VMware recommends that you use only 2-3 snapshots in a chain.
Use no single snapshot for more than 24-72 hours.
- This prevents snapshots from growing so large as to cause issues when deleting/committing them to the original virtual machine disks. Take the snapshot, make the changes to the virtual machine, and delete/commit the snapshot as soon as you have verified the proper working state of the virtual machine.
- Be especially diligent with snapshot use on high-transaction virtual machines such as email and database servers. These snapshots can very quickly grow in size, filling datastore space. Commit snapshots on these virtual machines as soon as you have verified the proper working state of the process you are testing.|
If using a third party product that takes advantage of snapshots (such as virtual machine backup software), regularly monitor systems configured for backups to ensure that no snapshots remain active for extensive periods of time.
- Snapshots should only be present for the duration of the backup process.
- Snapshots taken by third party software (called via API) may not show up in the vCenter Snapshot Manager. Routinely check for snapshots via the command-line.
An excessive number of snapshots in a chain or snapshots large in size may cause decreased virtual machine and host performance.

You can find some more details from VMware on troubleshooting snapshots here.

About Brian

Mail | Web | Twitter | LinkedIn | More Posts (169)

Use web browser to view vSphere Configuration and Log files

Brian — Thu, 29 Jul 2010 18:57:52 +0000

Sure this nothing earth shattering but it’s just something simple that can make your life easier. With a web browser and some links that I will provide below you can view some of the vSphere configuration files and messages from logs. This is probably the fastest way to get a view into your host with out having to SSH into the server or use another method. This method works for both vSphere 4.0 and 4.1 hosts and it works on both ESX and ESXi hosts.

You can view the VMware vSphere Configuration files from a browser using a link formatted like the following. https://hostname/host From that link you will need to authenticate to your host and then will be able to view a list of files from the host. In the list of files presented with be configuration files and some logs.

There is another page viewable with a web browser that will show you log messages from your ESX or ESXi host. Use the following syntax for the link. https://hostname/host/messages

About Brian

Mail | Web | Twitter | LinkedIn | More Posts (169)

Chicago VMware VMUG recap and notes

Brian — Thu, 29 Jul 2010 03:10:19 +0000

I would like to say a Big Thanks to everyone that attended today and a special thanks to our sponsor Compellent. Compellent gave a nice presentation about their technology and what they have accomplished in their 5 years since they released the first product. I will give a short break down of the different presentations today and if we are able to get the slide decks from the presenters we will publish them over on the Chicago VMUG blog.

In our first presentation of the day Chris Fox of VMware was in and gave an overview of the new features of vSphere 4.1. There were discussions about SIOC (Storage IO Control), NIOC (Network IO Control), VAAI the API’s for array integration were some of the most talked about features. It was also discussed that VMware ESX 4.1 classic will be the last release of the ESX flavor of Hypervisor. Sometime in 2011 there is expected to be the next major release of vSphere and it will only be available in ESXi flavor.

In the second session of the day Russ Taddiken of Compellent talked to us about their storage virtualization technology. Russ gave a presentation that explained many of the features that make Compellent a strong competitor in the storage market. He spoke about Storage Auto Tiering that has been a feature in their product for about 5 years. Some of the other points that stood out to me was CoPilot their support organization and the Portable Volume feature. With portable volume it allows for the initial data replication to be placed on an encrypted USB disk that can be shipped to a remote site that might have a slow link. You will then only have to replicate the changes rather then the entire amount. Russ also mentioned that Compellent will be in the 2nd round of Vendors that will be supporting VMware VAAI API for storage functions.

In the last session of the day Mark from VMware spent time to talk about migrating your ESX infrastructure to ESXi. He covered the different ways to convert your hosts over to VMware ESXi. There was discussion around some of the reasons for the VMware’s decision to move in the ESXi direction. An estimated 80% of patches that VMware released for the ESX classic version were related to the console (COS) due to it’s Linux base that it was built on. With ESXi the COS was removed and the amount of patching required is greatly reduced. VMware is also working in the direction of building the ability to have a stateless hypervisor. Mark spent some time showing some of the commands that are the vCLI versions of the console commands that many are used to using.

We had a pretty nice showing for this meeting and hope that our community continues to grow. We had a couple of higher profile members from the VMware community show up to the meeting. David Davis from Train Signal was in attendance at the meeting. David has created a large number for training videos from Train Signal as well as for his blog VMwareVideos.com. Thanks again to David and the Train Signal team for providing several copies of their VMware vSphere training videos that we were able to give away to our members. Also in attendance today was Justin Lauer of EMC and a vSpecialist from Chad Sakac’s vArmy Team. I’ve knowing Justin for a bit now and it’s always great to chat with him, his involvement in our VMUG community will help many.

Update: We have posted a few of the slide decks from the presentations today here.

Took a couple of quick photos with an iPhone today as I forgot my camera but will do a better job in the future.

About Brian

Mail | Web | Twitter | LinkedIn | More Posts (169)

Different ways to enable Tech Support Mode TSM on ESXi 4.1

Brian — Thu, 15 Jul 2010 02:59:14 +0000

For anyone that has been running or played with ESXi in the previous versions you should have a good idea of what Tech Support Mode is. The Tech Support Mode or TSM is a sort of simple version of the system console that was available on the classic versions of ESX. Except that the TSM mode is not Linux based and does not have all the capabilities that the old COS had. But you can now access Tech Support Mode locally or via SSH if you follow the instructions below to enable them. I have become very comfortable with the old console access and that’s probably my biggest complaint about having to use ESXi. I’ve been playing around with the vMA or virtual management appliance that can be used to remotely manage ESXi hosts in the lab but its just not the same. I guess it will become second nature the more that I use it since classic ESX will no longer be offered in ESX 5.0 when it is released in the future.

To enable local or remote TSM from the Direct Console User Interface (DCUI):

At the DCUI of the ESXi host, press F2 and provide credentials when prompted.
Scroll to Troubleshooting Options, and press Enter.
If you want to enable local TSM, select Local Tech Support and press Enter once. This allows users to login on the virtual console of the ESXi host.
If you want to enable remote TSM, select Remote Tech Support (SSH) and press Enter once. This allows users to login via SSH on the virtual console of the ESXi host.
Optionally, if you want to configure the timeout for TSM:
1. Select Modify Tech Support timeout and press Enter.
2. Enter the desired timeout value in minutes and press Enter.
Press Esc three times to return to the main DCUI screen.

To enable local or remote TSM from the vSphere Client:

Select the host and click the Configuration tab.
Click Security profile > Properties.
Click Local Tech Support or Remote Tech Support (SSH) and click Options.
Choose the desired startup policy and click Start, then click OK.
Verify that the daemon selected in step 3 shows as running in the Services Properties window.

To configure the TSM timeout value using the vSphere Client:

Select the host and click the Configuration tab.
Click Advanced Settings.
Change the UserVars.TSMTimeOut field to the desired value in minutes.
Click OK.

To access the local TSM:

At the main DCUI screen, press ALT+F1 simultaneously. This opens a virtual console window to the host.
Provide credentials when prompted.
Note: When typing the password, characters are not displayed on the console.

To access the remote TSM:

Open an SSH client.
Specify the IP address or domain name of the ESX host.
Notes:
- Directions may vary depending on what SSH client you are using. For more information, consult vendor documentation and support.
- By default, SSH works on TCP port 22.
Provide credentials when prompted.

About Brian

Mail | Web | Twitter | LinkedIn | More Posts (169)

Install vSphere ESX 4.0 with EDA Deployment appliance

Brian — Tue, 29 Jun 2010 03:50:16 +0000

I’ve seen a few other guides for this same type of install. I just wanted to put something together to save for myself. I will show you how to use the EDA deployment appliance to install vSphere in VMware Workstation. You can follow most of the same steps for a physical machine install with just a few changes.

For this I was testing things for my home lab and created this on my laptop just using Workstation 7.0. I created two virtual machines, one for the EDA appliance and one blank that will get ESX 4.0 installed from the Deployment script.

Step 1: Download the EDA appliance http://www.vmware.com/appliances/directory/va/89313/download

Step 2: Import the EDA virtual appliance into VMware workstation.

Step 3: After the input I had to add in a CD-Rom to the VM and attached the vSphere .iso file to the CD drive.

Step 4: You can then start up the EDA appliance and choose Reconfigure.

Step 5: Fill out your network information on the screen shown below. Then press enter or click on OK.

Step 6: You can now log into the configuration web page for the EDA appliance. In my configuration it was http://192.168.1.19
The default login for the page is root - root. You will be presented with several sections that will allow you to configure options for the install. I will briefly cover these parts as I did not change much on them for this test. The Post installation commands section allows you to script any custom changes or adds that you want to make for your install. The General settings section is used for basic network and disk settings for each install, this section is pretty easy to understand.

Step 7: Now you are ready to prepare the Host specific details. Refer to the image below as I describe the steps. You can either modify the sample host that is loaded by default. By clicking on the host name or selecting “new host” you will get a screen like the 2nd picture below. This will allow you to specify host name and IP addresses needed for your base install. Once created if you click on the “ks” link for the hostname you will see what the kick start file contains for commands.

Step 8: Now you will need to setup the options for the DHCP server. Upon clicking on the “configure DHCP server” link from the top menu you will be presented with the following configuration file. I highlighted a few of the important lines that I changed to work for my setup. You will need to make sure all of the normal DHCP options are set in this file.

Step 9: Now you will need to “import PXE files” so that you can boot the server that you will be installing ESX onto. You need to make sure that the ESX CD or .iso file is mounted to your EDA appliance. If the ESX disc or file is properly mounted you will see the confirmation in the top right just like the image below. Then you select the “Import PXE bootfiles” from the menu and you will see a screen like the second image below.

Step 10: If you are using a physical server to install onto you can start now and have the machine PXE boot.
If you are going to use a virtual machine you need to create a VM with the proper OS select, disk size, memory and number of Nic’s that you wish to use. Once this is done you will see a screen like the one below you start the install. I trust that you know what settings need to be used for installing ESX into VMware workstation, if not there are plenty of good guides for it. If you are using Workstation 7.x the process is now built in and very easy to select from the OS choices.

Note: If you created multiple host install names you will get presented with the options and can type the name of the one you wish to install. I circled in red above as an example.

About Brian

Mail | Web | Twitter | LinkedIn | More Posts (169)

Location of VMware log files for ESX, ESXi, SRM and vCenter

Brian — Wed, 02 Jun 2010 14:20:11 +0000

Whether your tracking down an issue on your own or collection data to submit a support request to VMware your gonna need to round up the necessary log files. I have collected and listed some of the main log locations from VMware and also linked to KB articles covering a full list of log file locations.

VMware ESX log files

You can see ESX logs: (KB Link)

From the Service Console
From the vSphere Client connected directly to the ESX host (click Home > Administration > System Logs)
From the VMware Infrastructure Client connected directly to the ESX host (click Administration > System Logs)

The vmkernel logs (which log everything related to the kernel/core of the ESX) are located at /var/log/vmkernel.

The vmkwarning logs (which log warnings from the vmkernel) are located at /var/log/vmkwarning.

The vmksummary logs (which provide a summary of system activities such as uptime, downtime, reasons for downtime) are located at /var/log/vmksummary.

The hostd log (which is the log of the ESX management service of the ESX) are located at /var/log/vmware/hostd.log.

The messages log (which log activity on the Service Console operating system) is located at /var/log/messages.

The VirtualCenter Agent log is located at /var/log/vmware/vmware/vpx/vpxa.log.

The Automatic Availability Manager (AAM) logs are located at /var/log/vmware/aam/vmware_-xxx.log.

The SW iSCSI logs are located at /var/log/vmkiscsid.log.

The System boot log is located at /var/log/boot-logs/sysboot.log.

VMware ESXi log files – (KB Article)

The vmkernel, vmkwarning, and hostd logs are located at /var/log/messages.

The Host Management service (hostd = Host daemon) log is located at /var/log/vmware/hostd.log\.

The VirtualCenter Agent log is located at /var/log/vmware/vmware/vpx/vpxa.log.

The System boot log is located at /var/log/sysboot.log.

The Automatic Availability Manager (AAM) logs are located at /var/log/vmware/aam/vmware_-xxx.log.

vCenter log files – (KB Article)

SRM log files – (KB Article)

The SRM configuration files are located at:

C:\Program Files\VMware\VMware Site Recovery Manager\config\extension.xml
C:\Program Files\VMware\VMware Site Recovery Manager\config\vmware-dr.xmlOr
C:\Program Files\VMware\VMware vCenter Site Recovery Manager\config\extension.xml
C:\Program Files\VMware\VMware vCenter Site Recovery Manager\config\vmware-dr.xml

The SRM Logs (on vCenter Server for connection with SRM and on SRM for SRM workflow) are located at:

%ALLUSERSPROFILE%\VMware\VMware Site Recovery Manager\Logs, which translates by default to C:\Documents and Settings\All Users\Application Data\VMware\VMware Site Recovery Manager\LogsOr
%ALLUSERSPROFILE%\VMware\VMware vCenter Site Recovery Manager\Logs, which translates by default to C:\Documents and Settings\All Users\Application Data\VMware\VMware vCenter Site Recovery Manager\Logs

The SRM Installation Logs (on the SRM Server, which may not be the vCenter Server) are located at C:\Documents and Settings\Administrator\Local Settings\Temp\1.

The location of the SRA Logs (on the SRM server) depends on the SRA type and vendor. They may be located in:

C:\Program Files\VMware\VMware vCenter Site Recovery Manager\scripts\SAN\*\logOr
C:\Program Files\\

About Brian

Mail | Web | Twitter | LinkedIn | More Posts (169)

Failed HA configuration when adding vSphere host to cluster

Brian — Tue, 11 May 2010 18:54:28 +0000

Recently while setting up a new cluster on vSphere I had an issue with adding one of the hosts to the cluster. It would fail the HA configuration piece each time I would try. The host would join the cluster but HA would have a Red alarm for its failure. Nothing seemed to be wrong with the host, hardware or configuration. I would get the error listed below on each attempt.

cmd addnode failed for primary node: Internal AAM Error agent could not start

I found the following VMware KB article to help troubleshoot these types of errors. My issue ended up being an issue with the cluster that was created. I created a new cluster and moved my ESX Hosts and Virtual Machines over to it and the issue was gone. Before trying this route I had examined several of the options listed in the KB article.

About Brian

Mail | Web | Twitter | LinkedIn | More Posts (169)

How to enable Maintenance mode on ESX host from command line

Brian — Tue, 11 May 2010 13:48:13 +0000

This is something that I’ve been meaning to post for a while now. It’s a simple thing but can save you from having to open up the vSphere client or allow you to use this in a script. The commands listed below will allow you to place a ESX host in and out of Maintenance mode from the command line or Service Console (what ever you like to call it).

To enter maintenance mode run the following command

vimsh -n -e /hostsvc/maintenance_mode_enter

To exit maintenance mode run the following command

vimsh -n -e /hostsvc/maintenance_mode_exit

So your probably wondering how do I know if the host is in Maintenance Mode, here you go:

vimsh -n -e /hostsvc/runtimeinfo | grep inMaintenanceMode | awk ‘{print $3}’

Update: March 1st 2011

I thought an update to cover the commands you can run from vMA to enter maintenance mode from vCLI. From the vMA command prompt you can enter the following command to enter maintenance mode.

vicfg-hostops -o enter –server 192.168.1.114

To exit maintenance mode from vMA enter the following.

vicfg-hostops -o exit –server 192.168.1.114

You can also issue shutdown or reboot commands from vMA.

vicfg-hostops -o shutdown –server 192.168.1.114
vicfg-hostops -o reboot –server 192.168.1.114

Also note that after entering any of the above commands from the vMA it will make you provide Username and Password for the Host that you are working on. Unless you choose to use Fast Pass or Passthrough authentication.

About Brian

Mail | Web | Twitter | LinkedIn | More Posts (169)