I have grabbed a list of the new commands added to vCLI 4.1, these command will help narrow the gap that had existed between what you could run on the ESX console (COS) and what you could do via the vCLI with an ESXi host. Notice the part at the end where it lists some of the commands that cannot be executed against a vCenter server for a host in lock down mode.

• vicfg-hostops – Allows you to examine, stop, and reboot hosts and to instruct hosts to enter and exit maintenance mode.
• vicfg-authconfig – Allows you to add an ESX/ESXi host to an Active Directory domain, remove the host, and list Active Directory domain information.
• vicfg-ipsec – Supports IPsec setup.

vSphere CLI 4.1 also includes the following new functionality:

• The following options have been added to esxcli:
  • esxcli swiscsi session – Manage iSCSI sessions.
  • esxcli swiscsi nic – Manage iSCSI network interfaces.
  • esxcli swiscsi vmknic – List VMkernel network interfaces available for binding to particular iSCSI adapter.
  • esxcli swiscsi vmnic – List available uplink adapters for use with a specified iSCSI adapter.
  • esxcli vaai device – Display information about devices claimed by the VMware VAAI (vStorage APIs for Array Integration) Filter Plugin.
  • esxcli corestorage – List devices or plugins. Used in conjunction with hardware acceleration.
  • esxcli network – List active connections or list active ARP table entries.
  • esxcli vms – List and forcibly stop virtual machines that do not respond to normal stop operations.
• Some of the parity issues between vSphere CLI and the ESX service console have been resolved.
• You can now run vCLI commands using SSPI (--passthroughauth) against both vCenter Server and ESX/ESXi systems.
• Lockdown mode allows vSphere administrators to block direct access to ESXi systems. With lockdown mode enabled, all operations must go through a vCenter Server system. The following commands cannot run against vCenter Server systems and can therefore not be used in lockdown mode:
  • vicfg-snmp
  • vifs
  • vicfg-user
  • vicfg-cfgbackup
  • vihostupdate
  • vmkfstools
  • esxcli
  • vicfg-ipsec

If you want to run these commands against an ESXi system, turn off lockdown mode using the vSphere Client.

The release also adds support for mixed chargeback models. VKernel Chargeback 2.0 now supports both allocated and actual resource consumption models. By reporting on both measures, infrastructure teams can chargeback for actual resource consumption, allocated resources, or simply show application teams the difference in real dollars between their allocated capacity and actual usage. This cost visibility is critical to capacity management, reducing VM spral and supporting private cloud initiatives.

Currently only the operating systems listed below are the only 64 bit options support by the new vCenter 4.1 server.

Something new with this release is the vCenter Server Data Migration Tool. This new tool will help to migrate some of your data and settings over to a new server depending on what your configuration was. There are some very rigid rules on what it will and wont move across. The list below is the only options that it will migrate. Judging from that list every production setup I’ve ever seen is going to require the usual manual steps to be done. Since I don’t think anyone out there is using SQL express as their DB in anything other than maybe a small lab environment. Your gonna need to configure 64 bit DSN’s for your vCenter and Update Manger Databases on the new server yourself.

You can use the vCenter Data Migration Tool to automatically migrate the following to a new server:

• vCenter Server Software and its configuration
• vCenter Update Manager Software and its configuration
• VMware Orchestrator Software and its configuration
• The default SQL Express 2005 database that comes installed with vCenter Server.

To see the full details from VMware on this process refer to this article.

A quote from the SRM 4.1 release notes

SRM 4.1 server software can be installed only on 64-bit host platforms. The SRM 4.1 server is incompatible with 32-bit hosts. The SRM 4.1 client plug-in remains compatible with 32-bit hosts. Regardless of the database server you are using, you must install a 32-bit database client (ODBC driver) on the SRM server host. The SRM server is not compatible with 64-bit database clients.

• Scripted Install for ESXi. Scripted installation of ESXi to local and remote disks allows rapid deployment of ESXi to many machines. You can start the scripted installation with a CD-ROM drive or over the network by using PXE booting. You cannot use scripted installation to install ESXi to a USB device.
• vSphere Client Removal from ESX/ESXi Builds. For ESX and ESXi, the vSphere Client is available for download from the VMware Web site. It is no longer packaged with builds of ESX and ESXi. After installing ESX and ESXi, users are directed to the download page on the VMware Web site to get the compatible vSphere Client for that release. The vSphere Client is still packaged with builds of vCenter Server.

• Boot from SAN. vSphere 4.1 enables ESXi boot from SAN (BFN). iSCSI, FCoE, and Fibre Channel boot are supported. Refer to the Hardware Compatibility Guide for the latest list of NICs and Converged Adapters that are supported with iSCSI boot.
• Hardware Acceleration with vStorage APIs for Array Integration (VAAI). ESX can offload specific storage operations to compliant storage hardware. With storage hardware assistance, ESX performs these operations faster and consumes less CPU, memory, and storage fabric bandwidth.
• Storage Performance Statistics. vSphere 4.1 offers enhanced visibility into storage throughput and latency of hosts and virtual machines, and aids in troubleshooting storage performance issues. NFS statistics are now available in vCenter Server performance charts, as well as esxtop. New VMDK and datastore statistics are included. All statistics are available through the vSphere SDK.
• Storage I/O Control. This feature provides quality-of-service capabilities for storage I/O in the form of I/O shares and limits that are enforced across all virtual machines accessing a datastore, regardless of which host they are running on. Using Storage I/O Control, vSphere administrators can ensure that the most important virtual machines get adequate I/O resources even in times of congestion.
• Network I/O Control. Traffic-management controls allow flexible partitioning of physical NIC bandwidth between different traffic types, including virtual machine, vMotion, FT, and IP storage traffic (vNetwork Distributed Switch only).
• Lockdown Mode Enhancements. VMware ESXi 4.1 lockdown mode allows the administrator to tightly restrict access to the ESXi Direct Console User Interface (DCUI) and Tech Support Mode (TSM). When lockdown mode is enabled, DCUI access is restricted to the root user, while access to Tech Support Mode is completely disabled for all users. With lockdown mode enabled, access to the host for management or monitoring using CIM is possible only through vCenter Server. Direct access to the host using the vSphere Client is not permitted.
• Improved Support for Handling Recalled Patches in vCenter Update Manager. Update Manager 4.1 immediately sends critical notifications about recalled ESX and related patches. In addition, Update Manager prevents you from installing a recalled patch that you might have already downloaded. This feature also helps you identify hosts where recalled patches might already be installed.
• ESX/ESXi Active Directory Integration. Integration with Microsoft Active Directory allows seamless user authentication for ESX/ESXi. You can maintain users and groups in Active Directory for centralized user management and you can assign privileges to users or groups on ESX/ESXi hosts. In vSphere 4.1, integration with Active Directory allows you to roll out permission rules to hosts by using Host Profiles.
• Memory Compression. Compressed memory is a new level of the memory hierarchy, between RAM and disk. Slower than memory, but much faster than disk, compressed memory improves the performance of virtual machines when memory is under contention, because less virtual memory is swapped to disk.

You can see the full feature list on the VMware page, the list is pretty long.

A few things to keep in mind for now. That SIOC is enabled per datastore, does not have a reservation limit at this time, only enforces limits when a threshold is reached and can enforce limits for IOPS when specified at the VM level.  Thanks to Duncan for these details.

In the images below you can get an idea of how Storage IO Control is working along with Shares applied to Virtual Machines to stop any VM from hogging all of the IO.

www.youtube.com/watch?v=5GN5f1u7pcc

When cloning the virtual machine there are still a lot of registry entries left over with the old server name for tools like Opsware, McAfee and an asset mgmt application. These old registry entries would cause the new VM to knock the old VM out of the tools console and cause issues with reporting and management of the source server. For example Opsware would start talking to the newly cloned server thinking it was the source server and this would leave the source machine not in the patching scheduled.

To clean this up it was necessary to uninstall these tools from the new virtual machine then go back through the registry and clean up any entries that referenced the old server name. Then reboot the server and reinstall the tools. This will clean up the issues and allow everything to work without conflicting with the source server.

So for future requests of this type they have agreed to either use a base OS build and load applications on by hand or create their Citrix template with applications and clone prior to installing any monitoring applications. This will eliminate the issues that they had been experiencing.

Here are some of the release notes for ESX 4.0 U2, see full list here.

• Enablement of Fault Tolerance Functionality for Intel Xeon 56xx Series processors— vSphere 4.0 Update 1 supports the Intel Xeon 56xx Series processors without Fault Tolerance. vSphere 4.0 Update 2 enables Fault Tolerance functionality for the Intel Xeon 56xx Series processors.
• Enablement of Fault Tolerance Functionality for Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors— vSphere 4.0 Update 1 supports the Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors without Fault Tolerance. vSphere 4.0 Update 2 enables Fault Tolerance functionality for the Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors.
• Enablement of IOMMU Functionality for AMD Opteron 61xx and 41xx Series processors— vSphere 4.0 Update 1 supports the AMD Opteron 61xx and 41xx Series processors without input/output memory management unit (IOMMU). vSphere 4.0 Update 2 enables IOMMU functionality for the AMD Opteron 61xx and 41xx Series processors.
• Enhancement of the esxtop/resxtop utility— vSphere 4.0 Update 2 includes an enhancement of the performance monitoring utilities, esxtop and resxtop. The esxtop/resxtop utilities now provide visibility into the performance of NFS datastores in that they display the following statistics for NFS datastores: Reads/s, writes/s, MBreads/s, MBwrtn/s, cmds/s, GAVG/s(guest latency).
• Additional Guest Operating System Support— ESX/ESXi 4.0 Update 2 adds support for Ubuntu 10.04.

Here are some of the release notes for ESX 4.0 U2, see full list here.

• Enablement of Fault Tolerance Functionality for Intel Xeon 56xx Series processors— vSphere 4.0 Update 1 supports the Intel Xeon 56xx Series processors without Fault Tolerance. vSphere 4.0 Update 2 enables Fault Tolerance functionality for the Intel Xeon 56xx Series processors.
• Enablement of Fault Tolerance Functionality for Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors— vSphere 4.0 Update 1 supports the Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors without Fault Tolerance. vSphere 4.0 Update 2 enables Fault Tolerance functionality for the Intel i3/i5 Clarkdale Series and Intel Xeon 34xx Clarkdale Series processors.
• Enablement of IOMMU Functionality for AMD Opteron 61xx and 41xx Series processors— vSphere 4.0 Update 1 supports the AMD Opteron 61xx and 41xx Series processors without input/output memory management unit (IOMMU). vSphere 4.0 Update 2 enables IOMMU functionality for the AMD Opteron 61xx and 41xx Series processors.
• Enhancement of the resxtop utility— vSphere 4.0 U2 includes an enhancement of the performance monitoring utility, resxtop. The resxtop utility now provides visibility into the performance of NFS datastores in that it displays the following statistics for NFS datastores: Reads/s, writes/s, MBreads/s, MBwrtn/s, cmds/s, GAVG/s (guest latency).
• Additional Guest Operating System Support— ESX/ESXi 4.0 Update 2 adds support for Ubuntu 10.04.

Here are some of the release notes for vCenter 4.0 U2, see full list here.

• Guest Operating System Customization Improvements: vCenter Server now supports customization of the following guest operating systems:
  • Windows XP Professional SP2 (x64) serviced by Windows Server 2003 SP2
  • SLES 11 (x32 and x64)
  • SLES 10 SP3 (x32 and x64)
  • RHEL 5.5 Server Platform (x32 and x64)
  • RHEL 5.4 Server Platform (x32 and x64)
  • RHEL 4.8 Server Platform (x32 and 64)
  • Debian 5.0 (x32 and x64)
  • Debian 5.0 R1 (x32 and x64)
  • Debian 5.0 R2 (x32 and x64)

I was able to get pass this issue by downloading the latest copy of Virtual Center server 2.5 update 6 and then extracting the License server install file from the VPX folder. Seems that the copy from the install .iso file is newer than the file available for direct downloading. Maybe VMware can fix this soon. I did not try it but according to some other posts the version in VC 2.5 U5 might also work. Once I installed this version it worked the first time.

Error 1920.Service VMware License Server failed to start. Verify that you have sufficient privileges to start system services.

cmd addnode failed for primary node: Internal AAM Error agent could not start

I found the following VMware KB article to help troubleshoot these types of errors. My issue ended up being an issue with the cluster that was created. I created a new cluster and moved my ESX Hosts and Virtual Machines over to it and the issue was gone. Before trying this route I had examined several of the options listed in the KB article.