Tag Archives: vSphere

vSphere Encryption- Knowing your limits

SecurityI’ve been running a Proof of Concept system for vSphere Encryption using vSphere 6.5u1 and a HyTrust KeyControl 4.0 KMS cluster. This has been very straightforward to implement and use and there’s plenty of documentation out there on how to do so, but in this post I’ll be highlighting some of the limitations. A few of these are things you can’t do that you may currently do day-to-day with normal VMs but there’s usually a sound technical reason why it’s not possible to do so with an encrypted VM.

Encrypting and Decrypting

The power state of a VM limits some encryption processes. For example, a powered-on Virtual machine can not be encrypted or decrypted. This example shows what happens when PowerCLI (with the encryption module described here) is used to encrypt a running VM:

PS C:\> Get-VM -Name "KMSTest6" | Disable-VMEncryption
Disable-VMEncryption : The VM can only be decrypted when powered off,
 but the current power state of KMSTest6 is PoweredOn!

The encryption can be changed (a re-key operation), possibly to a different KMS server- however whilst a “Shallow Re-key” operation where the key is re-encrypted is fine, the “Deep Re-key” where the disk itself is re-encrypted with the new key is not possible when the VM is powered on.

Snapshots and Clones

Similarly, a snapshot including memory of a running VM is not possible. It is possible to snapshot a powered off VM, or a running VM without the memory state (which creates a powered-off snapshot), but not to include the memory. This makes sense as the VM Encryption process runs as the hypervisor writes to disk, memory would be outside this process and potentially reveal unencrypted data in the snapshot.

It’s also not possible to decrypt or encrypt a VM with snapshots:

PS C:\> Get-VM -Name "KMSTest5" | Enable-VMEncryption
Enable-VMEncryption : KMSTest5 has snapshots,
 please remove all snapshots and try again!

You can however clone an encrypted VM- the resulting clone is also encrypted and uses the same keys. vMotion also works as expected.


vSphere Replication does not work with encrypted VMs. Replication can be configured but will fail when it tries to sync.



In most environments where vSphere Encryption is in use, the hosts will probably be all licensed with Enterprise Plus (see the license comparison table). However, if you are running a mixture of licenses (including any regular non-plus Enterprise licenses) the limitations of  those licenses comes into play. It’s not possible to turn on encryption on a VM allocated to a host with anything but a full Enterprise-Plus license.

Any hosts with Standard, or the no longer available to purchase Enterprise licenses will not allow their VMs to be encrypted- or for encrypted VMs to be migrated onto them. Additionally if you have one or more of these “inferior” hosts in a cluster you will not be able to power on an encrypted machine in that cluster- even if other hosts are licensed to Enterprise Plus.


There’s lots of flexibility down at the disk level. You can use different keys (or even KMS Clusters) for different virtual hard disks (i.e. each .vmdk has a different key) and you can take an encrypted disk and attach it to another VM. The limitation here is you cannot attach an encrypted virtual hard disk to an unencrypted VM- this again makes sense as the key information in the configuration would then be in the clear.


The encryption model introduced in vSphere 6.5 is a very useful feature and straightforward to implement however consideration needs to be taken on the continuing activities surrounding virtual machines post-encryption to ensure that operational processes are still valid.


Missing Content Libraries.


I came to deploy a Virtual Machine from a Content Library on vSphere 6.5 and discovered that the Content Library had disappeared.



The Content Library Service was stopped.


Attempting to start the service caused an error- both through the GUI and command line.

login as: root
VMware vCenter Server Appliance
Type: vCenter Server with an embedded Platform Services Controller
[email protected]'s password:
Last login: Thu Sep 7 12:55:51 2017 from

[email protected] [ ~ ]# service-control --status vmware-content-library
[email protected] [ ~ ]# service-control --start vmware-content-library
  Perform start operation. vmon_profile=None, svc_names=['vmware-content-library'], include_coreossvcs=False, include_leafossvcs=False
  2017-09-07T13:27:38.208Z Service content-library state STOPPED
  Error executing start on service content-library. Details {
  "resolution": null,
  "detail": [
  "args": [
  "id": "install.ciscommon.service.failstart",
  "localized": "An error occurred while starting service 'content-library'",
  "translatable": "An error occurred while starting service '%(0)s'"
  "componentKey": null,
  "problemId": null
  Service-control failed. Error {
  "resolution": null,
  "detail": [
  "args": [
  "id": "install.ciscommon.service.failstart",
  "localized": "An error occurred while starting service 'content-library'",
  "translatable": "An error occurred while starting service '%(0)s'"
  "componentKey": null,
  "problemId": null
[email protected] [ ~ ]#

The symptoms possibly started following an upgrade of vCenter to 6.5 Update 1.



Removing the ts-config.properties files (see VMware KB2151085 here) allowed me to restart the service.

[email protected]</a> [ ~ ]# cd /etc/vmware-content-library/config
[email protected]</a> [ /etc/vmware-content-library/config ]# ls
  cls-config.properties ts-config.properties ts-config.properties.rpmnew vdcs-config.properties
[email protected]</a> [ /etc/vmware-content-library/config ]# cp ts-config.properties ts-config.properties.orig
[email protected]</a> [ /etc/vmware-content-library/config ]# cp ts-config.properties.rpmnew ts-config.properties.rpmnew.orig
[email protected]</a> [ /etc/vmware-content-library/config ]# mv ts-config.properties.rpmnew ts-config.properties

[email protected]</a> [ /etc/vmware-content-library/config ]# service-control --stop vmware-content-library;service-control --start vmware-content-library
  Perform stop operation. vmon_profile=None, svc_names=['vmware-content-library'], include_coreossvcs=False, include_leafossvcs=False
  Successfully stopped service content-library
  Perform start operation. vmon_profile=None, svc_names=['vmware-content-library'], include_coreossvcs=False, include_leafossvcs=False
  2017-09-07T13:29:15.212Z Service content-library state STOPPED
  Successfully started service content-library
[email protected]</a> [ /etc/vmware-content-library/config ]#


The Content Library however was still not visible until after a reboot of the vCenter Server Appliance.


Exploring Tags and PowerCLI

Tags were added to vSphere back in version 5.1 so they’re not a new feature but are still often overlooked. One or more tags can be applied to items (entities) in the inventory and then used as a search term or metadata not only in the GUI but also through tools such as PowerCLI. This post covers a few useful cmdlets for working with tags.


There are a number of cmdlets which deal with tags, here’s a quick list using Get-Command.


Notice that there’s three Nouns used here- “Tag” represents the tag itself. “TagAssignment” represents a relationship between a tag and another object (for example “This VM has been assigned This (or These) tags). Finally there’s “TagCategory” which represents the category that a tag belongs to.

Getting Tags

So, what can we do with tags in PowerCLI? Well, first we can look at a list of all the tags using Get-Tag. This returns a lot of information, particularly if you have assigned tags already, so we can neaten the quick view using the PowerShell “Select” function to show just the tag name and description:

Get-Tag | Select Name, Description

Name                 Description
– –                 – – – – – –
UrlShortener         URL Shortener Service
Documents            Document Management Service
Change               Change Management Service

In this example, I’ve created three tags to represent three different services operating in my environment. We can carry on from here and find out which entities have been assigned the “Documents” tag- i.e. what VMs form the Document Management Service.

(Get-TagAssignment |
  Where {$_.Tag.Name -eq 'Documents'}).Entity

Name                 PowerState Num CPUs MemoryGB
– –                  – – – – – – – – – – – – –
DocuWebServ          PoweredOn  1        4.000

DocuDBServ           PoweredOn  2        16.000
DocuFileServ         PoweredOn  1        4.000

Or we could flip that and ask the question- “What tags does this VM have assigned?”

Get-VM "DocuWebServ" |
     Get-TagAssignment | Select Tag

– –


Getting Bigger

As we’re using PowerCLI we can join more and more functions together and make bigger and bigger queries. For example, we can  list all VMs with their tags in a table.

Get-VM |
      Select Name,@{Name="Tags";Expression={(Get-TagAssignment -Entity $_).Tag.Name}} |
      Where {$_.Tags} |
      Format-Table - Autosize

Name         Tags
– –          – –
DocuWebServ  {Documents, WebServers}
DocuDBServ   Documents
DocuFileServ Documents
URLShort1    {UrlShortener, WebServers}
URLShort2  {UrlShortener, WebServers, TestAndDev}

This is only scratching the surface of the possibilities- by having useful metadata that lives with the VM and can be accessed programmatically we have plenty of avenues to explore in automation and reporting.

The All New vSphere 6.5


  1. A new version of VMware vSphere, 6.5, will be released shortly
  2. Migration/Upgrade tools from previous versions (including Windows vCenter) to new VCSA.
  3. VCSA Native High Availability
  4. VCSA Integrated VMware Update Manager
  5. Native vCenter Backup and Restore
  6. Improved Appliance Management
  7. vSphere Clients
  8. Encryption

New vSphere coming soon

VMware has bucked the trend in versioning adopted by other major software companies and decided not to call it’s new vSphere version “10” and opted for the more traditional “vSphere 6.5” to succeed version 6.0 which was originally released back in March 2015. Announced at VMworld Europe 2016 with GA to follow, vSphere 6.5 is a continuation of the product which forms the core of the Software Defined Datacentre chunk of VMware’s “Any Cloud” Cross-Cloud Architecture portfolio. A lot of work has been put into making the experience of installing and operating a vSphere virtualised environment easier; Ignoring any improvements under the hood, and just looking at what’s on the surface there’s a whole bunch of features designed to make life run smoother for the IT Professional, some of which are highlighted in this post.

The new vCenter Server Appliance is a core part to this simplicity, and VMware have answered the requirements of anyone currently sticking to the Windows-based vCenter. If you can get more features and more reliability for less cost and less effort then it’s definitely the way forwards in my opinion. Some of the features discussed here- notably Native HA and Backup/Restore- will only be available in the appliance version of vCenter.

VCSA Upgrade and Migration


image Again out to both simplify the life of IT Professionals and encourage vCenter Appliance adoption, VMware has put a lot of effort into creating straightforward, and comprehensive, upgrade and migration tools. As more and more operations and data are handled by vCenter it becomes more and more important that the system can be smoothly navigated from version to version with minimal human effort.

Migrations are possible from Windows vCenters running version 5.5 or 6.0, and both the embedded and external database topologies are supported. Additionally, the new vCenter will assume the identity of the old Windows vCenter so any external interfaces, scripts, and automation should continue to work post-migration.

VCSA Native High Availability

VCSA 6.5 offers a built-in high availability deployment taking away the need for any 3rd party clustering or database solutions. The appliance deploys as an active/passive pair (plus witness) which automatically sets up replication of the integrated database and required vCenter files. The basic setup option also places these nodes intelligently using DRS and SDRS technology and automatically creates the necessary affinity rules and private IP comms, keeping everything simple. For infrastructures with unique and challenging topologies, there’s still an advanced workflow that can be used.


Integrated VMware Update Manager

Prior to 6.5 using VUM to manage the patching of a vSphere infrastructure based on the vCenter Appliance has been, how can we put it?, “annoying”. After deploying the slick appliance it was then necessary to spin up (and license) a separate Windows VM just to handle the update system. This requirement has been removed in the new version- VUM is now integrated into the VCSA, enabled by default, and shares the same database instance. The new VUM integration also leverages the VCSA High Availability and Backup functionality.

Native vCenter Backup and Restore

Also new to the vCenter Server Appliance is integrated backup and restore functionality. A great step forward in the simplification of deploying a system this provides a built in solution to backup vCenter to an external location (SCP, SFTP, HTTPS locations for example) and then be able to recover by deploying a clean OVA and choosing the Restore option. image


Improved Appliance Management and Monitoring

The vCenter Server Appliance Management Interface- VAMI – has also had a makeover, with many features being added. The 6.0 version had an interface limited to changing IP and NTP settings, rebooting the appliance, and little else. 6.5 adds in built in monitoring of Network, CPU, Memory and the vPostgres database. There is also the option to configure Syslog for deeper external monitoring of the vCenter infrastructure- this allows fully verbose logs to be kept for auditing and troubleshooting processes.


vCenter Server Appliance 6.0 Management Interface


vCenter Server Appliance 6.5 Management Interface

vSphere Client(s)

Work continues to focus on delivering a fully functioned HTML5 client, but in the interim vCenter 6.5 will come shipped with a new (limited) HTML5 based “vSphere Client”- evolved from the current fling – as well as an improved flash based “vSphere Web Client”. Expect the “vSphere Client” to see continuous improvement and feature addition through the lifetime of the platform –driven through the Fling programme.


As with the other topics here encryption in the new vSphere could easily be a post in itself (or a whole series), but to summarise the new features in this area, vSphere is now offering built-in VM encryption. The encryption happens between the VM and the storage so is invisible to the guest.

Local keys are generated within vSphere, and encrypted using keys held in an external (third-party) KMS- this would usually be managed by the IT Security team. Back in vCenter encryption is implemented through Storage Policies, so a VM can be encrypted simply by assigning the correct policy to it. Through the GUI (or API/PowerCLI) it’s possible to set  encryption covering  the Disks, the VMX/Swap files, or the whole lot on a per-VM basis. Through the API/PowerCLI it’s also possible to arrange encryption on a per-VHD level, potentially encrypting different disks on a VM with different keys.

VSAN encryption is on the way- there’s currently an ongoing beta – but will not be available in the 6.5 release. Based on the recent cadence I’d expect to see something in Spring 2017, but that’s just my speculation.


In summary, there’s lots to look for in the new vSphere release and in particular the vCenter Server Applicance. This week’s VMworld should reveal a lot more in depth into these advances.

License error connecting ESXi server to vCenter

Yesterday I was connecting up an ESXi Virtual Host which has been running in standalone mode for a while to vCenter. The host has recently been upgraded to a VSphere 4 Essentials Plus License.

When trying to setup the connection so the host could be managed by the vCenter server I received the following error message:

License not available to perform the operation.

The vSphere 4 Essentials Plus license For Host myhostname does not include VMotion. Upgrade the license.


This was a bit perplexing, as I don’t need vMotion on this setup and didn’t think it was necessary to have it installed.

The solution came with a bit of help from a page on the Dutch VMWare Users Group site, helpfully translated with Google:

    1. Open vSphere Client and connect to the virtual host.
    2. Go to the “configuration” tab for the host and choose “Networking” from the left hand side of the tab.
    3. Select the vSwitch with the VMkernel Port (usually vSwitch0) and click on Properties.
    4. In the vSwitch properties window select the Management Network and click on Edit.
    5. On the “General” tab of the Management Network window, untick the “VMotion” checkbox.
    6. OK everything and try to link to the vCenter server again.