Tag Archives: vSphere

Email Alerts not sending in vCenter 6.5

Symptoms

  • vCenter (in my case VCSA 6.5 ) is not sending alert emails even though the mail server appears to be configured properly (in the vSphere Web Client this is on the vCenter node under Configure, Settings, General).
  • Network connectivity between vCenter and port 25 on the SMTP server is fine and has been tested.
  • The logs on the SMTP server show no evidence of emails beings sent.
  • When connecting via SSH to the vCenter Server Appliance the sendmail logs are showing errors such as:
    cat /var/log/messages | grep sendmail

    2018-07-14T04:24:46.569978+00:00 my-vc sendmail[21502]: ……: [email protected], [email protected] (0/0),
      delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=30528, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0,
      stat=Deferred: Connection refused by [127.0.0.1]
  • Cause

    The “Connection refused by [127.0.0.1]” in the logs is the giveaway here- the settings in the client have not been applied to the sendmail service and vCenter is trying to connect to the localhost as the SMTP server.

    Solution

    Go to the client and reconfigure the mail server to point at something else (it doesn’t have to be a real hostname). Then change it back to the correct settings. Normal alerts will be resumed automatically straight away.  You can test if it works by temporarily adding an additional trigger with a very low threshold to an existing alert with notification action.

    vCenter Mail Settings in vSphere Web Client

    vCenter Mail Settings in vSphere Web Client


    Thanks to vExpert Laurens van Duijin for sending me down the right path with this issue.

Find VMs being replicated to a datastore with Get-VmdkFolders

I’m using vSphere Replication to replicate a number of VMs to a datastore which is being decommissioned. I can’t reconfigure replication by code (I’ve mentioned that there’s no public API or PowerCLI around the vSphere Replication functionality in vSphere 6.5 before- most recently on Twitter)  and I can’t multi-select VMs in the vSphere Web Client and reconfigure en-mass through the GUI.

There’s also no way of telling which VMs are being replicated to this datastore without going into the reconfigure dialogues for each one individually. That’s a lot of clicking before even starting to update the configuration.

I’ve put together a PowerCLI function takes a little of this pain away by establishing which folders on a given datastore contain VMDK files. I’ve already migrated any Virtual Machines off so any remaining VMDKs on the volume should be associated with a replication process. Additionally in this case the default folder name was used when setting up the replication so all the folder names correspond to a VM.

The function is called “Get-VmdkFolders” (as there’s potentially other uses for it) and is available on GitHub here. With this I now have a list of VMs that need manually reconfiguring.

PS C:\> Get-VmdkFolders "MyReplicationDatastore"
MyVM1
MyVM2
MyVM3
MyVM5
MyVM7

 

vSphere Encryption- Knowing your limits

SecurityI’ve been running a Proof of Concept system for vSphere Encryption using vSphere 6.5u1 and a HyTrust KeyControl 4.0 KMS cluster. This has been very straightforward to implement and use and there’s plenty of documentation out there on how to do so, but in this post I’ll be highlighting some of the limitations. A few of these are things you can’t do that you may currently do day-to-day with normal VMs but there’s usually a sound technical reason why it’s not possible to do so with an encrypted VM.

Encrypting and Decrypting

The power state of a VM limits some encryption processes. For example, a powered-on Virtual machine can not be encrypted or decrypted. This example shows what happens when PowerCLI (with the encryption module described here) is used to encrypt a running VM:

PS C:\> Get-VM -Name "KMSTest6" | Disable-VMEncryption
Disable-VMEncryption : The VM can only be decrypted when powered off,
 but the current power state of KMSTest6 is PoweredOn!

The encryption can be changed (a re-key operation), possibly to a different KMS server- however whilst a “Shallow Re-key” operation where the key is re-encrypted is fine, the “Deep Re-key” where the disk itself is re-encrypted with the new key is not possible when the VM is powered on.

Snapshots and Clones

Similarly, a snapshot including memory of a running VM is not possible. It is possible to snapshot a powered off VM, or a running VM without the memory state (which creates a powered-off snapshot), but not to include the memory. This makes sense as the VM Encryption process runs as the hypervisor writes to disk, memory would be outside this process and potentially reveal unencrypted data in the snapshot.

It’s also not possible to decrypt or encrypt a VM with snapshots:

PS C:\> Get-VM -Name "KMSTest5" | Enable-VMEncryption
Enable-VMEncryption : KMSTest5 has snapshots,
 please remove all snapshots and try again!

You can however clone an encrypted VM- the resulting clone is also encrypted and uses the same keys. vMotion also works as expected.

Replication

vSphere Replication does not work with encrypted VMs. Replication can be configured but will fail when it tries to sync.

image

Licensing

In most environments where vSphere Encryption is in use, the hosts will probably be all licensed with Enterprise Plus (see the license comparison table). However, if you are running a mixture of licenses (including any regular non-plus Enterprise licenses) the limitations of  those licenses comes into play. It’s not possible to turn on encryption on a VM allocated to a host with anything but a full Enterprise-Plus license.

Any hosts with Standard, or the no longer available to purchase Enterprise licenses will not allow their VMs to be encrypted- or for encrypted VMs to be migrated onto them. Additionally if you have one or more of these “inferior” hosts in a cluster you will not be able to power on an encrypted machine in that cluster- even if other hosts are licensed to Enterprise Plus.

Disks

There’s lots of flexibility down at the disk level. You can use different keys (or even KMS Clusters) for different virtual hard disks (i.e. each .vmdk has a different key) and you can take an encrypted disk and attach it to another VM. The limitation here is you cannot attach an encrypted virtual hard disk to an unencrypted VM- this again makes sense as the key information in the configuration would then be in the clear.

Summary

The encryption model introduced in vSphere 6.5 is a very useful feature and straightforward to implement however consideration needs to be taken on the continuing activities surrounding virtual machines post-encryption to ensure that operational processes are still valid.

Missing Content Libraries.

Symptoms

I came to deploy a Virtual Machine from a Content Library on vSphere 6.5 and discovered that the Content Library had disappeared.

clip_image001

Cause

The Content Library Service was stopped.

image

Attempting to start the service caused an error- both through the GUI and command line.

login as: root
VMware vCenter Server Appliance 6.5.0.10000
Type: vCenter Server with an embedded Platform Services Controller
[email protected]'s password:
Last login: Thu Sep 7 12:55:51 2017 from

[email protected] [ ~ ]# service-control --status vmware-content-library
Stopped:
vmware-content-library
[email protected] [ ~ ]# service-control --start vmware-content-library
  Perform start operation. vmon_profile=None, svc_names=['vmware-content-library'], include_coreossvcs=False, include_leafossvcs=False
  2017-09-07T13:27:38.208Z Service content-library state STOPPED
  Error executing start on service content-library. Details {
  "resolution": null,
  "detail": [
  {
  "args": [
  "content-library"
  ],
  "id": "install.ciscommon.service.failstart",
  "localized": "An error occurred while starting service 'content-library'",
  "translatable": "An error occurred while starting service '%(0)s'"
  }
  ],
  "componentKey": null,
  "problemId": null
  }
  Service-control failed. Error {
  "resolution": null,
  "detail": [
  {
  "args": [
  "content-library"
  ],
  "id": "install.ciscommon.service.failstart",
  "localized": "An error occurred while starting service 'content-library'",
  "translatable": "An error occurred while starting service '%(0)s'"
  }
  ],
  "componentKey": null,
  "problemId": null
  }
[email protected] [ ~ ]#

The symptoms possibly started following an upgrade of vCenter to 6.5 Update 1.

 

Solution

Removing the ts-config.properties files (see VMware KB2151085 here) allowed me to restart the service.

[email protected]</a> [ ~ ]# cd /etc/vmware-content-library/config
[email protected]</a> [ /etc/vmware-content-library/config ]# ls
  cls-config.properties ts-config.properties ts-config.properties.rpmnew vdcs-config.properties
[email protected]</a> [ /etc/vmware-content-library/config ]# cp ts-config.properties ts-config.properties.orig
[email protected]</a> [ /etc/vmware-content-library/config ]# cp ts-config.properties.rpmnew ts-config.properties.rpmnew.orig
[email protected]</a> [ /etc/vmware-content-library/config ]# mv ts-config.properties.rpmnew ts-config.properties

[email protected]</a> [ /etc/vmware-content-library/config ]# service-control --stop vmware-content-library;service-control --start vmware-content-library
  Perform stop operation. vmon_profile=None, svc_names=['vmware-content-library'], include_coreossvcs=False, include_leafossvcs=False
  Successfully stopped service content-library
  Perform start operation. vmon_profile=None, svc_names=['vmware-content-library'], include_coreossvcs=False, include_leafossvcs=False
  2017-09-07T13:29:15.212Z Service content-library state STOPPED
  Successfully started service content-library
[email protected]</a> [ /etc/vmware-content-library/config ]#

 

The Content Library however was still not visible until after a reboot of the vCenter Server Appliance.

image

Exploring Tags and PowerCLI

Tags were added to vSphere back in version 5.1 so they’re not a new feature but are still often overlooked. One or more tags can be applied to items (entities) in the inventory and then used as a search term or metadata not only in the GUI but also through tools such as PowerCLI. This post covers a few useful cmdlets for working with tags.

CmdLets

There are a number of cmdlets which deal with tags, here’s a quick list using Get-Command.

image

Notice that there’s three Nouns used here- “Tag” represents the tag itself. “TagAssignment” represents a relationship between a tag and another object (for example “This VM has been assigned This (or These) tags). Finally there’s “TagCategory” which represents the category that a tag belongs to.

Getting Tags

So, what can we do with tags in PowerCLI? Well, first we can look at a list of all the tags using Get-Tag. This returns a lot of information, particularly if you have assigned tags already, so we can neaten the quick view using the PowerShell “Select” function to show just the tag name and description:

Get-Tag | Select Name, Description

Name                 Description
– –                 – – – – – –
UrlShortener         URL Shortener Service
Documents            Document Management Service
Change               Change Management Service

In this example, I’ve created three tags to represent three different services operating in my environment. We can carry on from here and find out which entities have been assigned the “Documents” tag- i.e. what VMs form the Document Management Service.

(Get-TagAssignment |
  Where {$_.Tag.Name -eq 'Documents'}).Entity


Name                 PowerState Num CPUs MemoryGB
– –                  – – – – – – – – – – – – –
DocuWebServ          PoweredOn  1        4.000

DocuDBServ           PoweredOn  2        16.000
DocuFileServ         PoweredOn  1        4.000

Or we could flip that and ask the question- “What tags does this VM have assigned?”

Get-VM "DocuWebServ" |
     Get-TagAssignment | Select Tag

Tag
– –
Documents

WebServers

Getting Bigger

As we’re using PowerCLI we can join more and more functions together and make bigger and bigger queries. For example, we can  list all VMs with their tags in a table.

Get-VM |
      Select Name,@{Name="Tags";Expression={(Get-TagAssignment -Entity $_).Tag.Name}} |
      Where {$_.Tags} |
      Format-Table - Autosize

Name         Tags
– –          – –
DocuWebServ  {Documents, WebServers}
DocuDBServ   Documents
DocuFileServ Documents
URLShort1    {UrlShortener, WebServers}
URLShort2  {UrlShortener, WebServers, TestAndDev}

This is only scratching the surface of the possibilities- by having useful metadata that lives with the VM and can be accessed programmatically we have plenty of avenues to explore in automation and reporting.