Tag Archives: powercli

vSAN Cluster Shutdown

A few weeks ago I had to shutdown a vSAN Cluster temporarily for a planned site-wide 24 hour power outage that was blacking out a datacentre. With the amount of warning and a multi-datacentre design this wasn’t an issue, but I made use of vSphere tags and some Powershell/PowerCLI to help with the evacuation and repopulation of the affected cluster. Hopefully some of this may be useful to others.

The infrastructure has two vSAN Clusters – Cluster-Alpha and Cluster-Beta. Cluster-Beta was the one being affected by the power outage, and there was sufficient space on Cluster-Alpha to absorb migrated workloads. Whilst they exist in different datacentres both clusters are on the same LAN and under the same vCenter.

I divided the VMs on Cluster-Beta into three categories:

  1. Powered-Off VMs and Templates. These were to stay in place, they would be inaccessible for the outage but I determined this wouldn’t present any issues.
  2. VMs which needed to migrate and stay on. These were tagged with the vSphere tag “July2019Migrate”
  3. VMs which needed to be powered off but not migrated. For example test/dev boxes which were not required for the duration. These were tagged with “July2019NOMigrate”

The tagging was important, not only to make sure I knew what was migrating and what was staying, but also what we needed to move back or power on once the electrical work had completed. PowerCLI was used to check that all powered-on VMs in Cluster-Beta were tagged one way or another.

Get the VMs in CLuster-Beta where the tag “July2019Migrate” is not assigned and the tag “July 2019NOMigrate” is not assigned and the VM is Powered On.

Get-Cluster -Name "Cluster-Beta" |Get-VM | where {
 (Get-TagAssignment -Entity $_).Tag.Name –notcontains "July2019Migrate" –and
 (Get-TagAssignment -Entity $_).Tag.Name –notcontains "July2019NOMigrate" –and
 $_.PowerState –eq “PoweredOn”}

In the week approaching the shutdown the migration was kicked off:

#Create a List of the VMs in the Source Cluster which are tagged to migrate
$MyTag= Get-Tag -Name "July2019Migrate"
$MyVMs=Get-Cluster "Cluster-Beta" | Get-VM | Where-Object {(Get-TagAssignment -Entity $_).Tags -contains $MyTag }
#Do the Migration
$TargetCluster= "Cluster-Alpha" #Target Cluster
$TargetDatastore= "vSANDatastore-Alpha" #Target Datastore on Target Cluster
$MyVMs | Move-VM -Destination (Get-Cluster -Name $TargetCluster) -Datastore (Get-Datastore -Name $TargetDatastore) -DiskStorageFormat Thin -VMotionPriority High

At shutdown time, a quick final check of the remaining powered on VMs was done and then all remaining VMs in Cluster-Beta were shut down. Once there were no running workloads on Beta it was time to shut down the vSAN cluster. This part I didn’t automate as I’m not planning on doing it a lot, and there’s comprehensive documentation in the VMware Docs site. The process is basically one of putting all the hosts into maintenance mode and then once the whole cluster is done, powering them off.

You are in a dark, quiet datacentre. There are many servers, all alike. There may be Grues here.

When power was restored, the process was largely reversed. I powered on the switches providing the network interconnect between the nodes, and then powered on those vSAN hosts and waited for them to come up. Once all the hosts were visible to vCenter, it was just a case of selecting them all and choosing “Exit Maintenance Mode”

2019-07-29 (8)

There was a momentary flash of alerts as nodes come up and wonder where their friends are, but in under a minute the cluster was passing the vSAN Health Check

image

At this point it was all ready to power on the VMs that had been shutdown and left on the cluster, and vMotion the migrated virtual machines back across. Again, PowerCLI simplified this process:

#Create a List of the VMs in the Source Cluster which are tagged to stay but need powering on.
$MyTag= Get-Tag -Name "July2019NOMigrate"
$MyVMs=Get-Cluster “Cluster-Alpha” | Get-VM | Where-Object {(Get-TagAssignment -Entity $_).Tags -contains $MyTag }
#Power on those VMs
$MyVMs | Start-VM

#Create a List of the VMs in the Source Cluster which are tagged to migrate (back)
$MyTag= Get-Tag -Name "July2019Migrate"
$MyVMs=Get-Cluster “Cluster-Alpha” | Get-VM | Where-Object {(Get-TagAssignment -Entity $_).Tags -contains $MyTag }
#Do the Migration
$TargetCluster= "Cluster-Beta" #New Target Cluster
$TargetDatastore= "vSANDatastore-Beta" #Target Datastore on Target Cluster
$MyVMs | Move-VM -Destination (Get-Cluster -Name $TargetCluster) -Datastore (Get-Datastore -Name $TargetDatastore) -DiskStorageFormat Thin -VMotionPriority High

Then it was just a case of waiting for the data to flow across the network and finally check that everything had migrated successfully and normality had been restored.

we have normality, I repeat we have normality…Anything you still can’t cope with is therefore your own problem. Please relax.

Trillian, via the keyboard of Douglas Adams. The Hitchhiker’s Guide to the Galaxy

Quick PowerCLI- Getting VM hardware versions

A quick PowerCLI snippet for examining what VM Hardware versions exist in your virtual environment:

Using the “Group-Object” cmdlet we can run up a quick count of all the VMs on each hardware version

Get-VM | Group-Object Version

Count Name                      Group
----- ----                      -----
42    v13                       {VM1,VM2,VM3...}
257   v8                        {VM4,VM5,VM6...}
70    v11                       {VM7,VM8,VM9...}
2     v4                        {VM10,VM11}
5     v10                       {VM12,VM13,VM14...}
2     v9                        {VM15,VM16}
2     v7                        {VM17,VM18}

This can be refined using “Sort-Object” to put the most common hardware version at the top of the list.

Get-VM  | Group-Object Version | Sort-Object Count -Descending
Count Name                      Group
----- ----                      -----
257   v8                        {VM4,VM5,VM6...}
70    v11                       {VM7,VM8,VM9...}
42    v13                       {VM1,VM2,VM3...}
5     v10                       {VM12,VM13,VM14...}
2     v7                        {VM17,VM18}
2     v9                        {VM15,VM16}
2     v4                        {VM10,VM11}

We may only be concerned with VMs that are Powered On, so “Where-Object” can be used to filter the original list.

Get-VM  | Where-Object {$_.PowerState -eq "PoweredOn"} | Group-Object Version | Sort-Object Count -Descending
Count Name                      Group
----- ----                      -----
66    v8                        {VM4,VM5,VM19...}
51    v11                       {VM7,VM8,VM9...}
33    v13                       {VM1,VM21,VM22...}
5     v10                       {VM12,VM13,VM20...}
2     v9                        {VM15,VM16}
1     v4                        {VM10}

This quick snippet can be useful when establishing the range of hardware versions in an environment, or estimating the amount of work involved in updating VM hardware to a modern standard across an estate.

Find VMs being replicated to a datastore with Get-VmdkFolders

I’m using vSphere Replication to replicate a number of VMs to a datastore which is being decommissioned. I can’t reconfigure replication by code (I’ve mentioned that there’s no public API or PowerCLI around the vSphere Replication functionality in vSphere 6.5 before- most recently on Twitter)  and I can’t multi-select VMs in the vSphere Web Client and reconfigure en-mass through the GUI.

There’s also no way of telling which VMs are being replicated to this datastore without going into the reconfigure dialogues for each one individually. That’s a lot of clicking before even starting to update the configuration.

I’ve put together a PowerCLI function takes a little of this pain away by establishing which folders on a given datastore contain VMDK files. I’ve already migrated any Virtual Machines off so any remaining VMDKs on the volume should be associated with a replication process. Additionally in this case the default folder name was used when setting up the replication so all the folder names correspond to a VM.

The function is called “Get-VmdkFolders” (as there’s potentially other uses for it) and is available on GitHub here. With this I now have a list of VMs that need manually reconfiguring.

PS C:\> Get-VmdkFolders "MyReplicationDatastore"
MyVM1
MyVM2
MyVM3
MyVM5
MyVM7

 

PowerShell Snippet – Find Windows 2008 VMs before EOL

Support for Windows Server 2008 and 2008 R2 ends in January 2020– just two years from the date of this post. Four years ago I put up a script to find XP and 2003 VMs, and I’ve modified this to search a vSphere environment for powered on VMs running Server 2008 as a Guest OS:

Get-VM |
 Where {$_.PowerState -eq "PoweredOn" -and
 ($_.Guest -like "*Server 2008*")} |
 get-VMGuest |
 select VmName, OSFullName

VmName    OSFullName
------    ----------
MyVM1     Microsoft Windows Server 2008 R2 (64-bit)
MyVM2     Microsoft Windows Server 2008 R2 (64-bit)
MyVM3     Microsoft Windows Server 2008 (64-bit)
MyVM4     Microsoft Windows Server 2008 R2 (64-bit)

Exploring Tags and PowerCLI

Tags were added to vSphere back in version 5.1 so they’re not a new feature but are still often overlooked. One or more tags can be applied to items (entities) in the inventory and then used as a search term or metadata not only in the GUI but also through tools such as PowerCLI. This post covers a few useful cmdlets for working with tags.

CmdLets

There are a number of cmdlets which deal with tags, here’s a quick list using Get-Command.

image

Notice that there’s three Nouns used here- “Tag” represents the tag itself. “TagAssignment” represents a relationship between a tag and another object (for example “This VM has been assigned This (or These) tags). Finally there’s “TagCategory” which represents the category that a tag belongs to.

Getting Tags

So, what can we do with tags in PowerCLI? Well, first we can look at a list of all the tags using Get-Tag. This returns a lot of information, particularly if you have assigned tags already, so we can neaten the quick view using the PowerShell “Select” function to show just the tag name and description:

Get-Tag | Select Name, Description

Name                 Description
– –                 – – – – – –
UrlShortener         URL Shortener Service
Documents            Document Management Service
Change               Change Management Service

In this example, I’ve created three tags to represent three different services operating in my environment. We can carry on from here and find out which entities have been assigned the “Documents” tag- i.e. what VMs form the Document Management Service.

(Get-TagAssignment |
  Where {$_.Tag.Name -eq 'Documents'}).Entity


Name                 PowerState Num CPUs MemoryGB
– –                  – – – – – – – – – – – – –
DocuWebServ          PoweredOn  1        4.000

DocuDBServ           PoweredOn  2        16.000
DocuFileServ         PoweredOn  1        4.000

Or we could flip that and ask the question- “What tags does this VM have assigned?”

Get-VM "DocuWebServ" |
     Get-TagAssignment | Select Tag

Tag
– –
Documents

WebServers

Getting Bigger

As we’re using PowerCLI we can join more and more functions together and make bigger and bigger queries. For example, we can  list all VMs with their tags in a table.

Get-VM |
      Select Name,@{Name="Tags";Expression={(Get-TagAssignment -Entity $_).Tag.Name}} |
      Where {$_.Tags} |
      Format-Table - Autosize

Name         Tags
– –          – –
DocuWebServ  {Documents, WebServers}
DocuDBServ   Documents
DocuFileServ Documents
URLShort1    {UrlShortener, WebServers}
URLShort2  {UrlShortener, WebServers, TestAndDev}

This is only scratching the surface of the possibilities- by having useful metadata that lives with the VM and can be accessed programmatically we have plenty of avenues to explore in automation and reporting.