Support Bundle taking up log file space on VCSA

The log file disk was nearly full on a vCenter Server Appliance instance, showing a warning message in the vCenter console and the VAMI (Login to https://vCenterHostName:5480). After a bit of investigation (and I’d recommend looking at this article by Brandon Lee and Knowledge Base 2143565 ) I found an old Support Bundle was taking up 2GB of the log disk unnecessarily and tripping the alert threshold.

SSH into the VCSA using the username root (see KB2143565) and navigate to the folder


And look for files named “*_vmsupport.tgz“, check the timestamp on these files and remove any old ones that are no longer required.
Hopefully the warning should clear and the health status should return to green in the vSphere Appliance Management Interface
Healthy VCSA


Photon, Photon, Photon

imageVMware’s cloud-native application stack is here, and it uses several things called “Photon”. In this quick post, I’ll have a look at what’s what in the stack and how the components of this container-optimised enterprise cloud platform fit together.

At the base level is Photon OS. This is a Linux distribution created from the ground up by VMware for this infrastructure.

The Photon Controller runs on Photon OS. This is a web-scale control plane which manages the workloads.

Photon Platform is the entire stack. This includes the Photon Controllers but also encompasses the underlying ESX Hypervisor, Network Virtualisation, and Storage underneath. Photon Platform is designed to be multi-tenant, so within one platform multiple tenants can be allocated resources using a hierarchical model and then individual tenants can divide their resources up across projects.


The Amazing World of IT in Higher Education

There are around 17,000 IT Professionals working in the UK Higher Education industry supporting world-leading, cutting edge research and the development of the next generation of minds. They are also stewards of the (slightly less glamorous) systems for HR, payroll, email, printing and so on.
Whilst much of this environment will be familiar to those employed in a more corporate setting, there are a number of striking differences to IT operations found elsewhere in both the public and private sectors which Higher Education straddles.
This quick talk from VMworld Europe 2016 covers some of the surprises that encountered in a HE IT organisation, and how Virtualisation can save the day.

VMworld Europe 2016 Day 2

Day 2, Tuesday, is when the real meat of VMworld starts- kicking off with the General Session (a.k.a. Keynote) at 9am. We were expecting some big announcements and weren’t disappointed. Here’s a look at some of my sessions today- these are the highlights only and more detail will follow.

 General Session









Ten thousand attendees from ninety-six countries packed into the keynote arena for the opening General Session. It borrowed many parts from the US event, but with a European slant and the added announcements of the new vSphere and VSAN 6.5 platforms and the AWS deal revealed only last week. Mike Clayville, VP of AWS, joned VMware’s Pat Gelsinger on stage to talk about how that deal is going to join the leaders in the private cloud with the leaders in the public cloud. General Availability of that platform will be mid-2017.

We were treated to demos of some of the new features offered through the Cloud Foundation and Cross-cloud architecture, including spinning up a vSphere cluster sat on Amazon Web Services, and vMotioning a workload between an on-premises Datacentre and the public cloud. One of the interesting features for me was “Elastic DRS”- this harnesses the flexibility of the public cloud and allows vCenter to automatically, dynamically, expand and contract the size of a cluster by adding and removing hosts on the fly as workload requirements change. This could deliver that real promise of an elastic SDDC capable of handling “bursty” loads smoothly, it’s also got the potential of landing you a big bill at the end of the month so hopefully there’s some checks and balances in the configuration.

The Power Hour: Deep Dive, DevOps, and New Features of PowerCLI [INF8092]

Always a fun experience, this session didn’t disappoint. With a star-wars-esque intro with voice over by Alan Renouf, and further sci-fi references added throughout by himself and Luc Dekens, it offered an in depth look into some of the new happenings in the PowerCLI world without making anyone’s head hurt with the complexity.  Announced this morning was the release of PowerCLI-core which offers the opportunity to run PowerCLI on a Linux (including PhotonOS) or Mac platform.

There’s also the announcement of VMware (no longer “vSphere”) PowerCLI 6.5 which is offering many new features including a switch to modules from the traditional snapins, cross-vCenter vMotion using Move-VM (even to AWS!), and improvements around Virtual Disk Management and Horizon View controls. There’s too much to mention it all here, so I suggest keeping a lookout for the session online.

What’s New with vSphere [INF8375R]

With the new vSphere 6.5 announcement this session was really full, with people sitting on the floor round the sides of the room. The session served as a jumping-off platform for the more in-depth sessions this week, but summarised the new features in Migration, User Interfaces, Encryption, ESXi lifecycle, Secure Boot and the Universal App Platform to enable containers within vSphere.


The All New vSphere 6.5


  1. A new version of VMware vSphere, 6.5, will be released shortly
  2. Migration/Upgrade tools from previous versions (including Windows vCenter) to new VCSA.
  3. VCSA Native High Availability
  4. VCSA Integrated VMware Update Manager
  5. Native vCenter Backup and Restore
  6. Improved Appliance Management
  7. vSphere Clients
  8. Encryption

New vSphere coming soon

VMware has bucked the trend in versioning adopted by other major software companies and decided not to call it’s new vSphere version “10” and opted for the more traditional “vSphere 6.5” to succeed version 6.0 which was originally released back in March 2015. Announced at VMworld Europe 2016 with GA to follow, vSphere 6.5 is a continuation of the product which forms the core of the Software Defined Datacentre chunk of VMware’s “Any Cloud” Cross-Cloud Architecture portfolio. A lot of work has been put into making the experience of installing and operating a vSphere virtualised environment easier; Ignoring any improvements under the hood, and just looking at what’s on the surface there’s a whole bunch of features designed to make life run smoother for the IT Professional, some of which are highlighted in this post.

The new vCenter Server Appliance is a core part to this simplicity, and VMware have answered the requirements of anyone currently sticking to the Windows-based vCenter. If you can get more features and more reliability for less cost and less effort then it’s definitely the way forwards in my opinion. Some of the features discussed here- notably Native HA and Backup/Restore- will only be available in the appliance version of vCenter.

VCSA Upgrade and Migration


image Again out to both simplify the life of IT Professionals and encourage vCenter Appliance adoption, VMware has put a lot of effort into creating straightforward, and comprehensive, upgrade and migration tools. As more and more operations and data are handled by vCenter it becomes more and more important that the system can be smoothly navigated from version to version with minimal human effort.

Migrations are possible from Windows vCenters running version 5.5 or 6.0, and both the embedded and external database topologies are supported. Additionally, the new vCenter will assume the identity of the old Windows vCenter so any external interfaces, scripts, and automation should continue to work post-migration.

VCSA Native High Availability

VCSA 6.5 offers a built-in high availability deployment taking away the need for any 3rd party clustering or database solutions. The appliance deploys as an active/passive pair (plus witness) which automatically sets up replication of the integrated database and required vCenter files. The basic setup option also places these nodes intelligently using DRS and SDRS technology and automatically creates the necessary affinity rules and private IP comms, keeping everything simple. For infrastructures with unique and challenging topologies, there’s still an advanced workflow that can be used.


Integrated VMware Update Manager

Prior to 6.5 using VUM to manage the patching of a vSphere infrastructure based on the vCenter Appliance has been, how can we put it?, “annoying”. After deploying the slick appliance it was then necessary to spin up (and license) a separate Windows VM just to handle the update system. This requirement has been removed in the new version- VUM is now integrated into the VCSA, enabled by default, and shares the same database instance. The new VUM integration also leverages the VCSA High Availability and Backup functionality.

Native vCenter Backup and Restore

Also new to the vCenter Server Appliance is integrated backup and restore functionality. A great step forward in the simplification of deploying a system this provides a built in solution to backup vCenter to an external location (SCP, SFTP, HTTPS locations for example) and then be able to recover by deploying a clean OVA and choosing the Restore option. image


Improved Appliance Management and Monitoring

The vCenter Server Appliance Management Interface- VAMI – has also had a makeover, with many features being added. The 6.0 version had an interface limited to changing IP and NTP settings, rebooting the appliance, and little else. 6.5 adds in built in monitoring of Network, CPU, Memory and the vPostgres database. There is also the option to configure Syslog for deeper external monitoring of the vCenter infrastructure- this allows fully verbose logs to be kept for auditing and troubleshooting processes.


vCenter Server Appliance 6.0 Management Interface


vCenter Server Appliance 6.5 Management Interface

vSphere Client(s)

Work continues to focus on delivering a fully functioned HTML5 client, but in the interim vCenter 6.5 will come shipped with a new (limited) HTML5 based “vSphere Client”- evolved from the current fling – as well as an improved flash based “vSphere Web Client”. Expect the “vSphere Client” to see continuous improvement and feature addition through the lifetime of the platform –driven through the Fling programme.


As with the other topics here encryption in the new vSphere could easily be a post in itself (or a whole series), but to summarise the new features in this area, vSphere is now offering built-in VM encryption. The encryption happens between the VM and the storage so is invisible to the guest.

Local keys are generated within vSphere, and encrypted using keys held in an external (third-party) KMS- this would usually be managed by the IT Security team. Back in vCenter encryption is implemented through Storage Policies, so a VM can be encrypted simply by assigning the correct policy to it. Through the GUI (or API/PowerCLI) it’s possible to set  encryption covering  the Disks, the VMX/Swap files, or the whole lot on a per-VM basis. Through the API/PowerCLI it’s also possible to arrange encryption on a per-VHD level, potentially encrypting different disks on a VM with different keys.

VSAN encryption is on the way- there’s currently an ongoing beta – but will not be available in the 6.5 release. Based on the recent cadence I’d expect to see something in Spring 2017, but that’s just my speculation.


In summary, there’s lots to look for in the new vSphere release and in particular the vCenter Server Applicance. This week’s VMworld should reveal a lot more in depth into these advances.

VMworld Europe

VMworld Europe 2016 Day 1

WP_20161017_09_39_34_Pro_LIMonday is always a quieter day at VMworld Europe; the Keynotes kick the event off fully tomorrow and most of the sessions on Monday are reserved for Partners and TAM clients. However, there’s still plenty going on for us regular attendees.


Sunday night once again saw vRockstar provide an “unofficial” opening to the event. Thanks to the team and sponsors (Rubrik, Nutanix, Veeam, and the rest) for putting the event on, it was great once again to see faces old and new- several who I hadn’t seen in person for a whole year.


WP_20161017_08_15_49_Pro_LI (2)Back over at Fira Grand Via, my Monday agenda featured a couple of sessions, plus the opportunity to explore the Hang Space and Hands-On-Labs area. Here’s a quick run down of some of the content, in all of my posts this week I’m planning on giving the highlights, and linking into deeper-dive pages (or session videos where available) so you can have a closer look if something piques your interest.


Hands-on Lab: SPL-1703-SDC-1: VMware NSX Feature Tour

I worked through the first couple of modules of this, and intend to come back to look at the rest after the event is over. It’s aimed at people looking at NSX and starting a deployment from scratch. The first module was a click-through interactive slideshow of a basic NSX deployment, getting the components installed on a virtual infrastructure. This was extended in the second module where an existing deployment was used to work through adding a logical switch, connecting up some workloads, and checking connectivity. I’d recommend having a look if you have an interest in NSX and aren’t sure where to start. (To find the lab, visit and search for “HOL-1703-SDC-1”)


Virtual Networking: Moving from Hype to Reality with Deep Traffic Analytics [MGT8486-QT]

Staying on the NSX theme, this was a quick-talk format session which formed a useful introduction both on a technical level but also helped me warm up to the more intense sessions coming later in the week. A half-hour long version of an hour-long presentation it showed how important analytics is to making sure a virtual network is correctly configured and traffic is running as it should.


Hands-on Introduction to VMware Cloud Foundation [ELW-1799-FEL-1]

Cloud Foundation was announced at VMworld US in August (plus an announcement of an AWS partnership last week) and this was an ideal opportunity to dive in and get acquainted with some of the new technology through a Hands-On-Lab, with the added benefit of a knowledgeable guide to take the group through it. The product “integrates vSphere, VSAN, and NSX into a single unified stack” that can be run both in a private cloud on-premises but also on a public cloud provider – or both through a elastic hybrid cloud model.

This lab skips the initial setup of the Cloud Foundation and dives straight in with a guided walkthrough of the user interface followed by a practical look at creating and destroying work domains- essentially the independent vSphere clusters carved from the stack and given to a particular task (VDI, generic compute, management etc). A simulated version of this lab is available here – .


Roll on tomorrow, with the “General Session” keynote kicking off the day at 9am. I’m expecting some big announcements there so it should be good.

Finding Deprecated VMFS Volumes

Following an upgrade to vSphere 6, hosts popped up an error message reporting that some old VMFS volumes were found on the host. Whilst these still worked, it would be prudent to tidy them up and replace them with modern datastores. The message is:

“Deprecated VMFS volume(s) found on the host. Please consider upgrading volume(s) to the latest version”


Whilst this highlights that there is an issue, the GUI message doesn’t indicate which volumes (or how many) are affected. The following quick bit of PowerCLI produces a table of the datastores with a VMFS version of less than 5.

Get-Datastore |
 Where-Object { $_.ExtensionData.Info.VMFS.MajorVersion -lt 5} |
 Select-Object Name, FreeSpaceGB, CapacityGB, @{n='VMFS Version';e={$_.ExtensionData.Info.VMFS.MajorVersion}}

Example Output:

Name           FreeSpaceGB CapacityGB VMFS Version
----           ----------- ---------- ------------
VOL-VM01            574.77    2047.75            3
VOL-VM02           1924.05    2047.75            3
VMworld Europe

Travel Changes for #VMworld Europe 2016

For those that have previously been to VMworld (or other conferences) at the Fira Grand Via Barcelona venue there’s a few changes to transport this year. If you’ve walked round the conference centre from the “Europa Fira” metro station to the North entrance used by VMworld (and formerly TechEd Europe) in the past couple of years you’ll have spotted the closed entrance to the metro station ten minutes closer to the front door. Well good news- it’s now open!

The new metro station is served by an all new line that links the airport terminals to the conference venue- as shown in this tuneful video from TMB Barcelona.

So, what does this all mean for VMworld attendees?

Firstly, there are still Airport Bus Shuttles running on Sunday, Monday, and Thursday (see for times). But if you arrive/leave outside these times and want to travel between the airport and the conference venue, you can use this new “L9 Sud” metro line and an Airport ticket.

Secondly, the “Metro shuttles” that have run in previous years from the “Europa Fira” metro station round to the North entrance are not running this year- attendees are advised to use the L9 Sud metro line from “Europa Fira” to “Fira” instead (or you could take the 10 minute walk). Remember- this won’t cost extra on your metro ticket from the city centre because it will still count as the same journey

Finally, if you have a metro map of Barcelona from a previous trip, it’s worth downloading a new one.

Gran Via South Entrance

The South Entrance at Fira Gran Via by the Europa Fira metro station. VMworld is at the other end.

Walking round Fira

On a nice day it’s a pleasant walk from the South entrance to the North one.

North Entrance

The North Entrance to Fira Gran Via in Barcelona

Setting EVC Level on a Cluster with a VCSA and no shared storage


A second host has been added to a one-node vSphere cluster which has no shared storage and EVC cannot be enabled. Details as follows:

  • A vSphere environment consists of an existing cluster containing a single host running ESXi 6. On this host is (amongst other things) the vCenter Server Appliance VM.
  • A second ESXi6 host is added and joined it to the vCenter. This host has older hardware with an older EVC level (rather than the usual situation where the newer box has newer hardware).
  • EVC cannot be enabled on the cluster because the old host does not have the capabilities of the new host. The old host has VMs running which may be using the enhanced capabilities of the higher EVC level, one of these is the vCSA.

    “The host cannot be admitted to the cluster’s current Enhanced vMotion Compatibility mode. Powered-on or suspended virtual machines on the host may be using CPU features hidden by that mode.”

    Therefore the EVC level on the existing cluster cannot be lowered by powering off all the VMs because it cannot be changed without using vCenter.

  • The vCSA cannot be vMotioned to the old host because this requires EVC to be enabled.

    “The virtual machine requires hardware features that are unsupported or disabled on the target host”

  • There is no Shared Storage. VMs are stored on local datastores. The Knowledgebase article “How to enable EVC in vCenter Server (1013111)” has a solution but this doesn’t (as far as I can tell) work without shared storage. Without storage visible from both hosts the VM cannot be disabled in one host and brought back up in a second which is in a new cluster.

Possible Solutions

The problem boils down to needing to cold migrate a VM between ESXi hosts without using shared storage or vCenter. The following solutions came to mind.

  1. Create some shared storage (possibly using an NFS share on a laptop temporarily) and follow the procedure shown in KB1013111
  2. Power down the vCSA, use the host web client to move the files from the datastore to a laptop, then back up to the less-able host. Power it on and set EVC on the cluster.
  3. Dump the vCSA and setup a replacement instance on the less-able host. Reconfigure everything. The “Start Again” option.
  4. Use SCP to do a host-host local datastore transfer of the powered down and unregistered vCSA Virtual Machine files

My Chosen Solution

This is what I tried and tested and it worked in my environment, along with step-by-step instructions if anyone else finds themselves in this predicament (usual disclaimer applies).

Option 4:  Use SCP to do a host-host local datastore transfer of the powered down and unregistered vCSA Virtual Machine files.

Rough steps – “First Host” is the existing box with newer hardware, “Second Host” is the box with older hardware being added:

  1. Using vCSA setup a new cluster containing just the second host (the one with older hardware) and turn on EVC appropriately.
  2. Enable Secure Shell access on both hosts
  3. Shutdown all VMs including the vCSA on the first host
  4. Remove vCSA from inventory (Unregister) using web client on first host
  5. SSH into second host
  6. Enable SCP through the firewall with
    esxcli network firewall ruleset set -e true -r sshClient
    -thanks for that snippet
  7. Use SCP to copy VM files from local datastore on first host to local datastore on second host.
    For example- in SSH session on second host, something like this:
    mkdir /vmfs/volumes/datastore1/LABVC1
    scp ‘root@*.*’ /vmfs/volumes/datastore1/LABVC1/
  8. Connect to web client on second host and register the copy of the VMX file to inventory
  9. Turn on the vCSA VM. When prompted say “I moved it”
  10. Wait for vCSA to spin up then move the first host into the cluster with the second host.
  11. Tidy up: -Remember to go back and delete the old copy of the VCSA from the datastore on the first host and disable SSH on both hosts if it’s not required. The vCSA can be Storage-vMotioned to rethin disks if they inflated during the SCP operation.

VMworld US 2016 Day 1 Keynote

Yesterday saw the 2016 VMworld US formally kick off with the Keynote address. There are 23,000 attendees at the Las Vegas event, and I was amongst the many armchair supporters watching the live online broadcast.

imageThe keynote opened with drumming, poetry, and a lightshow, but it wasn’t long before Pat Gelsinger took the stage for his 5th VMworld as CEO of VMware. His presentation involved plenty of studying the past to predicting the future- “be_Tomorrow” is the tagline for the event.

Comparisons were made between the industrial revolution and todays’ digital revolution, and in more recent history we saw quotes from Eric Schmidt- the birth of the cloud- in 2006, and Ragu Raghuram on the SDDC in 2011. Some stats on the growth of the cloud vs traditional IT since 2006 were shown, workloads on a private/public cloud growing from 2% to 27% of the total in the last 10 years.

VMware’s predictions of “tomorrow” are interesting (and possibly a little too precise). Gelsinger told us to the minute when his team of experts have predicted that 50% of all workloads will be in the cloud.

Next came the real meat of the Keynote- the introduction of VMware’s latest cross cloud architecture- “Any Cloud, Any Device” using the upcoming products “Cloud Foundation” and “Cross Cloud Services”. Last year’s event had the tagline “Ready for Any”, perhaps this is finally being realised?

We heard examples from partner IBM and customer Marriot, before Guido Appenzeller took to the stage to dive deeper into the products.

The new Cloud products will allow IT to move workloads between an on-premises vCenter-based SDDC, AWS, Google Cloud, and Microsoft Azure. This opens up this possibility of bursting into various public cloud offerings when your workload demands it, but also taking that AWS-developed application and siting it appropriately across one or more locations- providing that security and resilience which IT is tasked with providing to the business.


The key here is security- and it’s doubtless no coincidence that the first guest speaker in this segment was from the banking industry- it’s not just the ability to vMotion between different providers “mega-clouds” but also to implement the rest of the SDDC architecture. We saw a demo using NSX to overlay network virtualisation, microsegmentation, and encryption to an AWS application.

One thing that struck me was the way vCloud Air was hardly mentioned. VMware has clearly shifted direction here and is not trying to compete with the AWS/Google Cloud/Azure marketplace but looking more to work with it. This also helps from an IT point of view- we don’t have to turn round to developers and say “Don’t use AWS, learn how to use vCloud Air instead”

imageThe Keynote was rounded off with Pat Gelsinger talking with Michael Dell. The Dell/EMC takeover deal was announced back at VMworld Europe 2015 in October, and currently looks like it’s almost ready to actually happen! As expected, Michael was quick to reassure everyone that it was business as usual for VMware and this Dell ownership wasn’t going to mean the end of the line for those partners who might be Dell competitors. He highlighted how the open partner ecosystem of VMware is critical to it’s continued success and mentioned the new Cloud Foundations and how “only an ecosystem of this size and power could pull this off.”

Both Dell and VMware seem keen to make private clouds easy to deploy, but are also looking to heavily push the public cloud alongside it. The next few years could be interesting, let’s have a look a cloud adoption at about four in the afternoon on June 29th 2021 and see what happened.

Full video of the session can be found on the VMworld website or below.