Tag Archives: backup

RTO With Cohesity @ vRetreat

How Cohesity’s Approach to VM Backup Affects the Recovery Time Objective

This week I attended another vRetreat online, this time featuring data vendor Cohesity who I saw presenting at the (in-person) event last year. These are great events, and the small panel of delegates works well in the virtual format.

One thing that stood out to me in their presentation was the focus on the Recovery Time Objective (RTO)- in essence how long it takes to recover from an incident. In this post I will briefly discuss how I understand the definition of RTO before looking at how the Cohesity products work to keep this time down when working with Virtual Machines.

Recovery Time Objective

There’s plenty of material out on the interwebs which will explain RTO in great detail, but I’m taking the definition to be:

the expected length of time between an incident occurring and users being able to work normally again

As this diagram shows, the Time can be split into a number of notable sections, I’ve chosen the following three:

RTO

  1. Discovering the Incident. How long is it before we notice something is broken? Do we have to wait for a user to contact the service desk, or do we have responsive monitoring and alerting in place?
  2. Starting the Restore. How long does it take to actually start the restore operation? Is there a clear process to be followed? There might be internal decisions to be made as to whether to kick off a backup restore or attempt an in-place repair. Does somebody need to physically power on some equipment or find and load some tapes before a backup restore can commence?
  3. The Restore Operation. How long does it take between “Go” being pushed on the restore console and the service being usable again?

You’ll notice there’s also a fourth section on the diagram- the “Tidy Up”. This is all those processes that need to happen after the user is working again to get the system back into a normal state. This might include things like tidying up the original (broken) copies of the VM, returning a backup tape to the library, or investigation of the root cause. In any of these cases, I’ve put this step outside of the RTO as by the definition above, the Users are working normally again.

Ransomware Detection

imageRecovery from ransomware attacks seem to be the current favoured feature pushed by backup vendors, and Cohesity are no exception. Their take here is that because the Cohesity Data Platform handles all the backups, it sees all the data and this position in the data flow gives the rest of the Cohesity stack an opportunity to spot both when an unusual number of files have been changed and also when files suddenly can’t be indexed because they’ve been encrypted.

Tied with an alerting mechanism, this helps address our question in point 1 above- “Can we discover the incident quickly?”. The sooner someone in IT is aware that a ransomware infection has happened, the quicker a response can be started.

Additionally, Regular point-in-time snapshot backups make it easier to spot the time the infection started (or if not the point of infection, at least when the malware started acting) and the more granular the timestamps the less data is potentially lost between a backup and the incident. But we’re straying into RPO, not RTO, there.

Starting Restore

Most of the time when responding to a major incident and orchestrating a restore operation the user interface will be key to assessing the situation and bringing services back online. Cohesity offers a clean and tidy web-based UI, complete with the now-obligatory Dark Mode.

2020-07-09_21-58-27

Whilst the platform isn’t going to make those go/no-go decisions on kicking off a restore- it can influence that decision. Because the restores are so quick (as we’ll see shortly) the discussion on whether to repair or restore might favour the latter. It’s also possible to bring up the VMs in a network-disconnected state without touching the production systems so that once any discussions are complete the restore is even quicker (or if the repair option is chosen then that restore can just be cancelled)

Restoring User Service

Once recovery is started in Cohesity Data Protect an NFS datastore is created on the Data Platform- the VMDK is already here so there is no need to spend time at this point moving blocks across the network. The NFS datastore is mounted within vCenter and the VM registered and at this point the VM can be powered on and the users can get working again.

Once service has been restored, the longer process of putting the VM files back where they belong is achieved with the hypervisors own Storage vMotion technology (the fourth step above). Applications are available throughout this, and once the Cohesity datastore has been cleared, it is unmounted from vCenter.

As this slide extract from the Cohesity presentation shows, one of their big selling points is this quick recovery process. Notice how the “Recover data to target storage device” is positioned after the User access is restored.

image

Thanks to Patrick Redknap and the Cohesity team for hosting this informative event, and I look forward to the next one. For more information about Cohesity, check out their website: https://www.cohesity.com/

Please read my standard Declaration/Disclaimer and before rushing out to buy anything bear in mind that this article is based on a sales discussion at a sponsored event rather than a POC or production installation. I wasn’t paid to write this article or offered any payment, although Cohesity did sponsor a prize draw for delegates at the event.

Check Azure WebApps have Backup Configured

Azure WebApps (depending on tier) come with an optional native backup service. This quick PowerShell snippet looks at all the WebApps in the current subscription and reports back on whether Backup has been set up. This should be helpful for spotting where a configuration has been missed.

Use Set-AzContext to set the subscription in advance, and to restrict to an individual Resource Group use the –ResourceGroupName on the Get-WebApp cmdlet in the first line.

foreach($WebApp in Get-AzWebApp ){
  if (Get-AzWebAppBackupConfiguration `
      -ResourceGroupName $WebApp.ResourceGroup `
      -Name $WebApp.Name `
      -ErrorAction SilentlyContinue) {
  $WebApp.Name+" Backup Configured"
  } else {
  if( (Get-Error -Last 1).Exception.Response.Content `
      -like "*Backup configuration not found for site*")
    {$WebApp.Name+" Backup Not Configured"}
 }
}

Cohesity Marketplace

During the recent #vRetreat event in London, Cohesity presented their latest release of DataPlatform – and with a launch happening the very weekend of the event, February 26 2019 this was timely presentation. This release included a number of new features- and when following up on the vRetreat event one which caught my attention is the Cohesity Marketplace.

The Marketplace is designed to allow third parties (plus your internal developers and Cohesity themselves) to release products that plug directly into the Cohesity framework- “bringing applications to the data, versus data to the applications”. From what I have seen of previous integrations they have been focussed on automating the backup/recovery process- for example using ServiceNow to provide end-users with self-service restores. This marketplace however allows third party applications to interact with and process the data on the Secondary Storage directly, without it leaving the appliance (or the public cloud storage). I see this as an interesting development, and visiting the website today you can get an idea of how this is going to grow.

Already in the list are analytics providers such as Splunk and Antivirus/ Threat Protection providers such as SentinelOne and ClamAV. The potential here for not just data protection but also analysis and business intelligence operations is intriguing- all that old, dark, data that companies hold but don’t make use of should be in this secondary storage and the ability to tap into that directly opens up many possibilities.

image

This all sits alongside a new Developer Portal and the existing REST API and PowerShell frameworks provided for the DataPlatform. Apps can be developed in-house but the big benefit I see is the third-party products being presented to admins to deploy- simplifying the traditional method of liaising with all the vendors in your environment separately to try and achieve a level of integration. And because the data is being processed within the Cohesity platform there’s the benefits of additional security, less duplicated storage, reduced network costs, and potentially better performance because we’re not spending time shifting data around to process it.

It’s early days yet so there’s only a handful of apps available (Mid March 2019), but it will be interesting to see how this develops and whether the work developing apps falls to Cohesity or will partners and third-party vendors take up the mantle.

For more information, check out this video from Cohesity.


Vendor Brief: Rubrik

clip_image001Rubrik have become the new “cool kid” of backups and in the past couple of years I’ve seen lots of enthusiasm in the community for the product. To reinforce this their stand at the Barcelona show featured a Mercedes Formula 1 car, their marketing giveaways include custom LEGO figures based on Mr Men books (you can’t not be cool with that combination), and they topped it all when they gave away 1000 copies of the Host Resources Deep Dive book by Niels Hagoort and Frank Denneman to attendees. Aside from the swag Rubrik have made impressive steps in the Gartner Magic Quadrant and received awards at VMworld US last month. Continue reading

The Home Lab

Virtual Machine Backups with Veeam

So, I’ve got my HomeLab on a NUC setup and running the VMware ESXi 6 Hypervisor. Before I get too carried away “experimenting” I want to be a good sysadmin and ensure that I can backup (and restore) my Virtual Machines. Even though on the whole they are test/dev environments there is still plenty of work going into them that I either don’t want to lose during that endeavour, or want to archive and possibly return to at another time. I’ve chosen to start with the Veeam Backup Free Edition v9. This powerful yet free tool offers the ability to archive an entire VM into a single compressed “VeeamZIP” file and restore the VM (or individual files from the Guest OS) to the same hypervisor or a different location. Continue reading