ESXi Host Patching with PowerShell & Update Manager

Update Manager certainly makes host patching simple, but leaves a few things to be desired. How many times have you attempted to update a host in Update Manager only to have the host never enter maintenance mode because of a DRS rule, VMware tools installation or local ISO mapped to a VM? I wanted to find a way to check for all these things as I’m performing the patching process and be able to accomplish it at the cluster level.

For the script itself I have broken it down into the different sections along with screenshots of what you’ll see when running the script. It makes it a little busy to follow along with for this entry, but hopefully it makes sense. At the bottom of the page I have the whole script put together to make it easier to copy and run it on your own.

Let’s dig into the script.

1. While you can manually define the vCenter server in the script, I prefer being prompted as I have multiple vCenter servers that I work with. The multiple lines and color emphasis was for a customer that would forget to enter the vCenter name and instead enter the ESXi host name.

Write-Host "Enter the FQDN of the " -NoNewline
Write-Host "[vCenter Server]" -ForegroundColor Red -NoNewline
Write-Host " to connect to: " -NoNewline
$vCenterFqdn = Read-Host
Connect-viserver $vCenterFqdn

2. Here we’re going to list all the clusters. I use this menu system all the time now in my PowerShell scripts to make it easier to make selections instead of having to remember and manually enter the name of an object. This is getting all the clusters then converting the number selection that’s entered into the cluster name.

$global:i=0
Get-Cluster | Sort Name | Select @{Name="Number";Expression={$global:i++;$global:i}},Name -OutVariable menu | format-table -AutoSize
$clusterNum = Read-Host "Select the number of the Cluster to be patched"
$clusterName = $menu | where {$_.Number -eq $clusterNum}


3. Now that we have the cluster we’re going to work with we search for DRS rules. Specifically, we’re looking for “Must Run” rules. This will prevent a VM from moving to another host. While every environment is different and they have “must run” rules for a variety of reasons, I’m comfortable disabling this during patch events. If there are any rules we’re going to list the rule names in the PowerShell console and give you the option to disable or not.

a. Remember, this is only looking at “Must Run” DRS rules for the entire cluster, not for an individual host. If you’re patching, odds are you’ll be doing the entire cluster anyway so I didn’t break this down on a host-by-host basis.

$drsRules = Get-Cluster $($clusterName.Name) | Get-DrsVMHostRule | Where {$_.Type -eq "MustRunOn"} | Where {$_.Enabled -eq $True}
IF ($drsRules.Count -gt "0"){Write-Host "The following rules may prevent a host from entering Maintenance mode:" -foreground "Yellow"; $drsRules.Name; $disableRules = Read-Host "Press Y to disable these rules. Anything else to continue";
IF ($disableRules -eq "Y"){Write-Host "Disabling DRS Rules..." -foreground "Yellow";
foreach ($name in $drsRules){Set-DrsVMHostRule -rule $name -enabled:$false}} ELSE {Write-Host "Skipping disabling of DRS Rules. Continuing..." -foreground "Yellow"}} ELSE {Write-Host "No "Must Run" Rules in $($clusterName.Name). Continuing..." -foreground "Yellow"}

In the picture I have the name of the DRS rule highlighted (the VM name was in the rule so it’s been obscured).


4. Now that we’ve decided what to do with our DRS rules, we can get down to selecting the baseline. This script can be used for both patching and for Upgrades. There is a check later on in the script that will skip the “Staging” step and go right to remediation if it’s an upgrade. Once again, we’re using that menu selection function to display all upgrades/baselines and let us choose the one to use.

$global:i=0
Get-Baseline | Select Name | Sort Name | Select @{Name="Number";Expression={$global:i++;$global:i}},Name -OutVariable menu | format-table -AutoSize
$baselineNum = Read-Host "Select the number of the Baseline to be attached"
$baselineName = $menu | where {$_.Number -eq $baselineNum}
Write-Host "Attaching $($baselineName.Name) Baseline to $($clusterName.Name)..." -Foreground "Yellow"
$baseline = Get-Baseline $baselineName.Name
Attach-Baseline -Baseline $baseline -Entity $clusterName.Name


5. Here’s where we’re going to complicate things a bit. I have 2 loops in this script. Loop number 1 is for checking if a host has any patches available. We’ll check a selected host against the attached baseline, if there are no available updates/upgrades then we report that in the PowerShell console and return to the host selection screen. The second loop is when a selected host has been patched we return to the host selection screen to choose the next one in the list.

DO
{
DO
{

6. Now that we’ve opened up our loop, we can start with selecting a host in the cluster. Once again, menu selection, this time we’re getting all the hosts in the chosen cluster and we’re displaying the host name, build, esxi version, and state. This makes it easier to know what hosts have been patched, which ones are still left, and what hosts are already in maintenance mode. In a larger environment you may forget what host name you were working on so seeing if a host was in maintenance mode and ready to be upgrade may be beneficial.

$global:i=0
Get-Cluster $clusterName.Name | Get-VMhost | Sort Name | Select @{Name="Number";Expression={$global:i++;$global:i}},Name,Build,Version,State -OutVariable menu | format-table -AutoSize
$hostNum = Read-Host "Select the number of the Host to be patched"
$hostName = $menu | where {$_.Number -eq $hostNum}


7. With our first host chosen we’re going to scan its inventory to see what patches it currently has installed.

Write-Host "Scanning $($hostName.Name) patch inventory..." -foreground "Yellow"
Scan-Inventory -Entity $hostName.Name


8. Now that we’ve scanned it, we’re going to check it for compliance. If there are patches available, we’ll move on to the next step to see if there are any VMs with ISO or Vmware tools installations. If there aren’t any patches, we’re reporting that and then sending us back to the host selection screen.

a. As a note, the second ‘}’ after the “Write-Host ‘Host is out of date” command is to close the second loop from step 5.

Write-Host "Scanning $($hostName.Name) for patch compliance..." -foreground "Yellow"
$compliance = Get-Compliance $hostName.Name
IF ($compliance.Status -eq "Compliant"){Write-Host "No available patches for $($hostName.Name). Choose a different host" -foreground "Red"}ELSE{Write-Host "Host is out of date" -foreground "Yellow"}}
UNTIL ($compliance.Status -ne "Compliant")


9. Now that we have some patches to apply, we check for active VMware tools installations. We perform the lookup for VMs with tools installer mounted then we perform a count on that output. If there are more than 0, we list all the VMs. Now that you see all the VMs, you can press ‘Y’ to force the unmount and continue or you can ignore it and hope the VMs move.

a. The unmount command works most of the time, but on some Linux OS’s I’ve run into issues with it. Just keep that in mind

$vmtools = Get-VMHost $hostName.Name | Get-VM | Where {$_.ExtensionData.RunTime.ToolsInstallerMounted -eq "True"} | Get-View
IF ($vmtools.Count -gt "0"){Write-Host "The following VMs on $($hostName.Name) have VMTools Installer Mounted:";
$vmtools.Name;
$unmountTools = Read-Host "Press "Y" to unmount VMTools and continue. Anything else to skip VMTools unmounting";
IF ($unmountTools -eq "Y") {Write-Host "Unmounting VMTools on VMs..." -foreground "Yellow"; foreach ($vm in $vmtools) {$vm.UnmountToolsInstaller()}}ELSE{Write-Host "Skipping VMTools unmounting..." -foreground "Yellow"}}ELSE{Write-Host "No VMs found with VMTools Installer mounted. Continuing..." -foreground "Yellow"}


10. With all our VMware tools installations killed, we move on to ISOs. ISO’s that are stored in shared datastores won’t have an issue moving, but if ISOs have been mounted directly to a VM through a console window those can cause a hang up. Again, you know your environment better than me so use your best judgement when picking what to do.

$mountedCDdrives = Get-VMHost $hostName.Name | Get-VM | Where { $_ | Get-CdDrive | Where { $_.ConnectionState.Connected -eq "True" } }
IF ($mountedCDdrives.Count -gt "0"){Write-Host "The following VMs on $($hostName.Name) have mounted CD Drives:";
$mountedCDdrives.Name;
$unmountDrives = Read-Host "Press "Y" to unmount these ISOs and continue. Anything else to skip ISO unmounting";
IF ($unmountDrives -eq "Y") {Write-Host "Unmounting ISOs on VMs..." -foreground "Yellow"; foreach ($vm in $mountedCDdrives) {Get-VM $vm | Get-CDDrive | Set-CDDrive -NoMedia -Confirm:$False}}ELSE{Write-Host "Skipping ISO unmounting..." -foreground "Yellow"}}ELSE{Write-Host "No VMs found with ISOs mounted. Continuing..." -foreground "Yellow"}


11. Now we check if the host is in maintenance mode. This check isn’t required and we could just try to put a host in maintenance mode that’s already in maintenance mode without any errors, I just prefer to have this called out so people know that the host will be placed in maintenance mode. Also, if you don’t want to confirm and just want the host to automatically go into maintenance mode, you can remove the “Read-Host “Press Enter to place $($hostName.Name in Maintenance mode”;” section and it will automatically place the host in maintenance mode.

$hostState = Get-VMHost $hostname.Name
IF ($hostState.State -eq "Maintenance"){Write-Host "$($hostName.Name) is already in maintenance mode. Continuing to patch Staging/Remediation" -foreground "Yellow"}ELSE{Read-Host "Press Enter to place $($hostName.Name) in Maintenance mode"; Start-Sleep 7; Write-Host "Enabling Maintenance mode for $($hostName.Name). This may take a while..." -foreground "Yellow"; Set-VMHost $hostName.Name -State "Maintenance"}


12. This was an interesting issue I ran into. I had a customer running ESXi 6.0 with PernixData installed which wasn’t compatible with ESXi 6.5 which we were upgrading to. When we attempted to upgrade we’d fail because the PernixData VIB was present. I threw this check in to see if this VIB existed on their hosts and to remove it before proceeding. I also added a second placeholder VIB name in case you have multiple VIBs to remove you can just replace the name with the appropriate VIB name and even add additional VIBs with another -OR $_.ID -eq “vibname”

$esxcli = Get-esxcli -vmhost $hostName.Name
$vibCheck = $esxcli.software.vib.list() | Where {($_.ID -eq "PernixData_bootbank_pernixcore-vSphere6.0.0_3.5.0.2-39793" -OR $_.ID -eq "Other_vib_name_xxxxxx")}
IF ($vibCheck.Count -gt "0"){Write-Host "Incompatible VIB found. Removing from host..." -foreground "Yellow"; foreach ($a in $vibCheck){$esxcli.software.vib.remove($null, $true, $false, $true, $a.Name)}}ELSE{Write-Host "No known incompatible VIBs found. Continuing..." -foreground "Green"}


13. And, of course, if removing a VIB we need to reboot so now we throw this reboot check in there as well. If there were no VIBs found in Step 12, this will be ignored. Otherwise, we prompt for reboot, enter the reboot command, check for the host to enter the NotResponding state and report on the state until it responds in vCenter and returns to Maintenance state.

IF ($vibCheck.Count -gt "0" -AND $baseline.BaselineType -eq "Upgrade"){Read-Host "VIBs were removed from host. Press enter to reboot host before attempting upgrade";Restart-VMhost $hostName.Name -confirm:$false}ELSE{$skip = "1"; Write-Host ""}
IF ($skip -ne "1"){
Write-Host "$($hostName.Name) is going to reboot..." -foreground "Yellow"
do {
Start-Sleep 3
$hostState = (get-vmhost $hostName.Name).ConnectionState
}
while ($hostState -ne "NotResponding")
Write-Host "$($hostName.Name) is currently down..." -foreground "Yellow"

#Wait for server to reboot
do {
Start-Sleep 5
$hostState = (get-vmhost $hostName.Name).ConnectionState
Write-Host "Waiting for $($hostName.Name) to finish rebooting..." -foreground "Yellow"
}
while ($hostState -ne "Maintenance")
Write-Host "$($hostName.Name) is back up..." -foreground "Yellow"}ELSE{Write-Host ""}

14. Now that all that work is done, we can start staging patches. If this is a patch baseline we run stage command. If it’s an upgrade baseline, we’ll skip this step

IF ($baseline.BaselineType -eq "Upgrade"){Write-Host "$($baseline.Name) is an Upgrade Baseline. Skipping to remediation..." -foreground "Yellow"}ELSE{Write-Host "Staging patches to $($hostName.Name) in Cluster $($clusterName.Name)..." -foreground "Yellow"; Stage-Patch -entity $hostName.Name -baseline $baseline}


15. Once patches have been staged (or upgrades ready to push) it’s time for remediation. We prompt that the host will reboot on its own once the patch has completed and we set a few advanced options. These are the defaults, but can still be environment specific so check to make sure this is what you want to use.

Write-Host "Remediating patches on $($hostName.Name) in Cluster $($clusterName.Name). Host will reboot when complete" -foreground "Yellow"
Remediate-Inventory -Entity $hostName.Name -Baseline $baseline -HostFailureAction Retry -HostNumberofRetries 2 -HostRetryDelaySeconds 120 -HostDisableMediaDevices $true -ClusterDisableDistributedPowerManagement $true -ClusterDisableHighAvailability $true -confirm:$false -ErrorAction SilentlyContinue


At the top of our PowerShell window we get the percentage of completion for our task. It’s not very accurate as it stays at 30% then goes to 92% when it’s nearly complete.

16. Once the host has been rebooted and comes back online we want to see the current status of that host to ensure updates were successful. We are comparing the build number we grabbed before we started patching against the build number after the reboot. If they are the same, something didn’t work and we need to check into it. Otherwise, we do nothing.

Write-Host "Retrieving Host build status..." -foreground "Yellow"
$hostBuild = Get-VMHost $hostName.Name
IF ($hostBuild.Build -eq $hostState.Build){Write-Host "Patch/Upgrade was not applied. Check status in vCenter and re-run the script. Exiting..." -foreground "Red";$error;Start-Sleep 20;break}ELSE{}

17. Now that the host was patched, we show a list of all the hosts in that cluster along with their build, version, and state. This gives us a full view of the cluster so we can see if there are any hosts left to be patched and then we exit maintenance mode for this host.

Get-Cluster $clusterName.Name | Get-VMhost | Select Name,Build,Version,State | Sort Name | format-table -autosize
Write-Host "Exiting Maintenance mode for Host $($hostName.Name)..." -foreground "Yellow"
Get-VMHost $hostName.Name | Set-VMHost -State Connected


18. Based on that list will determine the answer to our next question. We are being prompted to re-enable the DRS rules we previously disabled (if any). If any rules were chosen to be disabled we captured that in a variable in step 3. We can choose to re-enable just those disabled rules by pressing ‘Y’ or if there are other hosts left to patch we just press any other key to continue.

IF ($disableRules -eq "Y") {$enableRules = Read-Host "If Cluster patching is complete press "Y" to re-enable DRS rules. Anything else to continue";
IF ($enableRules -eq "Y") {Write-Host "Re-enabling DRS Must Run rules" -foreground "Yellow"; 
foreach ($name in $drsRules){Set-DrsVMHostRule -rule $name -enabled:$true}} ELSE {
Write-Host "DRS Rules not being re-enabled. Continuing..." -foreground "Yellow"}} ELSE {}


19. In this last question we’re just displaying the output from our last host patched and prompting the user to quit patching or go back to step 6 and pick the next host in the cluster to patch.

$answer = Read-Host "$($hostname.Name) patched in Cluster $($clusterName.Name). Press "1" to re-run the script. Anything else to exit"


20. Finally, to close out the first loop, we have the following lines. In step 19 we have the variable $answer which asks the user to enter ‘1’ to re-run the script and pick another host. This line at the bottom is saying until the user enters something other than 1, keep performing that loop. If anything else is entered, the script exits. Answering “1” will start the script over from Step 6. We will perform another “Get-Cluster | Get-VMHost” on the chosen cluster and retrieve the current build and state information for each of the hosts and display the updated results. As you can see from the screenshot below, vmm-04 is no in a Connected state with a Build number of 9298722,

}
UNTIL ($answer -ne "1")


Below is the script all put together to copy and test. Like all scripts pulled from the internet, make sure you test them in a lab/isolated environment until you can ensure proper functionality.

Write-Host "Enter the FQDN of the " -NoNewline
Write-Host "[vCenter Server]" -ForegroundColor Red -NoNewline
Write-Host " to connect to: " -NoNewline
$vCenterFqdn = Read-Host
Connect-viserver $vCenterFqdn

$global:i=0
Get-Cluster | Sort Name | Select @{Name="Number";Expression={$global:i++;$global:i}},Name -OutVariable menu | format-table -AutoSize
$clusterNum = Read-Host "Select the number of the Cluster to be patched"
$clusterName = $menu | where {$_.Number -eq $clusterNum}

$drsRules = Get-Cluster $($clusterName.Name) | Get-DrsVMHostRule | Where {$_.Type -eq "MustRunOn"} | Where {$_.Enabled -eq $True}
IF ($drsRules.Count -gt "0"){Write-Host "The following rules may prevent a host from entering Maintenance mode:" -foreground "Yellow"; $drsRules.Name; $disableRules = Read-Host "Press Y to disable these rules. Anything else to continue";
IF ($disableRules -eq "Y"){Write-Host "Disabling DRS Rules..." -foreground "Yellow";
foreach ($name in $drsRules){Set-DrsVMHostRule -rule $name -enabled:$false}} ELSE {Write-Host "Skipping disabling of DRS Rules. Continuing..." -foreground "Yellow"}} ELSE {Write-Host "No "Must Run" Rules in $($clusterName.Name). Continuing..." -foreground "Yellow"}

$global:i=0
Get-Baseline | Select Name | Sort Name | Select @{Name="Number";Expression={$global:i++;$global:i}},Name -OutVariable menu | format-table -AutoSize
$baselineNum = Read-Host "Select the number of the Baseline to be attached"
$baselineName = $menu | where {$_.Number -eq $baselineNum}
Write-Host "Attaching $($baselineName.Name) Baseline to $($clusterName.Name)..." -Foreground "Yellow"
$baseline = Get-Baseline $baselineName.Name
Attach-Baseline -Baseline $baseline -Entity $clusterName.Name

DO
{
DO
{
$global:i=0
Get-Cluster $clusterName.Name | Get-VMhost | Sort Name | Select @{Name="Number";Expression={$global:i++;$global:i}},Name,Build,Version,State -OutVariable menu | format-table -AutoSize
$hostNum = Read-Host "Select the number of the Host to be patched"
$hostName = $menu | where {$_.Number -eq $hostNum}

Write-Host "Scanning $($hostName.Name) patch inventory..." -foreground "Yellow"
Scan-Inventory -Entity $hostName.Name

Write-Host "Scanning $($hostName.Name) for patch compliance..." -foreground "Yellow"
$compliance = Get-Compliance $hostName.Name 
IF ($compliance.Status -eq "Compliant"){Write-Host "No available patches for $($hostName.Name). Choose a different host" -foreground "Red"}ELSE{Write-Host "Host is out of date" -foreground "Yellow"}}
UNTIL ($compliance.Status -ne "Compliant")

$vmtools = Get-VMHost $hostName.Name | Get-VM | Where {$_.ExtensionData.RunTime.ToolsInstallerMounted -eq "True"} | Get-View
IF ($vmtools.Count -gt "0"){Write-Host "The following VMs on $($hostName.Name) have VMTools Installer Mounted:";
$vmtools.Name;
$unmountTools = Read-Host "Press "Y" to unmount VMTools and continue. Anything else to skip VMTools unmounting";
IF ($unmountTools -eq "Y") {Write-Host "Unmounting VMTools on VMs..." -foreground "Yellow"; foreach ($vm in $vmtools) {$vm.UnmountToolsInstaller()}}ELSE{Write-Host "Skipping VMTools unmounting..." -foreground "Yellow"}}ELSE{Write-Host "No VMs found with VMTools Installer mounted. Continuing..." -foreground "Yellow"}

$mountedCDdrives = Get-VMHost $hostName.Name | Get-VM | Where { $_ | Get-CdDrive | Where { $_.ConnectionState.Connected -eq "True" } }
IF ($mountedCDdrives.Count -gt "0"){Write-Host "The following VMs on $($hostName.Name) have mounted CD Drives:";
$mountedCDdrives.Name;
$unmountDrives = Read-Host "Press "Y" to unmount these ISOs and continue. Anything else to skip ISO unmounting";
IF ($unmountDrives -eq "Y") {Write-Host "Unmounting ISOs on VMs..." -foreground "Yellow"; foreach ($vm in $mountedCDdrives) {Get-VM $vm | Get-CDDrive | Set-CDDrive -NoMedia -Confirm:$False}}ELSE{Write-Host "Skipping ISO unmounting..." -foreground "Yellow"}}ELSE{Write-Host "No VMs found with ISOs mounted. Continuing..." -foreground "Yellow"}

$hostState = Get-VMHost $hostname.Name
IF ($hostState.State -eq "Maintenance"){Write-Host "$($hostName.Name) is already in maintenance mode. Continuing to patch Staging/Remediation" -foreground "Yellow"}ELSE{
#Read-Host "Press Enter to place $($hostName.Name) in Maintenance mode"; Start-Sleep 7; Write-Host "Enabling Maintenance mode for $($hostName.Name). This may take a while..." -foreground "Yellow"; Set-VMHost $hostName.Name -State "Maintenance"}
Write-Host "Enabling Maintenance mode for $($hostName.Name). This may take a while..." -foreground "Yellow"; ; Start-Sleep 7; Set-VMHost $hostName.Name -State "Maintenance"}

$esxcli = Get-esxcli -vmhost $hostName.Name
$vibCheck = $esxcli.software.vib.list() | Where {($_.ID -eq "PernixData_bootbank_pernixcore-vSphere6.0.0_3.5.0.2-39793" -OR $_.ID -eq "Other_vib_name_xxxxxx")}
IF ($vibCheck.Count -gt "0"){Write-Host "Incompatible VIB found. Removing from host..." -foreground "Yellow"; foreach ($a in $vibCheck){$esxcli.software.vib.remove($null, $true, $false, $true, $a.Name)}}ELSE{Write-Host "No known incompatible VIBs found. Continuing..." -foreground "Green"}

IF ($vibCheck.Count -gt "0" -AND $baseline.BaselineType -eq "Upgrade"){Read-Host "VIBs were removed from host. Press enter to reboot host before attempting upgrade";Restart-VMhost $hostName.Name -confirm:$false}ELSE{$skip = "1"; Write-Host ""}
IF ($skip -ne "1"){
Write-Host "$($hostName.Name) is going to reboot..." -foreground "Yellow"
do {
Start-Sleep 3
$hostState = (get-vmhost $hostName.Name).ConnectionState
}
while ($hostState -ne "NotResponding")
Write-Host "$($hostName.Name) is currently down..." -foreground "Yellow"

#Wait for server to reboot
do {
Start-Sleep 5
$hostState = (get-vmhost $hostName.Name).ConnectionState
Write-Host "Waiting for $($hostName.Name) to finish rebooting..." -foreground "Yellow"
}
while ($hostState -ne "Maintenance")
Write-Host "$($hostName.Name) is back up..." -foreground "Yellow"}ELSE{Write-Host ""}

IF ($baseline.BaselineType -eq "Upgrade"){Write-Host "$($baseline.Name) is an Upgrade Baseline. Skipping to remediation..." -foreground "Yellow"}ELSE{Write-Host "Staging patches to $($hostName.Name) in Cluster $($clusterName.Name)..." -foreground "Yellow"; Stage-Patch -entity $hostName.Name -baseline $baseline}

Write-Host "Remediating patches on $($hostName.Name) in Cluster $($clusterName.Name). Host will reboot when complete" -foreground "Yellow"
Remediate-Inventory -Entity $hostName.Name -Baseline $baseline -HostFailureAction Retry -HostNumberofRetries 2 -HostRetryDelaySeconds 120 -HostDisableMediaDevices $true -ClusterDisableDistributedPowerManagement $true -ClusterDisableHighAvailability $true -confirm:$false -ErrorAction SilentlyContinue

Write-Host "Retrieving Host build status..." -foreground "Yellow"
$hostBuild = Get-VMHost $hostName.Name
IF ($hostBuild.Build -eq $hostState.Build){Write-Host "Patch/Upgrade was not applied. Check status in vCenter and re-run the script. Exiting..." -foreground "Red";$error;Start-Sleep 20;break}ELSE{}

Get-Cluster $clusterName.Name | Get-VMhost | Select Name,Build,Version,State | Sort Name | format-table -autosize
Write-Host "Exiting Maintenance mode for Host $($hostName.Name)..." -foreground "Yellow"
Get-VMHost $hostName.Name | Set-VMHost -State Connected

IF ($disableRules -eq "Y") {$enableRules = Read-Host "If Cluster patching is complete press "Y" to re-enable DRS rules. Anything else to continue";
IF ($enableRules -eq "Y") {Write-Host "Re-enabling DRS Must Run rules" -foreground "Yellow"; 
foreach ($name in $drsRules){Set-DrsVMHostRule -rule $name -enabled:$true}} ELSE {
Write-Host "DRS Rules not being re-enabled. Continuing..." -foreground "Yellow"}} ELSE {}

$answer = Read-Host "$($hostname.Name) patched in Cluster $($clusterName.Name). Press "1" to re-run the script. Anything else to exit"

}
UNTIL ($answer -ne "1")

Backing Up Portgroup Data with PowerShell and XML

Managing virtual standard switches (vSwitches) in a VMware environment, large or small, can cause a lot of headaches. Ensuring each portgroup is named the same, has the same VLAN and is present on each host can be a challenge. Virtual distributed switches (dvSwitches) exist to make our lives easier, but aren’t always available due to licensing or other restrictions. They have their own drawbacks, but the good generally outweighs the bad.

As I have been spending a significant amount of time in PowerShell and XML recently I wrote a script designed to backup the Portgroup configuration of all your clusters to ensure that all hosts will have the proper networking configuration during host rebuilds or additions. Let’s walk through this script and show you all the different elements.

1. Here we define and connect to the vCenter server

$vCenterFqdn = "vcenter01.domain.local"
Connect-viserver $vCenterFqdn

2. Here we are defining the path for our Portgroups.xml file. We’ll reference this variable later on.

$xmlNetworkPath="C:\Scripts\XML\Portgroups.xml"

3. Here we are checking to see if this file already exists. If it does we’ll skip ahead to the population side, but if not we’ll have to create it. $networkCheck tests the path defined in the variable $xmlNetworkPath. If that file is there, we’re writing into the console that it exists and we’re moving on. If it doesn’t exist we move on to step 4.

$networkCheck = Test-Path $xmlNetworkPath
IF ($networkCheck -eq $True){Write-Host "Portgroup file already exists. Continuing..." -foreground "Green"}
ELSE
{Write-Host "Portgroup file not created. Creating..." -foreground "Yellow"}

4. Since this is our first run, this file doesn’t exist yet so we need to create it. Populating an empty XML file is not something I ever figured out how to do. Someone much smarter than me in XML and PowerShell can figure it out, but I found a nice work around by creating an XML template. This is an outline of what our XML file will be and we’re just filling in the data as we go.

$xmlNetwork=[xml]@’ is saying everything between @’ and ‘@ will be part of this file. At the end, we’re saving this output ($xmlNetwork.Save) to the file path we defined earlier ($xmlNetworkPath).


$xmlNetwork=[xml]@'

<vSwitchConfig>
  <templates>
    <Portgroup>
      <vlanId></vlanId>
      <virtualSwitch></virtualSwitch>
      <cluster></cluster>
    </Portgroup>
  </templates>
  <Portgroups>
    <test />
  </Portgroups>
</vSwitchConfig>
‘@
$xmlNetwork.Save($xmlNetworkPath)

5. In case we run into permissions issues we want to perform an additional check to ensure that file was actually written. For the most part we’re just repeating Step 3. The difference here is we’re changing the output in “Write-Host” but also if the file isn’t created, we need to kill the script. The message is written that the file was not created, then we wait 20 seconds, then exit the script. The reason we add the “Start-Sleep” is in case this script is run by just double-clicking and not triggered from within a powershell window, the script will exit and you’ll never know why.

$networkCheck = Test-Path $xmlNetworkPath
IF ($networkCheck -eq $True){Write-Host "Portgroup file created successfully. Continuing..." -foreground "Green"}
ELSE
{Write-Host "Portgroup file not created. Exiting..." -foreground "Red"; Start-Sleep 20; Break}}

6. Now we need to read the contents of that XML file we created in XML format. The [XML] denotes that this is an XML file. $xmlNetwork is just the variable name I chose (this can be anything you want) and $xmlNetworkPath is the path to the file we defined in step 2

[XML]$xmlNetwork = Get-Content $xmlNetworkPath

7. Now comes the fun part where we actually start pulling data from vCenter. We’re going to get all the clusters in the vCenter we’re connected to. The ForEach command says that for every cluster we need to run this same command. So in each cluster we’re looking for an ESXi host that is currently connected, then we choose a random host from the cluster (get-random) and on that random host we get all the virtual portgroups that aren’t on dvSwitches.

$getClusters = Get-Cluster
ForEach ($cluster in $getClusters) {
$getPortgroups = $cluster | Get-VMHost | Where {$_.ConnectionState -ne "NotResponding"} | Get-Random | Get-VirtualPortGroup | Where {$_.key -notlike "*dvportgroup*"}

8. Now that we have all these portgroups let’s get all the data we need in them. We perform another ForEach on every Portgroup that was on that host. The $testPg variable is used to see if the current Portgroup name ($pgName.Name) is present in the XML file. What we don’t want is to re-add the same Portgroup name over and over again, we just want a single reference to the portgroup name and we’ll add another entry for the clusters that contain it. $testPg -eq $null means if the portgroup name isn’t there, we’ll create a new entry.

foreach ($pgName in $getPortGroups) {
$testPg = $xmlNetwork.vSwitchConfig.Portgroups.Portgroup|Where {$_.name -eq $pgName.Name}
IF ($testPg -eq $null)

9. If the Portgroup isn’t there we need to create a new entry. Looking at the XML file outline we created in Step 4 you see the tag. This is used to work around my inability to create new entries in XML. We copy the section below and then fill out the data as we go


<Portgroup>
  <vlanId></vlanId>
  <virtualSwitch></virtualSwitch>
  <cluster></cluster>
</Portgroup>

The variable “$xmlNetwork” was defined in Step 6 and is the name of our XML file. The addition of “.vSwitchConfig” is the top level tag in our XML file from Step 4.
The variable “$parentNode” defines the top-level of the XML file we’ll be using from our template.
The variable “$destinationNode” defines where this copied data is going to be placed.
The variable “$cloneNode” defines what tag we’re copying from $parentNode.
The variable “$addNameAttribute” is creating an attribute called “name” and the name we’re giving it is the name of the portgroup we’re pulling from vCenter. Adding “.Value = ” to the end of this variable gives us the ability to define the name when we’re creating the new portgroup tag.
The variable $newNode is used to create the new Portgroup tag and then “.InnerXML =” allows us to chose the template tag from the XML file.
(The use of [void] is something I don’t fully grasp. This is the only way it works, but I can’t tell you why.)
Using $destinationNode.AppendChild is saying place this new tag into the $destinationNode location. $newNode is the data that’s being added. “.Attributes” allows us to assign the new name Attribute we defined and “.Append($addNameAttribute)” places that attribute tag.
The variable $updateNetwork is a lookup in the XML file to find a portgroup with the tag we just created.
Once we find that portgroup, $updateVLAN, $updateVirtualSwitch, and $updateCluster are used to assign the values to these empty tags that were created.


{
$parentNode = $xmlNetwork.vSwitchConfig.templates
$destinationNode = $xmlNetwork.vSwitchConfig.Portgroups
$cloneNode = $parentNode.SelectSingleNode("Portgroup")
$addNameAttribute = $xmlNetwork.CreateAttribute("name")
$addNameAttribute.Value = $pgName.Name
$newNode = $xmlNetwork.CreateElement("Portgroup")
$newNode.InnerXML = $cloneNode.InnerXML
[void]$destinationNode.AppendChild($newNode).Attributes.Append($addNameAttribute)
$updateNetwork = ($xmlNetwork.vSwitchConfig.Portgroups.Portgroup|Where {$_.name -eq $pgName.Name})
$updateVLAN = $updateNetwork.vlanId = $pgName.VLanId.ToString()
$updateVirtualSwitch = $updateNetwork.virtualSwitch = $pgName.VirtualSwitchName
$updateCluster = $updateNetwork.cluster = $cluster.Name
}

10. In the event this portgroup already exists, we have the ELSE portion of our IF statement we started in Step 8. Here we’re referencing the portgroup in question and checking to see if the cluster tag has already been added. We look up the portgroup name “$testPg” and search for a cluster that matches the cluster we’re currently working with. If that cluster doesn’t exist, we add a new element ($xmlNetwork.CreateElement(“cluster”)) and populate its value ($addCluster.InnerText = $cluster.Name)

If that cluster tag already exists for that Portgroup, we don’t do anything “{}”


ELSE {($testCluster = $testPg|Where {$_.cluster -eq $cluster.Name});
IF ($testCluster -eq $null) {
$addCluster = $xmlNetwork.CreateElement("cluster");
$addCluster.InnerText = $cluster.Name
$testPg.AppendChild($addCluster) | Out-Null }ELSE{}}
}}

11. Once all of our clusters and Portgroups have been looped through. We need to save that data to the XML file we created.

$xmlNetwork.Save($xmlNetworkPath)

12. Now that the file has been saved, we want to see what data has been written. This isn’t required, but just adds a nice little output to the screen so you can see that the data has been populated as expected.

We perform the same Get-Content command with our [XML] tag. Then we write the output of every portgroup, sorted by name, into a table so we can see all the data.


[XML]$xmlNetwork = Get-Content $xmlNetworkPath
Write-Host "The following Portgroup configuration exists in $($xmlNetworkPath)" -foreground "Yellow"
$xmlNetwork.vSwitchConfig.Portgroups.Portgroup | Sort name | Format-Table

I have obscured some of the networks and VLANs in use in my environment, but this is the output you can expect.

All together this is what the script looks like:


#Connect to the defined vCenter Server
$vCenterFqdn = "vcenter01.domain.local"
Connect-viserver $vCenterFqdn

#Define XML File Path
$xmlNetworkPath="C:\Scripts\XML\Portgroups.xml"

#Check for Portgroup XML File
$networkCheck = Test-Path $xmlNetworkPath
IF ($networkCheck -eq $True){Write-Host "Portgroup file already exists. Continuing..." -foreground "Green"}
ELSE
{Write-Host "Portgroup file not created. Creating..." -foreground "Yellow"

#Create XML Portgroup File

$xmlNetwork=[xml]@'

<vSwitchConfig>
  <templates>
    <Portgroup>
      <vlanId></vlanId>
      <virtualSwitch></virtualSwitch>
      <cluster></cluster>
    </Portgroup>
  </templates>
  <Portgroups>
    <test />
  </Portgroups>
</vSwitchConfig>
'@
$xmlNetwork.Save($xmlNetworkPath)

$networkCheck = Test-Path $xmlNetworkPath
IF ($networkCheck -eq $True){Write-Host “Portgroup file created successfully. Continuing…” -foreground “Green”}
ELSE
{Write-Host “Portgroup file not created. Exiting…” -foreground “Red”;Start-Sleep 20; BREAK}}

#Get contents of XML file
[XML]$xmlNetwork = Get-Content $xmlNetworkPath

#Gather all hosts in each cluster
$getClusters = Get-Cluster
foreach ($cluster in $getClusters) {
$getPortgroups = $cluster | Get-VMHost | Where {$_.ConnectionState -ne “NotResponding”} | Get-Random | Get-VirtualPortGroup | Where {$_.key -notlike “*dvportgroup*”}
foreach ($pgName in $getPortGroups) {
$testPg = $xmlNetwork.vSwitchConfig.Portgroups.Portgroup|Where {$_.name -eq $pgName.Name}
IF ($testPg -eq $null) {
$parentNode = $xmlNetwork.vSwitchConfig.templates
$destinationNode = $xmlNetwork.vSwitchConfig.Portgroups
$cloneNode = $parentNode.SelectSingleNode(“Portgroup”)
$addNameAttribute = $xmlNetwork.CreateAttribute(“name”)
$addNameAttribute.Value = $pgName.Name
$newNode = $xmlNetwork.CreateElement(“Portgroup”)
$newNode.InnerXML = $cloneNode.InnerXML
[void]$destinationNode.AppendChild($newNode).Attributes.Append($addNameAttribute)
$updateNetwork = ($xmlNetwork.vSwitchConfig.Portgroups.Portgroup|Where {$_.name -eq $pgName.Name})
$updateVLAN = $updateNetwork.vlanId = $pgName.VLanId.ToString()
$updateVirtualSwitch = $updateNetwork.virtualSwitch = $pgName.VirtualSwitchName
$updateCluster = $updateNetwork.cluster = $cluster.Name
} ELSE {($testCluster = $testPg|Where {$_.cluster -eq $cluster.Name});
IF ($testCluster -eq $null) {
$addCluster = $xmlNetwork.CreateElement(“cluster”);
$addCluster.InnerText = $cluster
$testPg.AppendChild($addCluster) | Out-Null }ELSE{}}
}}
$xmlNetwork.Save($xmlNetworkPath)

#Display Results
[XML]$xmlNetwork = Get-Content $xmlNetworkPath
Write-Host “The following Portgroup configuration exists in $($xmlNetworkPath)” -foreground “Yellow”
$xmlNetwork.vSwitchConfig.Portgroups.Portgroup | Sort name | Format-Table

Convergence Without Compromise

Hyperconverged Infrastructure (HCI) gets a lot of attention these days, and rightly so. With HCI we’ve seen a move towards an easy-to-use, pay-as-you-grow approach to the datacenter that was previously missing. Complex storage array that required you to purchase all your capacity up front is what I started with in my career. While expansion of these storage arrays was possible, often times we were buying all the storage we’d need for 3-5 years even though we wouldn’t be consuming it for multiple years.

While HCI certainly made things easier it was far from perfect. Mixing storage and compute nodes into a single server meant maintenance operations needed to account for both available compute resources as well as available storage capacity to accomodate offline storage. At times we would actually sacrifice our data protection scheme in order to takes nodes offline and hope there were no additional failures within a cluster at the same time. Not ideal when we’re talking about production storage.

Get Down with the DVX

Datrium and the DVX platform aim to address these problems in an interesting way. Datrium separates storage and compute nodes much like traditional two tier system, but utilizes SSDs inside each of the hosts to act as a read cache. By moving the cache into the host we’re able to increase performance with every host we add. This decoupling of cache from the storage layer means we’re not queuing up reads at a storage array that is trying to satisfy the requests of all the connected hosts over the same connected switches. While this sounds very similar to previous technologies we’ve seen before (Infinio and PernixData come to mind), the differentiator is the storage awareness.

The Datrium DVX solution utilizes their own storage nodes for the persistent storage piece. With the caching and storage being fully aware of each other, Datrium is able to offer end-to-end encryption from the hypervisor down to the persistent storage while still being able to take advantage of deduplication and compression. Often times encrypting data at the storage array level means we are forced to give up these data efficiencies, but not in the case of Datrium. We get an additional level of data security without having to make any compromises.

No Knobs, No Problems

HCI vendors have really pushed the configuration abilities within their systems. Customers can choose what data is deduplicated and compressed, whether or not it should be encrypted, how many copies of their data should be kept, and is erasure coding a better choice than traditional RAID just to name a few. This is where Datrium separates itself from its HCI competitors. Disaggregating compute nodes from the persistent storage layer, Datrium’s DVX system manages to deliver performance and features without penalty. Once again, no compromises.

Erasure coding, dedupe and compression, double-device failure protection, data encryption; every one of these features is always on and doesn’t require any separate licensing or configuration. The advantage here isn’t just in administrative overheard, but also in performance. Datrium’s performance numbers are based on each one of these features enabled. No tricks. No Gimmicks. What you see is what you get; unlike many of their competitors that hide behind unrealistic configurations many of these features being disabled.

3 Tiers, 1 Solution

Datrium aims to bring together a Tier 1 HCI-like solution, combined with scale-out backup storage and Cloud-based DR all in the same system. With integrated snapshots that utilize VMware snapshots as well as VSS integration, they are able to perform crash consistent and application consistent snapshots of virtual machines right on the box. This, of course, is table stakes when it comes to modern storage arrays. The differentiator is that Datrium is able to do this at the VM-level despite presenting NFS to the virtual hosts. Now we’re not just backing up all the VMs that live in a LUN or volume, we’re able to get as granular as the virtual disk itself. No VVOLs required.

Adding another level of visibility into the mix, Datrium reports its latency at the individual Virtual Machine level instead of at the storage array. Traditional storage array vendors talk about their ultra-low latency, but this reported latency is what the array is seeing not taking into account the latency imposed by virtual hosts and switching infrastructure. With each different component in the virtual infrastructure having its own queues, varying utilization and available bandwidth, the latency a Virtual Machine experiences is much greater than what the array is reporting. Datrium is offering this full visibility at the individual Virtual machine level so you know how your environment is actually performing. Dr. Traylor from The Math Citadel has an excellent overview of queuing theory, Little’s Law, and the math behind it.

The cloud-based integrations also allows for an additional level of data availability. Instead of requiring an additional backup software, Datrium allows for replication of your data to a DVX running in the cloud. Now we have an offsite copy of your data ready to be restored in the event of VM corruption or deletion. Replication is also dedupe-aware, meaning data isn’t being sent to the cloud if it is already present helping to minimize bandwidth requirements and speeding up the replication process.

Cloudy Skies Ahead

While I am very reluctant to trust one solution with my primary and backup data, in certain situations I can see the advantages. Integrations with AWS allowing for virtual machines to be restored from the Cloud-based DVX means your DR site can now be in AWS. Datrium has lowered the barrier to the cloud for a lot of customers with the features they’ve included in the DVX platform.

Datrium continues to make a good product even better. The additional features available in version 4.0 of DVX make this not only a great fit for SMB customers, but enterprises as well. A feature-rich, no-knobs approach to enterprise storage with backup and DR-capabilities all rolled into one. Datrium is definitely worth a look.

________________________________________

Disclaimer: During Storage Field Day 15, my expenses (flight, hotel, transportation) were paid for by Gestalt IT. I am under no obligation by Gestalt IT or Datrium to write about any of the presented content nor am I compensated for such writing.

Removing Values in XML Using PowerShell

We’ve gathered values from our XML file and we’ve managed to add values to an XML file using PowerShell, but what about deleting? What I’m talking about is deleting XML tags and attributes, not deleting the value in an existing tag. Let’s look at our sample file and I’ll explain.

Above is our sample.xml file that was updated in the last post. Inside we have individual portgroups with a name and elements that we were added. In the context of the project I was working on, I needed the ability to delete a portgroup and all of its elements (VLAN, virtual switch, and cluster), but in some situations I needed to just delete the cluster. If a portgroup needed to be removed from one specific cluster, but remaining available to another cluster I couldn’t have the entire portgroup removed as it could affect virtual machines that are still running.

How do we do it? Let’s find out.

Just like before, let’s get the contents of our XML file so we can work with the data.

$xmlFile = "C:\Scripts\sample.xml"
$xmlConfig = [System.Xml.XmlDocument](Get-Content $xmlFile)

The two things we need in order to remove a portgroup are the name of the portgroup and the name of the cluster. Based on the sample.xml file, we see that “VM Network” is on two clusters (cluster1 and NestedCluster) so let’s start there. Let’s get some simple variables setup just like we did last time.

$name = Read-Host -Prompt "Enter the name of the Portgroup (ex: Management Network)"
$cluster = Read-Host -Prompt "Enter the name of the cluster for this Portgroup (ex: Cluster1)"

I’ve created a new powershell file called “removePortgroup.ps1” and it has the above variables in there prompting me to enter the name and the cluster of the portgroup to be removed.

Now that we have the XML contents and our variables defined it’s time to see what data is there. If you recall from my earlier post about getting data from XML, we can see the contents of our file by following the hierarchy. With the following command we can see all portgroups that have been defined.

$xmlConfig.config.portgroups.ChildNodes

Now that we know what’s there, we need to filter the results and only display the Web Network. We’ll pass this as a variable so we can use that info later.

$clusters = $xmlConfig.config.portgroups.ChildNodes | ? {$name -eq $_.name}

Using the same command as before but passing it as a variable, we added a parameter to search for the name of our Portgroup (VM Network) using the $name variable we created earlier and we are trying to match it with another portgroup of the same name. When we run the command “$clusters” we see what portgroup came back as a match. We can see the VM Network portgroup that has 2 clusters.

With this portgroup living on two clusters we are only interested in removing it from one which is NestedCluster. We’ll work on the logic for this later one, for now let’s focus on how you remove a single from this portgroup.

The following commands will remove the element that matches our $name and $cluster variables we created earlier.

$text = "#text"
$clusterTag = $xmlConfig.config.portgroups.portgroup | Where {$_.name -eq $name}
$updateTag = $clusterTag.SelectNodes("cluster") | Where {$_.$text -eq $cluster}
$updateTag.ParentNode.RemoveChild($updateTag)

We are looking for the Text inside of the tag. As you can see from the output at the bottom, it has a label (or whatever it is called) of #text. The problem is, when working within PowerShell, if I add a “#” to a line, it will comment out everything after it. We get around that limitation but creating a variable called $text and given it a value of #text. Maybe there is a better way of doing it, but this worked so I went with it.

The next line is looking for a portgroup matching $name. The next line is selecting the tag that matches $cluster. The final line is performing the removal of that tag. This still needs to be saved, so we’re not done yet, but at this point we have removed from . If you run the command below you can see that “NestedCluster” has been removed as a cluster in the “VM Network” portgroup.

$xmlConfig.config.portgroups.ChildNodes

Now what if this portgroup is only on one cluster and we want to delete it and all of its attributes completely? We can do so with the following command.

$xmlConfig.config.portgroups.ChildNodes | ? {$name -eq $_.name} | % {$xmlConfig.config.portgroups.RemoveChild($_)} | Out-Null

In this command, we are getting all the portgroups in the first part, then filtering the results where $name matches the name of an existing portgroup. Then we are removing that entire and all its contents from the XML file. The Out-Null is required to make it work, but I don’t really know why. Once again, this isn’t saved yet, we can do that in the next step.

In order to save we need to run the following command.

$xmlConfig.Save($xmlFile)

Now that this file is saved, we can re-check the XML file using the commands below

$xmlFile = "C:\Scripts\sample.xml"
$xmlConfig = [System.Xml.XmlDocument](Get-Content $xmlFile)

If we run the command $xmlConfig.config.portgroups.childnodes we will see that “VM Network” is now gone

We can remove just an attribute or we can remove an entire element. How can we create a script to determine which one to use? If we wrap this all in an IF/ELSE statement we can do it pretty easily.

Using the $clusters variable we created earlier, we can perform a count of how many tags there are. Running the following command will show many clusters a portgroup is provisioned to.

$clusters.cluster.count

In our IF/ELSE statement, we are checking to see if the count is greater than 1 (-gt 1) and then performing the command to remove just the cluster tag. If it’s not greater than 1, we are running the command to remove the entire portgroup and all of its attributes.

$clusters = $xmlConfig.config.portgroups.ChildNodes | ? {$name -eq $_.name}
IF ($clusters.cluster.count -gt 1){
$text = "#text"
$clusterTag = $xmlConfig.config.portgroups.portgroup | Where {$_.name -eq $name}
$updateTag = $clusterTag.SelectNodes("cluster") | Where {$_.$text -eq $cluster}
$updateTag.ParentNode.RemoveChild($updateTag)
$xmlConfig.Save($xmlFile) } ELSE {
$xmlConfig.config.portgroups.ChildNodes | ? {$name -eq $_.name} | % {$xmlConfig.config.portgroups.RemoveChild($_)} | Out-Null
$xmlConfig.Save($xmlFile) }

Notice how we see the same #text and “NestedCluster” output after the commands ran when we were just trying to remove a cluster? That’s because powershell sees there are more than 1 cluster assigned to the VM Network portgroup and it just removed it.

Here is the full script to check if there are multiple XML attributes named “cluster” and remove it or remove the entire portgroup element.

#Define XML file and Get its contents
$xmlFile = "C:\Scripts\sample.xml"
$xmlConfig = [System.Xml.XmlDocument](Get-Content $xmlFile)

#Define variables and prompt for input
$name = Read-Host -Prompt "Enter the name of the Portgroup (ex: Management Network)"
$cluster = Read-Host -Prompt "Enter the name of the cluster for this Portgroup (ex: Cluster1)"

#IF/ELSE Statement to remove a single attribute or an entire element
$clusters = $xmlConfig.config.portgroups.ChildNodes | ? {$name -eq $_.name}
IF ($clusters.cluster.count -gt 1){
$text = "#text"
$clusterTag = $xmlConfig.config.portgroups.portgroup | Where {$_.name -eq $name}
$updateTag = $clusterTag.SelectNodes("cluster") | Where {$_.$text -eq $cluster}
$updateTag.ParentNode.RemoveChild($updateTag)
$xmlConfig.Save($xmlFile) } ELSE {
$xmlConfig.config.portgroups.ChildNodes | ? {$name -eq $_.name} | % {$xmlConfig.config.portgroups.RemoveChild($_)} | Out-Null
$xmlConfig.Save($xmlFile) }

Adding Values to XML Using PowerShell

Continuing on from the previous post on getting values from an XML file using PowerShell, I’m going to talk about how to add values to an XML files. With the recent project I was working on I needed a way to easily add new portgroups into the XML configuration file. Every environment changes and while having something manually update an XML file would work there is always the possibility of messing up the syntax or something much worse.

Let’s have a look at the sample XML file again.

What we have here is 3 port groups (Management Network, VM Network, Web Network) with a VLAN, VirtualSwitch, and Cluster defined. The entire file sits inside the “config” tags. Each portgroup lives inside the “portgroups” tag, and the values for each portgroup live inside the individual “portgroup” tags.

The goal here is to add a brand new portgroup to this XML file so when we can then add that portgroup with the correct values to our ESXi hosts. Given our existing structure we’ll need a name, VLAN, Virtual Switch, and Cluster for this new portgroup.

In order to update our XML file we need to get the XML file contents and we do that by using the following command (again, using the file path as a variable so I can easily change it later).

$xmlFile = "C:\Scripts\sample.xml"
$xmlConfig = [System.Xml.XmlDocument](Get-Content $xmlFile)

Now that we have the contents of the XML file, we need to gather the name, VLAN, Virtual Switch, and Cluster for this new portgroup. The question you’ll have to ask yourself is do you want to update the script file with each new portgroup you need to add or do you just want the script to prompt for those values each time it’s run?

If you want to manually update the file and let it run by itself just define the following variables in your script. For this example we’re creating a new portgroup called “App Network” with VLAN 31, on vSwitch1 and on cluster2.

$name = "App Network"
$vlanID = "31"
$virtualSwitch = "vSwitch1"
$cluster = "cluster2"

When we need to add these values into our XML file, we will refer to them using $name, $vlanID, $virtualSwitch, and $cluster.

Let’s do the same thing, but where we’re prompted for the values each time.

$name = Read-Host -Prompt "Enter the name of the new Portgroup (ex: Management Network)"
$vlanId = Read-Host -Prompt "Enter the VLAN Id number for the Portgroup (ex: 13)"
$virtualSwitch = Read-Host -Prompt "Enter the name of the Virtual Switch for Portgroup (ex: vSwitch1)"
$cluster = Read-Host -Prompt "Enter the name of the cluster for this Portgroup (ex: Cluster1)"

I saved the variables above as a file named “addPortgroup.ps1” in the C:\Scripts directory. Running that script in PowerShell prompted for my input for each of those lines. Instead of updating a script file each time I need to add a new Portgroup, I can run the script and enter the values myself.

Now that we’ve recorded these values, let’s start using them.

First off we need to create a new portgroup tag since we’re adding a new portgroup. We use a variable name (in this case $newPortgroup) to define where this tag will live. $xmlConfig.config.Portgroups is the parent of each of the Portgroups we currently have. What we’re doing here is saying make a change to by adding a new element called .

$newPortgroup = $xmlConfig.config.Portgroups.AppendChild($xmlConfig.CreateElement("Portgroup"))

Once we do that, now we’re going to start adding those attributes we already defined. We’ll start with name.

$newPortgroup.SetAttribute("Name",$name)

In the new Portgroup element we created, we set an attribute for and set it to the value of $name (“App Network” for our example).

Now, we can repeat that process for each attribute just like above and save our work (I’ll show how to save in a later step), but if we do that our XML formatting may not be what we’re looking for.

On line 20 of our sample.xml file we see how this data gets saved.

As you can see each attribute is added on the same line. Maybe this works for you, and if so, great. For me, I wanted to maintain the existing formatting, so there is a little more work to do.

Now we need to add each remaining attribute as a child to the already created “App Network” portgroup. We do that with the following command.

$newvlanIdAttribute = $newPortgroup.AppendChild($xmlConfig.CreateElement("vlanId"))
$newvlanIdValue = $newvlanIdAttribute.AppendChild($xmlConfig.CreateTextNode($vlanId))

The first line creates a new attribute called “vlanId” that’s been added to our new then the second line set the value of that attribute to $vlanId (VLAN 31 for this example).

We can now add Virtual Switch and Cluster the exact same way.

$newvirtualSwitchAttribute = $newPortgroup.AppendChild($xmlConfig.CreateElement("virtualSwitch"))
$newvirtualSwitchValue = $newvirtualSwitchAttribute.AppendChild($xmlConfig.CreateTextNode($virtualSwitch))
$newclusterAttribute = $newPortgroup.AppendChild($xmlConfig.CreateElement("cluster"))
$newclusterValue = $newclusterAttribute.AppendChild($xmlConfig.CreateTextNode($cluster))

With all the new elements created and the values set, we need to save this configuration. If we close down PowerShell right now none of our updates are saved. Save using the following command

$xmlConfig.Save($xmlFile)

Now that we’ve updated the xml file, let’s see what it looks like In PowerShell. We’ll need to re-check the contents of the XML file as it changed since we lasted checked.

$xmlFile = "C:\Scripts\sample.xml"
$xmlConfig = [System.Xml.XmlDocument](Get-Content $xmlFile)

And now let’s look at $xmlConfig.config.portgroup.portgroups

And if we look at the contents of the XML file we can see that “App Network” matches the layout of our other portgroups

And here is what the script looks like all together:

#Define XML file and Get its contents
$xmlFile = "C:\Scripts\sample.xml"
$xmlConfig = [System.Xml.XmlDocument](Get-Content $xmlFile)

#Define variables and prompt for input
$name = Read-Host -Prompt "Enter the name of the new Portgroup (ex: Management Network)"
$vlanId = Read-Host -Prompt "Enter the VLAN Id number for the Portgroup (ex: 13)"
$virtualSwitch = Read-Host -Prompt "Enter the name of the Virtual Switch for Portgroup (ex: vSwitch1)"
$cluster = Read-Host -Prompt "Enter the name of the cluster for this Portgroup (ex: Cluster1)"

#Create a new XML element with the input entered above and save
$newPortgroup = $xmlConfig.config.Portgroups.AppendChild($xmlConfig.CreateElement("Portgroup"));
$newPortgroup.SetAttribute("Name",$name);
$newvlanIdAttribute = $newPortgroup.AppendChild($xmlConfig.CreateElement("vlanId"));
$newvlanIdValue = $newvlanIdAttribute.AppendChild($xmlConfig.CreateTextNode($vlanId));
$newvirtualSwitchAttribute = $newPortgroup.AppendChild($xmlConfig.CreateElement("virtualSwitch"));
$newvirtualSwitchValue = $newvirtualSwitchAttribute.AppendChild($xmlConfig.CreateTextNode($virtualSwitch));
$newclusterAttribute = $newPortgroup.AppendChild($xmlConfig.CreateElement("cluster"));
$newclusterValue = $newclusterAttribute.AppendChild($xmlConfig.CreateTextNode($cluster));
$xmlConfig.Save($xmlFile)

Getting Values from XML Using PowerShell

Working on a recent project I was tasked with maintaining VMware host configuration via XML files. The goal was to have a central location to modify the settings of a host (DNS, vmkernel interfaces, vswitch config, port groups) without having to update each host by hand. This lead me down the path of using PowerShell with XML files as the configuration source.

Below is the sample of my XML file.

What we have here is 3 port groups (Management Network, VM Network, Web Network) with a VLAN, Virtual Switch, and Cluster defined. The entire file sits inside the “config” tags. Each portgroup lives inside the “portgroups” tag, and the values for each portgroup live inside the individual “portgroup” tags.

Below is a sample command to get values from an XML file. We pass the file location as a variable and then we store all those values inside another variable to make working with the values easier.

$xmlFile = "C:\Scripts\sample.xml"
$xmlConfig = [System.Xml.XmlDocument](Get-Content $xmlFile)

And once we type in $xmlConfig into Powershell we can see that it is pulling in the data from our XML file.

We can see the “config” section, but how do we dig deeper? Type in $xmlconfig.config and let’s look at the output

Now we can see the “portgroups” section. Let’s go deeper and see what’s inside of that by typing the following:

$xmlconfig.config.portgroups

Nice! Now we can see all the portgroups we have defined. In order to get detail on each of these, we can type $xmlconfig.config.portgroups.portgroup and we’ll get a list of all the portgroups and all their values.

Now let’s say we are only concerned about retrieving values from one of those portgroups. We can add a Where clause to find only the Portgroups that exist on vSwitch1 as an example with the following command:

$xmlconfig.config.portgroups.portgroup | Where {$_.virtualswitch -eq "vSwitch1"}

Grabbing values out of an XML file is pretty straightforward. In the next post we’ll talk about how to add information to an XML file.

vSAN – Check VM Storage Policy & Compliance

As I continue to work with vSAN I discover there’s way more to do than just move some VMs over and you’re on your way. With multiple vSAN clusters each with different configurations I needed a way to monitor the current setup and check for changes. While creating a simple script to check which VM Storage Policy is assigned to each VM isn’t very difficult, a creating a script to check the storage policy of VMs across multiple vSAN datastores proved to be a little more difficult.

We run multiple PowerCLI scripts to check health and configuration drift (thanks to a special tool created by Nick Farmer) in our environment. In the event that a new vCenter is added or new vSAN datastore is deployed, we needed a simple script that can be run without any intervention or modification. Now we can be alerted when the proper VM storage policies isn’t assigned or the current policy is out of compliance.

To further complicate things in our setup, we create a new VM Storage Policy that contains the name of the cluster in which it’s assigned. Due to the potential differences in each vSAN cluster (stripes, failures to tolerate, replication factor, RAID, etc) having a single Storage Policy does not work for us. In the event a VM is migrated from one vSAN cluster to another we need to check that the VM storage policy matches vSAN datastore cluster policy.

What this script does is grab all the clusters in a vCenter that have vSAN enabled. For each cluster that is found with vSAN enabled, it is filtering only the VMs that live on vSAN storage (with the name of “-vsan”. Then we get the storage based policy management (Get-SpbmEntityConfiguration) of those VMs. The script then filters for a storage policy that doesn’t contain the cluster name OR a compliance status that is compliant.

$vsanClusters = Get-cluster | Where-Object {$_.vsanenabled -eq "True"}
foreach ($cluster in $vsanClusters)
{
$Cluster | get-vm |?{($_.extensiondata.config.datastoreurl|%{$_.name}) -like "*-vsan*"} |
Get-SpbmEntityConfiguration | Where-Object {$_.storagepolicy -notlike "*$Cluster*" -or $_.compliancestatus -notlike "*compliant*"} |
Select-Object Entity,storagepolicy,compliancestatus
}

Once this is run we can see the output below. I’ve obscured the names of the VMs, but we can see that there are still 12 VMs that are using the default vSAN Storage Policy instead of the cluster-specific storage policy they should be using. In addition, we see that the compliance status is currently out of date on most of these VMs. These VMs reside on 2 separate clusters and there are also 2 VMs that were filtered because they are on local storage in these clusters instead of vSAN.

storagepolicy01-12202016

Cohesity – DataPlatform in the Cloud

cohesityWhat separates vendors is focus and execution. In a crowded market, finding the right backup provider is no easy task. While each product has its pros and cons, finding the differentiator can be a daunting task. While Cohesity is relatively new to this space (founded in 2013), they have that focus and execution necessary to be a leader in the backup space.

But Cohesity is more than just backups. The Cohesity storage appliance not only handles your backup storage needs, but can also run your dev and test workloads. Cohesity is focused on your secondary storage needs. That secondary storage consists of any workloads or data that isn’t production. By avoiding the draw of being another primary storage vendor, Cohesity is listening to customers, learning their needs and creating a solution that can fit any size business.

storageiceberg

The Cohesity solution was built for a virtualized (VMware-only) environment. Connecting directly to your vCenter servers and pulling your inventory allowing administrators to create backup jobs and policies. While their start was in virtualization, there are still many physical workloads in the datacenter. Creating agents for physical Windows, Linux, and SQL server all backing up to the same storage system and with the same policies prove no workloads can’t be protected by Cohesity.

But wait, there’s more!

While data protection is important, that’s only a small portion of the Cohesity offering. Running these backups directly from the Cohesity storage arrays allows you to free up primary storage resources and (potential) bottlenecks when running multiple instances of the same VM on a single array. Leveraging the SSDs that come in each Cohesity node as a cache tier, testing software patches and deployments from your backed up production VMs means that your performance doesn’t suffer. And with a built in QoS engine your dev/test workloads don’t have to affect the speed of your backups.

Cohesity provides a scale-out solution, meaning as storage demand increases so can your secondary storage space. Operating under a single namespace, as new nodes are added, your space increases without needing to reconfigure jobs to point to a new array or manually re-striping data. Cohesity has customers that have scaled up to as much as 60 nodes with over a petabyte of storage.

To the cloud!

Policy-based backups and replication ensures that your data will be available. Cohesity has the ability to distribute data across the nodes in a cluster, replicate to clusters in another locations, and also replicate your data to a cloud provider in order to satisfy offsite backup requirements. The latest addition to the Cohesity software portfolio is the DataPlatform Cloud Edition. This gives you the ability to run Cohesity in the cloud.

DataPlatform CE is more than just replicating data to the cloud. Your VMs can be backed up to your on-premises cluster and that data can be replicated to your cloud-based array. From that cloud-based array, you can then clone virtual machines to a native cloud format. This means your servers can be run in the cloud in their native format and available to test or even run in the event of migrations or datacenter outages.

Many backup and data protection software vendors are doing replication to the cloud such as Veeam and Zerto. While the features isn’t new, its addition makes Cohesity a serious contender in this space. DataPlatform CE is available currently in tech preview in the Microsoft Azure Marketplace, but Cohesity hopes to release it in the first half of 2017 with support for Azure as well as AWS.

Wrapping Up

Data protection and availability is never going to be exciting. Swapping tapes and deploying agents is tedious work. A fully integrated software solution that not only protects your data, but also helps solve the problem of data sprawl, a platform for developers to test against production data in an isolated environment and the ability to migrate workloads to the cloud. That’s about as exciting as it gets in data protection and that is just the tip of the (storage) iceberg.

________________________________________

Take a look at posts by my fellow delegates from Tech Field Day 12 and watch the videos here.

First Look at Cohesity Cloud Edition
The Silent Threat of Dark Data
Cohesity Provides All of Your Secondary Storage Needs
Secondary Storage is Cohesity’s Primary Goal

________________________________________

Disclaimer: During Tech Field Day 12, my expenses (flight, hotel, transportation) were paid for by Gestalt IT. Cohesity provided each delegate with a gift bag, but I am under no obligation to write about any of the presented content nor am I compensated for such writing.

VSAN – Compliance Status is Out of Date

Occasionally the Compliance status of the performance service will go to the “out of date” status. This is not an alert that is thrown anywhere within vCenter. You will have to check this status by logging into the vSphere web client, locating your vCenter, choose the cluster, clicking on “Manage” then choosing “Health and Performance” under “Virtual SAN”
ComplianceStatus-a

As I have recently fixed this issue the above screenshot shows the “Compliant” status. Below are the steps to get to that point.

1. In the box for “Performance Service” click “Edit storage policy”
ComplianceStatus-01

2. If there is a storage policy available in the drop down, select it and click “OK”. This will apply that policy and perform the compliance check.
ComplianceStatus-02

For the lucky few where that works, that’s all you need to do. If the storage policy list is empty you’ll need to restart the vsanmgmtd service on each of the hosts.

3. Enable SSH on each of the hosts in the VSAN cluster and using an SSH client (like putty), SSH to a host and run the following command to restart the vsanmgmtd service (this is a non-impactful operation and should be able to be performed during production hours with no impact)
a. /etc/init.d/vsanmgmtd restart

4. Repeat that command on each of the hosts in the cluster until they have all restarted their services
ComplianceStatus-04

5. Wait 5 minutes and then check to see if you are able to select a storage policy for the performance service. If not, move on to step 6

6. Now we’ll need to restart the vSphere Profile-Driven Storage Service on the vCenter server. This is also non-impactful and should be able to be performed in the middle of the day. If you’re using vCenter on windows, connect to the Windows server and restart the “Vmware vSphere Profile-Driven Storage Service”. If using VCSA (like this example) you’ll need to SSH to the VCSA and run the command below
a. Service vmware-sps restart

7. After the vmware-sps service restarts, log out of the web client and wait for 5 minutes while the storage profile service completes its restart.

8. Log back in to the web client, navigate to the vCenter server, click “Manage” then choose the “Storage Providers” tab
ComplianceStatus-08

9. Click the Synchronize Providers button to resync the state of the environment
ComplianceStatus-09

10. Wait another 5 minutes while these synchronize completes. After 5 minutes, navigate to the VSAN cluster in the web client. Click on “Manage” then choose “Settings” and locate “Health and Performance” under the “Virtual SAN” section
ComplianceStatus-10

11. In the Performance Service box, click the “Edit Storage Policy” button
ComplianceStatus-11

12. From the drop down list you should be able to select the appropriate VSAN storage policy and then click “OK”
ComplianceStatus-12

13. After this is selected the compliance status should change to “Compliant” and you should be all set.

So far these are the only steps that I have needed to follow in order to fix this issue. Let me know if there are any other fixes available.

Deploy VSAN Witness Appliance

The VSAN Witness Host is a virtual appliance that is deployed into an existing vCenter server. When deploying a 2 node or stretched cluster, the witness appliance acts as a tie breaker to determine which node(s) are still available in the event the nodes lose communication with each other. The witness The witness is deployed just like any other virtual appliance, but will require access to the management network and the network you’ve designated as your VSAN network. This appliance must be run OUTSIDE your VSAN cluster. This means that you cannot add this host as a member of the existing VSAN cluster and you also should not run it as a virtual appliance inside your existing VSAN cluster.

1. Choose the cluster that will host the appliance. Click on “File” then “Deploy OVF Template”
step01

2. Browse to select the .OVA file and click “Next”
step02

3. Review the details of the appliance and click “Next”
step03

4. Review the license agreement and click “Accept” followed by “Next”
step04

5. Enter the name of the Witness Appliance and its location then click “Next”
step05

6. Choose the appropriate size of the appliance and click “Next”
step06
a. As this is a test, I’m choose the “Tiny” size. You can ignore the disk component requires for any size. As this is a virtual appliance, it will deploy the appropriately sized drives that will designated as SSD and spinning disl

7. Choose the provisioning type and click “Next”
step07
a. This is appliance is being deployed to a separate VSAN cluster than the one it will be acting as the witness for. This appliance can be deployed on shared storage, local storage, or another VSAN datastore.

8. Choose the appropriate networks for management and witness (VSAN). In this deployment, management lives on the “VM Network” and witness (VSAN) traffic is on the “VM-VSANnetwork”. This network is shared with the vMotion network and just needed an additional VM Portgroup created on each of the hosts in the cluster where this appliance is being deployed. Click “Next”
step08

9. Enter a root password for this appliance. Remember, this is a host that you will need to login to in order to administer so if there is a standard root password that you use it would be a good idea to use that here. Click “Next”
step09

10. Review the deployment settings and then click “Finish”
step10

11. Once deployed, you will need to configure the appliance like any other host. Power on the appliance and open the console, press F2 and login as root with the password you assigned in step 9
step11

12. Scroll to “Configure Management Network” and press “Enter”
step12

13. Ensure the Network Adapter assigned to your management network is “vmnic0”
step13
a. Set a VLAN (if necessary) for the management network, then assign your IPv4 and/or IPv6 settings for the management network to make it accessible on your network. Assigned DNS as needed as well. Press “ESC” and then press “Y” to configure settings and restart the management network
step13a

14. Once the host can communicate on the network, add it as a new host in vCenter.
a. Remember that this host should not be part of your VSAN cluster or any other cluster. It should be a standalone host in your datacenter.

15. Select the host in the vCenter client and configure networking for it. Locate the “witnessSwitch” and click “Properties”
step15

16. Select the “witnessPg” and click “Edit”
step16

17. On the “IP Settings” tab, enter the IP and subnet mask for the VSAN traffic network. Click “OK” at the bottom”
step17

18. Once you have confirmed that network settings are successful, login to the vSphere web client and navigate to the VSAN cluster to be configured

19. Click the “Manage” tab, then choose “Fault Domains & Stretched Cluster” under “Virtual SAN”
step19

20. In the “Streteched Cluster” box click “Configure”
step20

21. Name the fault domains and place the hosts into the appropriate fault domain. This is a 2 node cluster with 1 host in each fault domain. Click “Next”
step21

22. Locate the VSAN witness appliance host that was added to this vcenter and click “Next”
step22

23. Choose the flash drive for and the HDD for cache and capacity and click “Next”
step23

24. Review the settings and click “Finish”
step24

25. Once completed, you will now see the status of the stretched cluster as “Enabled”, the preferred fault domain and the designated witness host.
step25