ESXi Host Patching with PowerShell & Update Manager

Update Manager certainly makes host patching simple, but leaves a few things to be desired. How many times have you attempted to update a host in Update Manager only to have the host never enter maintenance mode because of a DRS rule, VMware tools installation or local ISO mapped to a VM? I wanted to find a way to check for all these things as I’m performing the patching process and be able to accomplish it at the cluster level.

For the script itself I have broken it down into the different sections along with screenshots of what you’ll see when running the script. It makes it a little busy to follow along with for this entry, but hopefully it makes sense. At the bottom of the page I have the whole script put together to make it easier to copy and run it on your own.

Let’s dig into the script.

1. While you can manually define the vCenter server in the script, I prefer being prompted as I have multiple vCenter servers that I work with. The multiple lines and color emphasis was for a customer that would forget to enter the vCenter name and instead enter the ESXi host name.

Write-Host "Enter the FQDN of the " -NoNewline
Write-Host "[vCenter Server]" -ForegroundColor Red -NoNewline
Write-Host " to connect to: " -NoNewline
$vCenterFqdn = Read-Host
Connect-viserver $vCenterFqdn

2. Here we’re going to list all the clusters. I use this menu system all the time now in my PowerShell scripts to make it easier to make selections instead of having to remember and manually enter the name of an object. This is getting all the clusters then converting the number selection that’s entered into the cluster name.

$global:i=0
Get-Cluster | Sort Name | Select @{Name="Number";Expression={$global:i++;$global:i}},Name -OutVariable menu | format-table -AutoSize
$clusterNum = Read-Host "Select the number of the Cluster to be patched"
$clusterName = $menu | where {$_.Number -eq $clusterNum}


3. Now that we have the cluster we’re going to work with we search for DRS rules. Specifically, we’re looking for “Must Run” rules. This will prevent a VM from moving to another host. While every environment is different and they have “must run” rules for a variety of reasons, I’m comfortable disabling this during patch events. If there are any rules we’re going to list the rule names in the PowerShell console and give you the option to disable or not.

a. Remember, this is only looking at “Must Run” DRS rules for the entire cluster, not for an individual host. If you’re patching, odds are you’ll be doing the entire cluster anyway so I didn’t break this down on a host-by-host basis.

$drsRules = Get-Cluster $($clusterName.Name) | Get-DrsVMHostRule | Where {$_.Type -eq "MustRunOn"} | Where {$_.Enabled -eq $True}
IF ($drsRules.Count -gt "0"){Write-Host "The following rules may prevent a host from entering Maintenance mode:" -foreground "Yellow"; $drsRules.Name; $disableRules = Read-Host "Press Y to disable these rules. Anything else to continue";
IF ($disableRules -eq "Y"){Write-Host "Disabling DRS Rules..." -foreground "Yellow";
foreach ($name in $drsRules){Set-DrsVMHostRule -rule $name -enabled:$false}} ELSE {Write-Host "Skipping disabling of DRS Rules. Continuing..." -foreground "Yellow"}} ELSE {Write-Host "No "Must Run" Rules in $($clusterName.Name). Continuing..." -foreground "Yellow"}

In the picture I have the name of the DRS rule highlighted (the VM name was in the rule so it’s been obscured).


4. Now that we’ve decided what to do with our DRS rules, we can get down to selecting the baseline. This script can be used for both patching and for Upgrades. There is a check later on in the script that will skip the “Staging” step and go right to remediation if it’s an upgrade. Once again, we’re using that menu selection function to display all upgrades/baselines and let us choose the one to use.

$global:i=0
Get-Baseline | Select Name | Sort Name | Select @{Name="Number";Expression={$global:i++;$global:i}},Name -OutVariable menu | format-table -AutoSize
$baselineNum = Read-Host "Select the number of the Baseline to be attached"
$baselineName = $menu | where {$_.Number -eq $baselineNum}
Write-Host "Attaching $($baselineName.Name) Baseline to $($clusterName.Name)..." -Foreground "Yellow"
$baseline = Get-Baseline $baselineName.Name
Attach-Baseline -Baseline $baseline -Entity $clusterName.Name


5. Here’s where we’re going to complicate things a bit. I have 2 loops in this script. Loop number 1 is for checking if a host has any patches available. We’ll check a selected host against the attached baseline, if there are no available updates/upgrades then we report that in the PowerShell console and return to the host selection screen. The second loop is when a selected host has been patched we return to the host selection screen to choose the next one in the list.

DO
{
DO
{

6. Now that we’ve opened up our loop, we can start with selecting a host in the cluster. Once again, menu selection, this time we’re getting all the hosts in the chosen cluster and we’re displaying the host name, build, esxi version, and state. This makes it easier to know what hosts have been patched, which ones are still left, and what hosts are already in maintenance mode. In a larger environment you may forget what host name you were working on so seeing if a host was in maintenance mode and ready to be upgrade may be beneficial.

$global:i=0
Get-Cluster $clusterName.Name | Get-VMhost | Sort Name | Select @{Name="Number";Expression={$global:i++;$global:i}},Name,Build,Version,State -OutVariable menu | format-table -AutoSize
$hostNum = Read-Host "Select the number of the Host to be patched"
$hostName = $menu | where {$_.Number -eq $hostNum}


7. With our first host chosen we’re going to scan its inventory to see what patches it currently has installed.

Write-Host "Scanning $($hostName.Name) patch inventory..." -foreground "Yellow"
Scan-Inventory -Entity $hostName.Name


8. Now that we’ve scanned it, we’re going to check it for compliance. If there are patches available, we’ll move on to the next step to see if there are any VMs with ISO or Vmware tools installations. If there aren’t any patches, we’re reporting that and then sending us back to the host selection screen.

a. As a note, the second ‘}’ after the “Write-Host ‘Host is out of date” command is to close the second loop from step 5.

Write-Host "Scanning $($hostName.Name) for patch compliance..." -foreground "Yellow"
$compliance = Get-Compliance $hostName.Name
IF ($compliance.Status -eq "Compliant"){Write-Host "No available patches for $($hostName.Name). Choose a different host" -foreground "Red"}ELSE{Write-Host "Host is out of date" -foreground "Yellow"}}
UNTIL ($compliance.Status -ne "Compliant")


9. Now that we have some patches to apply, we check for active VMware tools installations. We perform the lookup for VMs with tools installer mounted then we perform a count on that output. If there are more than 0, we list all the VMs. Now that you see all the VMs, you can press ‘Y’ to force the unmount and continue or you can ignore it and hope the VMs move.

a. The unmount command works most of the time, but on some Linux OS’s I’ve run into issues with it. Just keep that in mind

$vmtools = Get-VMHost $hostName.Name | Get-VM | Where {$_.ExtensionData.RunTime.ToolsInstallerMounted -eq "True"} | Get-View
IF ($vmtools.Count -gt "0"){Write-Host "The following VMs on $($hostName.Name) have VMTools Installer Mounted:";
$vmtools.Name;
$unmountTools = Read-Host "Press "Y" to unmount VMTools and continue. Anything else to skip VMTools unmounting";
IF ($unmountTools -eq "Y") {Write-Host "Unmounting VMTools on VMs..." -foreground "Yellow"; foreach ($vm in $vmtools) {$vm.UnmountToolsInstaller()}}ELSE{Write-Host "Skipping VMTools unmounting..." -foreground "Yellow"}}ELSE{Write-Host "No VMs found with VMTools Installer mounted. Continuing..." -foreground "Yellow"}


10. With all our VMware tools installations killed, we move on to ISOs. ISO’s that are stored in shared datastores won’t have an issue moving, but if ISOs have been mounted directly to a VM through a console window those can cause a hang up. Again, you know your environment better than me so use your best judgement when picking what to do.

$mountedCDdrives = Get-VMHost $hostName.Name | Get-VM | Where { $_ | Get-CdDrive | Where { $_.ConnectionState.Connected -eq "True" } }
IF ($mountedCDdrives.Count -gt "0"){Write-Host "The following VMs on $($hostName.Name) have mounted CD Drives:";
$mountedCDdrives.Name;
$unmountDrives = Read-Host "Press "Y" to unmount these ISOs and continue. Anything else to skip ISO unmounting";
IF ($unmountDrives -eq "Y") {Write-Host "Unmounting ISOs on VMs..." -foreground "Yellow"; foreach ($vm in $mountedCDdrives) {Get-VM $vm | Get-CDDrive | Set-CDDrive -NoMedia -Confirm:$False}}ELSE{Write-Host "Skipping ISO unmounting..." -foreground "Yellow"}}ELSE{Write-Host "No VMs found with ISOs mounted. Continuing..." -foreground "Yellow"}


11. Now we check if the host is in maintenance mode. This check isn’t required and we could just try to put a host in maintenance mode that’s already in maintenance mode without any errors, I just prefer to have this called out so people know that the host will be placed in maintenance mode. Also, if you don’t want to confirm and just want the host to automatically go into maintenance mode, you can remove the “Read-Host “Press Enter to place $($hostName.Name in Maintenance mode”;” section and it will automatically place the host in maintenance mode.

$hostState = Get-VMHost $hostname.Name
IF ($hostState.State -eq "Maintenance"){Write-Host "$($hostName.Name) is already in maintenance mode. Continuing to patch Staging/Remediation" -foreground "Yellow"}ELSE{Read-Host "Press Enter to place $($hostName.Name) in Maintenance mode"; Start-Sleep 7; Write-Host "Enabling Maintenance mode for $($hostName.Name). This may take a while..." -foreground "Yellow"; Set-VMHost $hostName.Name -State "Maintenance"}


12. This was an interesting issue I ran into. I had a customer running ESXi 6.0 with PernixData installed which wasn’t compatible with ESXi 6.5 which we were upgrading to. When we attempted to upgrade we’d fail because the PernixData VIB was present. I threw this check in to see if this VIB existed on their hosts and to remove it before proceeding. I also added a second placeholder VIB name in case you have multiple VIBs to remove you can just replace the name with the appropriate VIB name and even add additional VIBs with another -OR $_.ID -eq “vibname”

$esxcli = Get-esxcli -vmhost $hostName.Name
$vibCheck = $esxcli.software.vib.list() | Where {($_.ID -eq "PernixData_bootbank_pernixcore-vSphere6.0.0_3.5.0.2-39793" -OR $_.ID -eq "Other_vib_name_xxxxxx")}
IF ($vibCheck.Count -gt "0"){Write-Host "Incompatible VIB found. Removing from host..." -foreground "Yellow"; foreach ($a in $vibCheck){$esxcli.software.vib.remove($null, $true, $false, $true, $a.Name)}}ELSE{Write-Host "No known incompatible VIBs found. Continuing..." -foreground "Green"}


13. And, of course, if removing a VIB we need to reboot so now we throw this reboot check in there as well. If there were no VIBs found in Step 12, this will be ignored. Otherwise, we prompt for reboot, enter the reboot command, check for the host to enter the NotResponding state and report on the state until it responds in vCenter and returns to Maintenance state.

IF ($vibCheck.Count -gt "0" -AND $baseline.BaselineType -eq "Upgrade"){Read-Host "VIBs were removed from host. Press enter to reboot host before attempting upgrade";Restart-VMhost $hostName.Name -confirm:$false}ELSE{$skip = "1"; Write-Host ""}
IF ($skip -ne "1"){
Write-Host "$($hostName.Name) is going to reboot..." -foreground "Yellow"
do {
Start-Sleep 3
$hostState = (get-vmhost $hostName.Name).ConnectionState
}
while ($hostState -ne "NotResponding")
Write-Host "$($hostName.Name) is currently down..." -foreground "Yellow"

#Wait for server to reboot
do {
Start-Sleep 5
$hostState = (get-vmhost $hostName.Name).ConnectionState
Write-Host "Waiting for $($hostName.Name) to finish rebooting..." -foreground "Yellow"
}
while ($hostState -ne "Maintenance")
Write-Host "$($hostName.Name) is back up..." -foreground "Yellow"}ELSE{Write-Host ""}

14. Now that all that work is done, we can start staging patches. If this is a patch baseline we run stage command. If it’s an upgrade baseline, we’ll skip this step

IF ($baseline.BaselineType -eq "Upgrade"){Write-Host "$($baseline.Name) is an Upgrade Baseline. Skipping to remediation..." -foreground "Yellow"}ELSE{Write-Host "Staging patches to $($hostName.Name) in Cluster $($clusterName.Name)..." -foreground "Yellow"; Stage-Patch -entity $hostName.Name -baseline $baseline}


15. Once patches have been staged (or upgrades ready to push) it’s time for remediation. We prompt that the host will reboot on its own once the patch has completed and we set a few advanced options. These are the defaults, but can still be environment specific so check to make sure this is what you want to use.

Write-Host "Remediating patches on $($hostName.Name) in Cluster $($clusterName.Name). Host will reboot when complete" -foreground "Yellow"
Remediate-Inventory -Entity $hostName.Name -Baseline $baseline -HostFailureAction Retry -HostNumberofRetries 2 -HostRetryDelaySeconds 120 -HostDisableMediaDevices $true -ClusterDisableDistributedPowerManagement $true -ClusterDisableHighAvailability $true -confirm:$false -ErrorAction SilentlyContinue


At the top of our PowerShell window we get the percentage of completion for our task. It’s not very accurate as it stays at 30% then goes to 92% when it’s nearly complete.

16. Once the host has been rebooted and comes back online we want to see the current status of that host to ensure updates were successful. We are comparing the build number we grabbed before we started patching against the build number after the reboot. If they are the same, something didn’t work and we need to check into it. Otherwise, we do nothing.

Write-Host "Retrieving Host build status..." -foreground "Yellow"
$hostBuild = Get-VMHost $hostName.Name
IF ($hostBuild.Build -eq $hostState.Build){Write-Host "Patch/Upgrade was not applied. Check status in vCenter and re-run the script. Exiting..." -foreground "Red";$error;Start-Sleep 20;break}ELSE{}

17. Now that the host was patched, we show a list of all the hosts in that cluster along with their build, version, and state. This gives us a full view of the cluster so we can see if there are any hosts left to be patched and then we exit maintenance mode for this host.

Get-Cluster $clusterName.Name | Get-VMhost | Select Name,Build,Version,State | Sort Name | format-table -autosize
Write-Host "Exiting Maintenance mode for Host $($hostName.Name)..." -foreground "Yellow"
Get-VMHost $hostName.Name | Set-VMHost -State Connected


18. Based on that list will determine the answer to our next question. We are being prompted to re-enable the DRS rules we previously disabled (if any). If any rules were chosen to be disabled we captured that in a variable in step 3. We can choose to re-enable just those disabled rules by pressing ‘Y’ or if there are other hosts left to patch we just press any other key to continue.

IF ($disableRules -eq "Y") {$enableRules = Read-Host "If Cluster patching is complete press "Y" to re-enable DRS rules. Anything else to continue";
IF ($enableRules -eq "Y") {Write-Host "Re-enabling DRS Must Run rules" -foreground "Yellow"; 
foreach ($name in $drsRules){Set-DrsVMHostRule -rule $name -enabled:$true}} ELSE {
Write-Host "DRS Rules not being re-enabled. Continuing..." -foreground "Yellow"}} ELSE {}


19. In this last question we’re just displaying the output from our last host patched and prompting the user to quit patching or go back to step 6 and pick the next host in the cluster to patch.

$answer = Read-Host "$($hostname.Name) patched in Cluster $($clusterName.Name). Press "1" to re-run the script. Anything else to exit"


20. Finally, to close out the first loop, we have the following lines. In step 19 we have the variable $answer which asks the user to enter ‘1’ to re-run the script and pick another host. This line at the bottom is saying until the user enters something other than 1, keep performing that loop. If anything else is entered, the script exits. Answering “1” will start the script over from Step 6. We will perform another “Get-Cluster | Get-VMHost” on the chosen cluster and retrieve the current build and state information for each of the hosts and display the updated results. As you can see from the screenshot below, vmm-04 is no in a Connected state with a Build number of 9298722,

}
UNTIL ($answer -ne "1")


Below is the script all put together to copy and test. Like all scripts pulled from the internet, make sure you test them in a lab/isolated environment until you can ensure proper functionality.

Write-Host "Enter the FQDN of the " -NoNewline
Write-Host "[vCenter Server]" -ForegroundColor Red -NoNewline
Write-Host " to connect to: " -NoNewline
$vCenterFqdn = Read-Host
Connect-viserver $vCenterFqdn

$global:i=0
Get-Cluster | Sort Name | Select @{Name="Number";Expression={$global:i++;$global:i}},Name -OutVariable menu | format-table -AutoSize
$clusterNum = Read-Host "Select the number of the Cluster to be patched"
$clusterName = $menu | where {$_.Number -eq $clusterNum}

$drsRules = Get-Cluster $($clusterName.Name) | Get-DrsVMHostRule | Where {$_.Type -eq "MustRunOn"} | Where {$_.Enabled -eq $True}
IF ($drsRules.Count -gt "0"){Write-Host "The following rules may prevent a host from entering Maintenance mode:" -foreground "Yellow"; $drsRules.Name; $disableRules = Read-Host "Press Y to disable these rules. Anything else to continue";
IF ($disableRules -eq "Y"){Write-Host "Disabling DRS Rules..." -foreground "Yellow";
foreach ($name in $drsRules){Set-DrsVMHostRule -rule $name -enabled:$false}} ELSE {Write-Host "Skipping disabling of DRS Rules. Continuing..." -foreground "Yellow"}} ELSE {Write-Host "No "Must Run" Rules in $($clusterName.Name). Continuing..." -foreground "Yellow"}

$global:i=0
Get-Baseline | Select Name | Sort Name | Select @{Name="Number";Expression={$global:i++;$global:i}},Name -OutVariable menu | format-table -AutoSize
$baselineNum = Read-Host "Select the number of the Baseline to be attached"
$baselineName = $menu | where {$_.Number -eq $baselineNum}
Write-Host "Attaching $($baselineName.Name) Baseline to $($clusterName.Name)..." -Foreground "Yellow"
$baseline = Get-Baseline $baselineName.Name
Attach-Baseline -Baseline $baseline -Entity $clusterName.Name

DO
{
DO
{
$global:i=0
Get-Cluster $clusterName.Name | Get-VMhost | Sort Name | Select @{Name="Number";Expression={$global:i++;$global:i}},Name,Build,Version,State -OutVariable menu | format-table -AutoSize
$hostNum = Read-Host "Select the number of the Host to be patched"
$hostName = $menu | where {$_.Number -eq $hostNum}

Write-Host "Scanning $($hostName.Name) patch inventory..." -foreground "Yellow"
Scan-Inventory -Entity $hostName.Name

Write-Host "Scanning $($hostName.Name) for patch compliance..." -foreground "Yellow"
$compliance = Get-Compliance $hostName.Name 
IF ($compliance.Status -eq "Compliant"){Write-Host "No available patches for $($hostName.Name). Choose a different host" -foreground "Red"}ELSE{Write-Host "Host is out of date" -foreground "Yellow"}}
UNTIL ($compliance.Status -ne "Compliant")

$vmtools = Get-VMHost $hostName.Name | Get-VM | Where {$_.ExtensionData.RunTime.ToolsInstallerMounted -eq "True"} | Get-View
IF ($vmtools.Count -gt "0"){Write-Host "The following VMs on $($hostName.Name) have VMTools Installer Mounted:";
$vmtools.Name;
$unmountTools = Read-Host "Press "Y" to unmount VMTools and continue. Anything else to skip VMTools unmounting";
IF ($unmountTools -eq "Y") {Write-Host "Unmounting VMTools on VMs..." -foreground "Yellow"; foreach ($vm in $vmtools) {$vm.UnmountToolsInstaller()}}ELSE{Write-Host "Skipping VMTools unmounting..." -foreground "Yellow"}}ELSE{Write-Host "No VMs found with VMTools Installer mounted. Continuing..." -foreground "Yellow"}

$mountedCDdrives = Get-VMHost $hostName.Name | Get-VM | Where { $_ | Get-CdDrive | Where { $_.ConnectionState.Connected -eq "True" } }
IF ($mountedCDdrives.Count -gt "0"){Write-Host "The following VMs on $($hostName.Name) have mounted CD Drives:";
$mountedCDdrives.Name;
$unmountDrives = Read-Host "Press "Y" to unmount these ISOs and continue. Anything else to skip ISO unmounting";
IF ($unmountDrives -eq "Y") {Write-Host "Unmounting ISOs on VMs..." -foreground "Yellow"; foreach ($vm in $mountedCDdrives) {Get-VM $vm | Get-CDDrive | Set-CDDrive -NoMedia -Confirm:$False}}ELSE{Write-Host "Skipping ISO unmounting..." -foreground "Yellow"}}ELSE{Write-Host "No VMs found with ISOs mounted. Continuing..." -foreground "Yellow"}

$hostState = Get-VMHost $hostname.Name
IF ($hostState.State -eq "Maintenance"){Write-Host "$($hostName.Name) is already in maintenance mode. Continuing to patch Staging/Remediation" -foreground "Yellow"}ELSE{
#Read-Host "Press Enter to place $($hostName.Name) in Maintenance mode"; Start-Sleep 7; Write-Host "Enabling Maintenance mode for $($hostName.Name). This may take a while..." -foreground "Yellow"; Set-VMHost $hostName.Name -State "Maintenance"}
Write-Host "Enabling Maintenance mode for $($hostName.Name). This may take a while..." -foreground "Yellow"; ; Start-Sleep 7; Set-VMHost $hostName.Name -State "Maintenance"}

$esxcli = Get-esxcli -vmhost $hostName.Name
$vibCheck = $esxcli.software.vib.list() | Where {($_.ID -eq "PernixData_bootbank_pernixcore-vSphere6.0.0_3.5.0.2-39793" -OR $_.ID -eq "Other_vib_name_xxxxxx")}
IF ($vibCheck.Count -gt "0"){Write-Host "Incompatible VIB found. Removing from host..." -foreground "Yellow"; foreach ($a in $vibCheck){$esxcli.software.vib.remove($null, $true, $false, $true, $a.Name)}}ELSE{Write-Host "No known incompatible VIBs found. Continuing..." -foreground "Green"}

IF ($vibCheck.Count -gt "0" -AND $baseline.BaselineType -eq "Upgrade"){Read-Host "VIBs were removed from host. Press enter to reboot host before attempting upgrade";Restart-VMhost $hostName.Name -confirm:$false}ELSE{$skip = "1"; Write-Host ""}
IF ($skip -ne "1"){
Write-Host "$($hostName.Name) is going to reboot..." -foreground "Yellow"
do {
Start-Sleep 3
$hostState = (get-vmhost $hostName.Name).ConnectionState
}
while ($hostState -ne "NotResponding")
Write-Host "$($hostName.Name) is currently down..." -foreground "Yellow"

#Wait for server to reboot
do {
Start-Sleep 5
$hostState = (get-vmhost $hostName.Name).ConnectionState
Write-Host "Waiting for $($hostName.Name) to finish rebooting..." -foreground "Yellow"
}
while ($hostState -ne "Maintenance")
Write-Host "$($hostName.Name) is back up..." -foreground "Yellow"}ELSE{Write-Host ""}

IF ($baseline.BaselineType -eq "Upgrade"){Write-Host "$($baseline.Name) is an Upgrade Baseline. Skipping to remediation..." -foreground "Yellow"}ELSE{Write-Host "Staging patches to $($hostName.Name) in Cluster $($clusterName.Name)..." -foreground "Yellow"; Stage-Patch -entity $hostName.Name -baseline $baseline}

Write-Host "Remediating patches on $($hostName.Name) in Cluster $($clusterName.Name). Host will reboot when complete" -foreground "Yellow"
Remediate-Inventory -Entity $hostName.Name -Baseline $baseline -HostFailureAction Retry -HostNumberofRetries 2 -HostRetryDelaySeconds 120 -HostDisableMediaDevices $true -ClusterDisableDistributedPowerManagement $true -ClusterDisableHighAvailability $true -confirm:$false -ErrorAction SilentlyContinue

Write-Host "Retrieving Host build status..." -foreground "Yellow"
$hostBuild = Get-VMHost $hostName.Name
IF ($hostBuild.Build -eq $hostState.Build){Write-Host "Patch/Upgrade was not applied. Check status in vCenter and re-run the script. Exiting..." -foreground "Red";$error;Start-Sleep 20;break}ELSE{}

Get-Cluster $clusterName.Name | Get-VMhost | Select Name,Build,Version,State | Sort Name | format-table -autosize
Write-Host "Exiting Maintenance mode for Host $($hostName.Name)..." -foreground "Yellow"
Get-VMHost $hostName.Name | Set-VMHost -State Connected

IF ($disableRules -eq "Y") {$enableRules = Read-Host "If Cluster patching is complete press "Y" to re-enable DRS rules. Anything else to continue";
IF ($enableRules -eq "Y") {Write-Host "Re-enabling DRS Must Run rules" -foreground "Yellow"; 
foreach ($name in $drsRules){Set-DrsVMHostRule -rule $name -enabled:$true}} ELSE {
Write-Host "DRS Rules not being re-enabled. Continuing..." -foreground "Yellow"}} ELSE {}

$answer = Read-Host "$($hostname.Name) patched in Cluster $($clusterName.Name). Press "1" to re-run the script. Anything else to exit"

}
UNTIL ($answer -ne "1")

Backing Up Portgroup Data with PowerShell and XML

Managing virtual standard switches (vSwitches) in a VMware environment, large or small, can cause a lot of headaches. Ensuring each portgroup is named the same, has the same VLAN and is present on each host can be a challenge. Virtual distributed switches (dvSwitches) exist to make our lives easier, but aren’t always available due to licensing or other restrictions. They have their own drawbacks, but the good generally outweighs the bad.

As I have been spending a significant amount of time in PowerShell and XML recently I wrote a script designed to backup the Portgroup configuration of all your clusters to ensure that all hosts will have the proper networking configuration during host rebuilds or additions. Let’s walk through this script and show you all the different elements.

1. Here we define and connect to the vCenter server

$vCenterFqdn = "vcenter01.domain.local"
Connect-viserver $vCenterFqdn

2. Here we are defining the path for our Portgroups.xml file. We’ll reference this variable later on.

$xmlNetworkPath="C:\Scripts\XML\Portgroups.xml"

3. Here we are checking to see if this file already exists. If it does we’ll skip ahead to the population side, but if not we’ll have to create it. $networkCheck tests the path defined in the variable $xmlNetworkPath. If that file is there, we’re writing into the console that it exists and we’re moving on. If it doesn’t exist we move on to step 4.

$networkCheck = Test-Path $xmlNetworkPath
IF ($networkCheck -eq $True){Write-Host "Portgroup file already exists. Continuing..." -foreground "Green"}
ELSE
{Write-Host "Portgroup file not created. Creating..." -foreground "Yellow"}

4. Since this is our first run, this file doesn’t exist yet so we need to create it. Populating an empty XML file is not something I ever figured out how to do. Someone much smarter than me in XML and PowerShell can figure it out, but I found a nice work around by creating an XML template. This is an outline of what our XML file will be and we’re just filling in the data as we go.

$xmlNetwork=[xml]@’ is saying everything between @’ and ‘@ will be part of this file. At the end, we’re saving this output ($xmlNetwork.Save) to the file path we defined earlier ($xmlNetworkPath).


$xmlNetwork=[xml]@'

<vSwitchConfig>
  <templates>
    <Portgroup>
      <vlanId></vlanId>
      <virtualSwitch></virtualSwitch>
      <cluster></cluster>
    </Portgroup>
  </templates>
  <Portgroups>
    <test />
  </Portgroups>
</vSwitchConfig>
‘@
$xmlNetwork.Save($xmlNetworkPath)

5. In case we run into permissions issues we want to perform an additional check to ensure that file was actually written. For the most part we’re just repeating Step 3. The difference here is we’re changing the output in “Write-Host” but also if the file isn’t created, we need to kill the script. The message is written that the file was not created, then we wait 20 seconds, then exit the script. The reason we add the “Start-Sleep” is in case this script is run by just double-clicking and not triggered from within a powershell window, the script will exit and you’ll never know why.

$networkCheck = Test-Path $xmlNetworkPath
IF ($networkCheck -eq $True){Write-Host "Portgroup file created successfully. Continuing..." -foreground "Green"}
ELSE
{Write-Host "Portgroup file not created. Exiting..." -foreground "Red"; Start-Sleep 20; Break}}

6. Now we need to read the contents of that XML file we created in XML format. The [XML] denotes that this is an XML file. $xmlNetwork is just the variable name I chose (this can be anything you want) and $xmlNetworkPath is the path to the file we defined in step 2

[XML]$xmlNetwork = Get-Content $xmlNetworkPath

7. Now comes the fun part where we actually start pulling data from vCenter. We’re going to get all the clusters in the vCenter we’re connected to. The ForEach command says that for every cluster we need to run this same command. So in each cluster we’re looking for an ESXi host that is currently connected, then we choose a random host from the cluster (get-random) and on that random host we get all the virtual portgroups that aren’t on dvSwitches.

$getClusters = Get-Cluster
ForEach ($cluster in $getClusters) {
$getPortgroups = $cluster | Get-VMHost | Where {$_.ConnectionState -ne "NotResponding"} | Get-Random | Get-VirtualPortGroup | Where {$_.key -notlike "*dvportgroup*"}

8. Now that we have all these portgroups let’s get all the data we need in them. We perform another ForEach on every Portgroup that was on that host. The $testPg variable is used to see if the current Portgroup name ($pgName.Name) is present in the XML file. What we don’t want is to re-add the same Portgroup name over and over again, we just want a single reference to the portgroup name and we’ll add another entry for the clusters that contain it. $testPg -eq $null means if the portgroup name isn’t there, we’ll create a new entry.

foreach ($pgName in $getPortGroups) {
$testPg = $xmlNetwork.vSwitchConfig.Portgroups.Portgroup|Where {$_.name -eq $pgName.Name}
IF ($testPg -eq $null)

9. If the Portgroup isn’t there we need to create a new entry. Looking at the XML file outline we created in Step 4 you see the tag. This is used to work around my inability to create new entries in XML. We copy the section below and then fill out the data as we go


<Portgroup>
  <vlanId></vlanId>
  <virtualSwitch></virtualSwitch>
  <cluster></cluster>
</Portgroup>

The variable “$xmlNetwork” was defined in Step 6 and is the name of our XML file. The addition of “.vSwitchConfig” is the top level tag in our XML file from Step 4.
The variable “$parentNode” defines the top-level of the XML file we’ll be using from our template.
The variable “$destinationNode” defines where this copied data is going to be placed.
The variable “$cloneNode” defines what tag we’re copying from $parentNode.
The variable “$addNameAttribute” is creating an attribute called “name” and the name we’re giving it is the name of the portgroup we’re pulling from vCenter. Adding “.Value = ” to the end of this variable gives us the ability to define the name when we’re creating the new portgroup tag.
The variable $newNode is used to create the new Portgroup tag and then “.InnerXML =” allows us to chose the template tag from the XML file.
(The use of [void] is something I don’t fully grasp. This is the only way it works, but I can’t tell you why.)
Using $destinationNode.AppendChild is saying place this new tag into the $destinationNode location. $newNode is the data that’s being added. “.Attributes” allows us to assign the new name Attribute we defined and “.Append($addNameAttribute)” places that attribute tag.
The variable $updateNetwork is a lookup in the XML file to find a portgroup with the tag we just created.
Once we find that portgroup, $updateVLAN, $updateVirtualSwitch, and $updateCluster are used to assign the values to these empty tags that were created.


{
$parentNode = $xmlNetwork.vSwitchConfig.templates
$destinationNode = $xmlNetwork.vSwitchConfig.Portgroups
$cloneNode = $parentNode.SelectSingleNode("Portgroup")
$addNameAttribute = $xmlNetwork.CreateAttribute("name")
$addNameAttribute.Value = $pgName.Name
$newNode = $xmlNetwork.CreateElement("Portgroup")
$newNode.InnerXML = $cloneNode.InnerXML
[void]$destinationNode.AppendChild($newNode).Attributes.Append($addNameAttribute)
$updateNetwork = ($xmlNetwork.vSwitchConfig.Portgroups.Portgroup|Where {$_.name -eq $pgName.Name})
$updateVLAN = $updateNetwork.vlanId = $pgName.VLanId.ToString()
$updateVirtualSwitch = $updateNetwork.virtualSwitch = $pgName.VirtualSwitchName
$updateCluster = $updateNetwork.cluster = $cluster.Name
}

10. In the event this portgroup already exists, we have the ELSE portion of our IF statement we started in Step 8. Here we’re referencing the portgroup in question and checking to see if the cluster tag has already been added. We look up the portgroup name “$testPg” and search for a cluster that matches the cluster we’re currently working with. If that cluster doesn’t exist, we add a new element ($xmlNetwork.CreateElement(“cluster”)) and populate its value ($addCluster.InnerText = $cluster.Name)

If that cluster tag already exists for that Portgroup, we don’t do anything “{}”


ELSE {($testCluster = $testPg|Where {$_.cluster -eq $cluster.Name});
IF ($testCluster -eq $null) {
$addCluster = $xmlNetwork.CreateElement("cluster");
$addCluster.InnerText = $cluster.Name
$testPg.AppendChild($addCluster) | Out-Null }ELSE{}}
}}

11. Once all of our clusters and Portgroups have been looped through. We need to save that data to the XML file we created.

$xmlNetwork.Save($xmlNetworkPath)

12. Now that the file has been saved, we want to see what data has been written. This isn’t required, but just adds a nice little output to the screen so you can see that the data has been populated as expected.

We perform the same Get-Content command with our [XML] tag. Then we write the output of every portgroup, sorted by name, into a table so we can see all the data.


[XML]$xmlNetwork = Get-Content $xmlNetworkPath
Write-Host "The following Portgroup configuration exists in $($xmlNetworkPath)" -foreground "Yellow"
$xmlNetwork.vSwitchConfig.Portgroups.Portgroup | Sort name | Format-Table

I have obscured some of the networks and VLANs in use in my environment, but this is the output you can expect.

All together this is what the script looks like:


#Connect to the defined vCenter Server
$vCenterFqdn = "vcenter01.domain.local"
Connect-viserver $vCenterFqdn

#Define XML File Path
$xmlNetworkPath="C:\Scripts\XML\Portgroups.xml"

#Check for Portgroup XML File
$networkCheck = Test-Path $xmlNetworkPath
IF ($networkCheck -eq $True){Write-Host "Portgroup file already exists. Continuing..." -foreground "Green"}
ELSE
{Write-Host "Portgroup file not created. Creating..." -foreground "Yellow"

#Create XML Portgroup File

$xmlNetwork=[xml]@'

<vSwitchConfig>
  <templates>
    <Portgroup>
      <vlanId></vlanId>
      <virtualSwitch></virtualSwitch>
      <cluster></cluster>
    </Portgroup>
  </templates>
  <Portgroups>
    <test />
  </Portgroups>
</vSwitchConfig>
'@
$xmlNetwork.Save($xmlNetworkPath)

$networkCheck = Test-Path $xmlNetworkPath
IF ($networkCheck -eq $True){Write-Host “Portgroup file created successfully. Continuing…” -foreground “Green”}
ELSE
{Write-Host “Portgroup file not created. Exiting…” -foreground “Red”;Start-Sleep 20; BREAK}}

#Get contents of XML file
[XML]$xmlNetwork = Get-Content $xmlNetworkPath

#Gather all hosts in each cluster
$getClusters = Get-Cluster
foreach ($cluster in $getClusters) {
$getPortgroups = $cluster | Get-VMHost | Where {$_.ConnectionState -ne “NotResponding”} | Get-Random | Get-VirtualPortGroup | Where {$_.key -notlike “*dvportgroup*”}
foreach ($pgName in $getPortGroups) {
$testPg = $xmlNetwork.vSwitchConfig.Portgroups.Portgroup|Where {$_.name -eq $pgName.Name}
IF ($testPg -eq $null) {
$parentNode = $xmlNetwork.vSwitchConfig.templates
$destinationNode = $xmlNetwork.vSwitchConfig.Portgroups
$cloneNode = $parentNode.SelectSingleNode(“Portgroup”)
$addNameAttribute = $xmlNetwork.CreateAttribute(“name”)
$addNameAttribute.Value = $pgName.Name
$newNode = $xmlNetwork.CreateElement(“Portgroup”)
$newNode.InnerXML = $cloneNode.InnerXML
[void]$destinationNode.AppendChild($newNode).Attributes.Append($addNameAttribute)
$updateNetwork = ($xmlNetwork.vSwitchConfig.Portgroups.Portgroup|Where {$_.name -eq $pgName.Name})
$updateVLAN = $updateNetwork.vlanId = $pgName.VLanId.ToString()
$updateVirtualSwitch = $updateNetwork.virtualSwitch = $pgName.VirtualSwitchName
$updateCluster = $updateNetwork.cluster = $cluster.Name
} ELSE {($testCluster = $testPg|Where {$_.cluster -eq $cluster.Name});
IF ($testCluster -eq $null) {
$addCluster = $xmlNetwork.CreateElement(“cluster”);
$addCluster.InnerText = $cluster
$testPg.AppendChild($addCluster) | Out-Null }ELSE{}}
}}
$xmlNetwork.Save($xmlNetworkPath)

#Display Results
[XML]$xmlNetwork = Get-Content $xmlNetworkPath
Write-Host “The following Portgroup configuration exists in $($xmlNetworkPath)” -foreground “Yellow”
$xmlNetwork.vSwitchConfig.Portgroups.Portgroup | Sort name | Format-Table

Removing Values in XML Using PowerShell

We’ve gathered values from our XML file and we’ve managed to add values to an XML file using PowerShell, but what about deleting? What I’m talking about is deleting XML tags and attributes, not deleting the value in an existing tag. Let’s look at our sample file and I’ll explain.

Above is our sample.xml file that was updated in the last post. Inside we have individual portgroups with a name and elements that we were added. In the context of the project I was working on, I needed the ability to delete a portgroup and all of its elements (VLAN, virtual switch, and cluster), but in some situations I needed to just delete the cluster. If a portgroup needed to be removed from one specific cluster, but remaining available to another cluster I couldn’t have the entire portgroup removed as it could affect virtual machines that are still running.

How do we do it? Let’s find out.

Just like before, let’s get the contents of our XML file so we can work with the data.

$xmlFile = "C:\Scripts\sample.xml"
$xmlConfig = [System.Xml.XmlDocument](Get-Content $xmlFile)

The two things we need in order to remove a portgroup are the name of the portgroup and the name of the cluster. Based on the sample.xml file, we see that “VM Network” is on two clusters (cluster1 and NestedCluster) so let’s start there. Let’s get some simple variables setup just like we did last time.

$name = Read-Host -Prompt "Enter the name of the Portgroup (ex: Management Network)"
$cluster = Read-Host -Prompt "Enter the name of the cluster for this Portgroup (ex: Cluster1)"

I’ve created a new powershell file called “removePortgroup.ps1” and it has the above variables in there prompting me to enter the name and the cluster of the portgroup to be removed.

Now that we have the XML contents and our variables defined it’s time to see what data is there. If you recall from my earlier post about getting data from XML, we can see the contents of our file by following the hierarchy. With the following command we can see all portgroups that have been defined.

$xmlConfig.config.portgroups.ChildNodes

Now that we know what’s there, we need to filter the results and only display the Web Network. We’ll pass this as a variable so we can use that info later.

$clusters = $xmlConfig.config.portgroups.ChildNodes | ? {$name -eq $_.name}

Using the same command as before but passing it as a variable, we added a parameter to search for the name of our Portgroup (VM Network) using the $name variable we created earlier and we are trying to match it with another portgroup of the same name. When we run the command “$clusters” we see what portgroup came back as a match. We can see the VM Network portgroup that has 2 clusters.

With this portgroup living on two clusters we are only interested in removing it from one which is NestedCluster. We’ll work on the logic for this later one, for now let’s focus on how you remove a single from this portgroup.

The following commands will remove the element that matches our $name and $cluster variables we created earlier.

$text = "#text"
$clusterTag = $xmlConfig.config.portgroups.portgroup | Where {$_.name -eq $name}
$updateTag = $clusterTag.SelectNodes("cluster") | Where {$_.$text -eq $cluster}
$updateTag.ParentNode.RemoveChild($updateTag)

We are looking for the Text inside of the tag. As you can see from the output at the bottom, it has a label (or whatever it is called) of #text. The problem is, when working within PowerShell, if I add a “#” to a line, it will comment out everything after it. We get around that limitation but creating a variable called $text and given it a value of #text. Maybe there is a better way of doing it, but this worked so I went with it.

The next line is looking for a portgroup matching $name. The next line is selecting the tag that matches $cluster. The final line is performing the removal of that tag. This still needs to be saved, so we’re not done yet, but at this point we have removed from . If you run the command below you can see that “NestedCluster” has been removed as a cluster in the “VM Network” portgroup.

$xmlConfig.config.portgroups.ChildNodes

Now what if this portgroup is only on one cluster and we want to delete it and all of its attributes completely? We can do so with the following command.

$xmlConfig.config.portgroups.ChildNodes | ? {$name -eq $_.name} | % {$xmlConfig.config.portgroups.RemoveChild($_)} | Out-Null

In this command, we are getting all the portgroups in the first part, then filtering the results where $name matches the name of an existing portgroup. Then we are removing that entire and all its contents from the XML file. The Out-Null is required to make it work, but I don’t really know why. Once again, this isn’t saved yet, we can do that in the next step.

In order to save we need to run the following command.

$xmlConfig.Save($xmlFile)

Now that this file is saved, we can re-check the XML file using the commands below

$xmlFile = "C:\Scripts\sample.xml"
$xmlConfig = [System.Xml.XmlDocument](Get-Content $xmlFile)

If we run the command $xmlConfig.config.portgroups.childnodes we will see that “VM Network” is now gone

We can remove just an attribute or we can remove an entire element. How can we create a script to determine which one to use? If we wrap this all in an IF/ELSE statement we can do it pretty easily.

Using the $clusters variable we created earlier, we can perform a count of how many tags there are. Running the following command will show many clusters a portgroup is provisioned to.

$clusters.cluster.count

In our IF/ELSE statement, we are checking to see if the count is greater than 1 (-gt 1) and then performing the command to remove just the cluster tag. If it’s not greater than 1, we are running the command to remove the entire portgroup and all of its attributes.

$clusters = $xmlConfig.config.portgroups.ChildNodes | ? {$name -eq $_.name}
IF ($clusters.cluster.count -gt 1){
$text = "#text"
$clusterTag = $xmlConfig.config.portgroups.portgroup | Where {$_.name -eq $name}
$updateTag = $clusterTag.SelectNodes("cluster") | Where {$_.$text -eq $cluster}
$updateTag.ParentNode.RemoveChild($updateTag)
$xmlConfig.Save($xmlFile) } ELSE {
$xmlConfig.config.portgroups.ChildNodes | ? {$name -eq $_.name} | % {$xmlConfig.config.portgroups.RemoveChild($_)} | Out-Null
$xmlConfig.Save($xmlFile) }

Notice how we see the same #text and “NestedCluster” output after the commands ran when we were just trying to remove a cluster? That’s because powershell sees there are more than 1 cluster assigned to the VM Network portgroup and it just removed it.

Here is the full script to check if there are multiple XML attributes named “cluster” and remove it or remove the entire portgroup element.

#Define XML file and Get its contents
$xmlFile = "C:\Scripts\sample.xml"
$xmlConfig = [System.Xml.XmlDocument](Get-Content $xmlFile)

#Define variables and prompt for input
$name = Read-Host -Prompt "Enter the name of the Portgroup (ex: Management Network)"
$cluster = Read-Host -Prompt "Enter the name of the cluster for this Portgroup (ex: Cluster1)"

#IF/ELSE Statement to remove a single attribute or an entire element
$clusters = $xmlConfig.config.portgroups.ChildNodes | ? {$name -eq $_.name}
IF ($clusters.cluster.count -gt 1){
$text = "#text"
$clusterTag = $xmlConfig.config.portgroups.portgroup | Where {$_.name -eq $name}
$updateTag = $clusterTag.SelectNodes("cluster") | Where {$_.$text -eq $cluster}
$updateTag.ParentNode.RemoveChild($updateTag)
$xmlConfig.Save($xmlFile) } ELSE {
$xmlConfig.config.portgroups.ChildNodes | ? {$name -eq $_.name} | % {$xmlConfig.config.portgroups.RemoveChild($_)} | Out-Null
$xmlConfig.Save($xmlFile) }

Adding Values to XML Using PowerShell

Continuing on from the previous post on getting values from an XML file using PowerShell, I’m going to talk about how to add values to an XML files. With the recent project I was working on I needed a way to easily add new portgroups into the XML configuration file. Every environment changes and while having something manually update an XML file would work there is always the possibility of messing up the syntax or something much worse.

Let’s have a look at the sample XML file again.

What we have here is 3 port groups (Management Network, VM Network, Web Network) with a VLAN, VirtualSwitch, and Cluster defined. The entire file sits inside the “config” tags. Each portgroup lives inside the “portgroups” tag, and the values for each portgroup live inside the individual “portgroup” tags.

The goal here is to add a brand new portgroup to this XML file so when we can then add that portgroup with the correct values to our ESXi hosts. Given our existing structure we’ll need a name, VLAN, Virtual Switch, and Cluster for this new portgroup.

In order to update our XML file we need to get the XML file contents and we do that by using the following command (again, using the file path as a variable so I can easily change it later).

$xmlFile = "C:\Scripts\sample.xml"
$xmlConfig = [System.Xml.XmlDocument](Get-Content $xmlFile)

Now that we have the contents of the XML file, we need to gather the name, VLAN, Virtual Switch, and Cluster for this new portgroup. The question you’ll have to ask yourself is do you want to update the script file with each new portgroup you need to add or do you just want the script to prompt for those values each time it’s run?

If you want to manually update the file and let it run by itself just define the following variables in your script. For this example we’re creating a new portgroup called “App Network” with VLAN 31, on vSwitch1 and on cluster2.

$name = "App Network"
$vlanID = "31"
$virtualSwitch = "vSwitch1"
$cluster = "cluster2"

When we need to add these values into our XML file, we will refer to them using $name, $vlanID, $virtualSwitch, and $cluster.

Let’s do the same thing, but where we’re prompted for the values each time.

$name = Read-Host -Prompt "Enter the name of the new Portgroup (ex: Management Network)"
$vlanId = Read-Host -Prompt "Enter the VLAN Id number for the Portgroup (ex: 13)"
$virtualSwitch = Read-Host -Prompt "Enter the name of the Virtual Switch for Portgroup (ex: vSwitch1)"
$cluster = Read-Host -Prompt "Enter the name of the cluster for this Portgroup (ex: Cluster1)"

I saved the variables above as a file named “addPortgroup.ps1” in the C:\Scripts directory. Running that script in PowerShell prompted for my input for each of those lines. Instead of updating a script file each time I need to add a new Portgroup, I can run the script and enter the values myself.

Now that we’ve recorded these values, let’s start using them.

First off we need to create a new portgroup tag since we’re adding a new portgroup. We use a variable name (in this case $newPortgroup) to define where this tag will live. $xmlConfig.config.Portgroups is the parent of each of the Portgroups we currently have. What we’re doing here is saying make a change to by adding a new element called .

$newPortgroup = $xmlConfig.config.Portgroups.AppendChild($xmlConfig.CreateElement("Portgroup"))

Once we do that, now we’re going to start adding those attributes we already defined. We’ll start with name.

$newPortgroup.SetAttribute("Name",$name)

In the new Portgroup element we created, we set an attribute for and set it to the value of $name (“App Network” for our example).

Now, we can repeat that process for each attribute just like above and save our work (I’ll show how to save in a later step), but if we do that our XML formatting may not be what we’re looking for.

On line 20 of our sample.xml file we see how this data gets saved.

As you can see each attribute is added on the same line. Maybe this works for you, and if so, great. For me, I wanted to maintain the existing formatting, so there is a little more work to do.

Now we need to add each remaining attribute as a child to the already created “App Network” portgroup. We do that with the following command.

$newvlanIdAttribute = $newPortgroup.AppendChild($xmlConfig.CreateElement("vlanId"))
$newvlanIdValue = $newvlanIdAttribute.AppendChild($xmlConfig.CreateTextNode($vlanId))

The first line creates a new attribute called “vlanId” that’s been added to our new then the second line set the value of that attribute to $vlanId (VLAN 31 for this example).

We can now add Virtual Switch and Cluster the exact same way.

$newvirtualSwitchAttribute = $newPortgroup.AppendChild($xmlConfig.CreateElement("virtualSwitch"))
$newvirtualSwitchValue = $newvirtualSwitchAttribute.AppendChild($xmlConfig.CreateTextNode($virtualSwitch))
$newclusterAttribute = $newPortgroup.AppendChild($xmlConfig.CreateElement("cluster"))
$newclusterValue = $newclusterAttribute.AppendChild($xmlConfig.CreateTextNode($cluster))

With all the new elements created and the values set, we need to save this configuration. If we close down PowerShell right now none of our updates are saved. Save using the following command

$xmlConfig.Save($xmlFile)

Now that we’ve updated the xml file, let’s see what it looks like In PowerShell. We’ll need to re-check the contents of the XML file as it changed since we lasted checked.

$xmlFile = "C:\Scripts\sample.xml"
$xmlConfig = [System.Xml.XmlDocument](Get-Content $xmlFile)

And now let’s look at $xmlConfig.config.portgroup.portgroups

And if we look at the contents of the XML file we can see that “App Network” matches the layout of our other portgroups

And here is what the script looks like all together:

#Define XML file and Get its contents
$xmlFile = "C:\Scripts\sample.xml"
$xmlConfig = [System.Xml.XmlDocument](Get-Content $xmlFile)

#Define variables and prompt for input
$name = Read-Host -Prompt "Enter the name of the new Portgroup (ex: Management Network)"
$vlanId = Read-Host -Prompt "Enter the VLAN Id number for the Portgroup (ex: 13)"
$virtualSwitch = Read-Host -Prompt "Enter the name of the Virtual Switch for Portgroup (ex: vSwitch1)"
$cluster = Read-Host -Prompt "Enter the name of the cluster for this Portgroup (ex: Cluster1)"

#Create a new XML element with the input entered above and save
$newPortgroup = $xmlConfig.config.Portgroups.AppendChild($xmlConfig.CreateElement("Portgroup"));
$newPortgroup.SetAttribute("Name",$name);
$newvlanIdAttribute = $newPortgroup.AppendChild($xmlConfig.CreateElement("vlanId"));
$newvlanIdValue = $newvlanIdAttribute.AppendChild($xmlConfig.CreateTextNode($vlanId));
$newvirtualSwitchAttribute = $newPortgroup.AppendChild($xmlConfig.CreateElement("virtualSwitch"));
$newvirtualSwitchValue = $newvirtualSwitchAttribute.AppendChild($xmlConfig.CreateTextNode($virtualSwitch));
$newclusterAttribute = $newPortgroup.AppendChild($xmlConfig.CreateElement("cluster"));
$newclusterValue = $newclusterAttribute.AppendChild($xmlConfig.CreateTextNode($cluster));
$xmlConfig.Save($xmlFile)

Getting Values from XML Using PowerShell

Working on a recent project I was tasked with maintaining VMware host configuration via XML files. The goal was to have a central location to modify the settings of a host (DNS, vmkernel interfaces, vswitch config, port groups) without having to update each host by hand. This lead me down the path of using PowerShell with XML files as the configuration source.

Below is the sample of my XML file.

What we have here is 3 port groups (Management Network, VM Network, Web Network) with a VLAN, Virtual Switch, and Cluster defined. The entire file sits inside the “config” tags. Each portgroup lives inside the “portgroups” tag, and the values for each portgroup live inside the individual “portgroup” tags.

Below is a sample command to get values from an XML file. We pass the file location as a variable and then we store all those values inside another variable to make working with the values easier.

$xmlFile = "C:\Scripts\sample.xml"
$xmlConfig = [System.Xml.XmlDocument](Get-Content $xmlFile)

And once we type in $xmlConfig into Powershell we can see that it is pulling in the data from our XML file.

We can see the “config” section, but how do we dig deeper? Type in $xmlconfig.config and let’s look at the output

Now we can see the “portgroups” section. Let’s go deeper and see what’s inside of that by typing the following:

$xmlconfig.config.portgroups

Nice! Now we can see all the portgroups we have defined. In order to get detail on each of these, we can type $xmlconfig.config.portgroups.portgroup and we’ll get a list of all the portgroups and all their values.

Now let’s say we are only concerned about retrieving values from one of those portgroups. We can add a Where clause to find only the Portgroups that exist on vSwitch1 as an example with the following command:

$xmlconfig.config.portgroups.portgroup | Where {$_.virtualswitch -eq "vSwitch1"}

Grabbing values out of an XML file is pretty straightforward. In the next post we’ll talk about how to add information to an XML file.

Track Datastore Add & Removes With PowerCLI

While working with the data protection team at my job I was asked if there was any way to track new datastores being added to a vSphere cluster. When new LUNs are allocated to our vSphere clusters, the data protection team isn’t always made aware ahead of time. Normally this isn’t a big deal, but in our case we have a product that requires access to specified datastores for backups. In order to maintain access to these virtual machines for backup purposes, we need to be notified when new datastores are added.

As I sat and thought about how I could accomplish this task I came up with a couple ideas, but figured a scheduled task with PowerCLI/PowerShell would be the easiest to implement. In this script we will connect to the vCenter server, get all the datastores in the cluster, write a file daily with a date stamp, then compare the current and previous day’s datastore output files and write that to a third file that will only display the new datastores that have been added or the datastores that have been removed.

I’ve broken down the script so I can explain each section making it easy to understand. Before I had any knowledge of PowerShell/PowerCLI, modifying something to fit my environment when I didn’t understand what was happening at each step was time consuming and frustrating.

1. This is where we define the name of the vCenter instance we’ll be connecting to and the name of the cluster we’re interested in.

$vCenter = "LabvCenter.domain.com"
$Cluster = "LabCluster"

2. This is where we define the output location for our datastores and difference file. I chose to drop it into a folder named for the cluster, but that can be removed.

$filePath = "C:\test\" + $Cluster + "\"

3. This is where we connect to vCenter and then immediately wait 15 seconds which can fix issues of commands running before security warnings are displayed

Connect-VIserver $vCenter
Start-Sleep -s 15

4. This will gather all the datastores in the cluster and exclude any datastore that has a name containing “*-local”. The wildcard is important because the local datastores contain “servername + -local” and if the wildcard wasn’t there all of the local datastores would be included because no datastore is named exactly “-local”

$Datastores = Get-Cluster -Name $Cluster | Get-Datastore | Where {$_.Name -notlike "*-local"}

5. I prefer the format of 2 digit month, 2 digit day, 2 digit year. This will get the current date of the system running this script, then convert it to this format of 051415 for example.

$today = (Get-Date).ToString("MMddyy")
$yesterday = (Get-Date).AddDays(-1).ToString("MMddyy")
$2DaysAgo = (Get-Date).AddDays(-2).ToString("MMddyy")

6. This will set the file name and location for the output from 2 days ago. If that file exists, it will be removed. Rather than have an output from every day saved until I manually remove it, this process seemed better. I chose to delete the file from 2 days ago as opposed to deleting yesterday’s file after we run the comparison in case we see a huge change in the difference file we can manually compare the 2 files to try to find the error.

$2DayOldFile = $filepath + $Cluster + $2DaysAgo + ".txt"
If (Test-Path $2DayOldFile){Remove-Item $2DayOldFile}

7. This will set the file path and name to the file path defined at the top, plus the cluster name, plus the date and add .txt to the end.

$CurrentFile = $filePath + $Cluster + $today + ".txt"
$YesterdaysFile = $filePath + $Cluster + $yesterday + ".txt"

8. Here we are exporting all the datastores from Step 4 by name and outputting to the file name/path defined in Step 7.

$Datastores | Select Name | Out-File $CurrentFile

9. This is where we set the name and path for the difference file that will track the datastore add/remove.

$DifferenceFile = $filePath + "Datastore-Changes" + ".txt"

10. This will read the content from today’s content and yesterday’s content.

$YesterdaysContent = Get-Content $YesterdaysFile
$CurrentContent = Get-Content $CurrentFile

11. Here we are comparing the content we just read in step 10.

$Compare = Compare-Object $YesterdaysContent $CurrentContent

12. The standard way “Compare-Object” outputs this data shows difference with a side indicator of <= or => depending on where the difference exists. Rather than remember which file was read first to determine whether a datastore was added or removed, we change the column names. If a datastore existed yesterday, but is missing today it is labeled as “Removed”. If a datastore didn’t exist yesterday, but does today it is labeled as “Added”.

$compare | foreach {
if ($_.sideindicator -eq '<=')
{$_.sideindicator = "Removed"}

if ($_.sideindicator -eq '=>')
{$_.sideindicator = "Added"}
}

13. This will take the results from step 11 with formatting of step 12 then change the column names. The list of objects compared is normally named “InputObject” and then “Added or Removed” is normally “SideIndicator”. Maybe that’s fine, but I prefer something a little easier to read. I’ve renamed “InputObject” to “Datastore” but also I add the current date and we change “SideIndicator” to “Added or Removed”. Once that is done, we output that file to the path and name defined in Step 9. The reason why we include the current date in the “Datastore” column is because we are using “-Append” with the “Out-File” command. This will add a dated entry of changes that occurred to the bottom of the existing (or new) output file. This means we aren’t overwriting the same file every day, we are just adding to it. In case you forget to check this file for a few days you won’t lose that data.

$Compare |
select @{l='Datastore' + ' - ' + (Get-Date);e={$_.InputObject}},@{l='Added or Removed';e={$_.SideIndicator}} |
Out-File -Append $DifferenceFile

Now that we know what this thing does, let’s see it in action. I have run the output over 3 days and this is how the output file is displayed. We can see that on 05-14-15 we added Lab-Datastore-10 which didn’t exist on 05-13-15. Then on 05-15-15 we removed Lab-Datastore-03 and we added -11 and -12.
image

When running the script I commented out the removal of the 2 day old file so we could compare manually. Now we have an output file created (Datastore-Changes.txt) that should show the differences.
image

Inside Datastore-Changes.txt we see that on 5/14 the datastore “Lab-Datastore-10” was added and on 5/15 we lost Lab-Datastore-03, but added 11 and 12.

image

We can delete this file at any time and the next time this script runs we’ll create a brand new file. This means there is no dependency on this file already existing in order for the script to run and doesn’t require us to keep a long list of all the datastore add/removes for all eternity. Now you’ll just need to save the script schedule it to run using Windows Task Scheduler.

Below is the full scripts with comments.

#Define the vCenter Server and Cluster
$vCenter = "LabvCenter.domain.com"
$Cluster = "LabCluster"

#Set the path location for the output files
$filePath = "C:\test\" + $Cluster + "\"

#Connect to the vCenter Server and sleep for 15 seconds (necessary for security warnings)
Connect-VIserver $vCenter
Start-Sleep -s 15

#Get a list of all the datastores
$Datastores = Get-Cluster -Name $Cluster | Get-Datastore | Where {$_.Name -notlike "*-local"}

#Get the current date in the correct format
$today = (Get-Date).ToString("MMddyy")
$yesterday = (Get-Date).AddDays(-1).ToString("MMddyy")
$2DaysAgo = (Get-Date).AddDays(-2).ToString("MMddyy")

#Delete the output from 2 days ago (Remove this section if you want to keep the history)
$2DayOldFile = $filepath + $Cluster + $2DaysAgo + ".txt"
If (Test-Path $2DayOldFile){Remove-Item $2DayOldFile}

#Set the filename to include today's date
$CurrentFile = $filePath + $Cluster + $today + ".txt"
$YesterdaysFile = $filePath + $Cluster + $yesterday + ".txt"

#Export those datastores to a TXT file
$Datastores | Select Name | Out-File $CurrentFile

#Set file name & path for difference file
$DifferenceFile = $filePath + "Datastore-Changes" + ".txt"

#Get the content for yesterday and today's files
$YesterdaysContent = Get-Content $YesterdaysFile
$CurrentContent = Get-Content $CurrentFile

#Compare yesterday's and today's files
$Compare = Compare-Object $YesterdaysContent $CurrentContent

#Change the source/target column to "Removed" and "Added"
$compare | foreach { 
      if ($_.sideindicator -eq '')
        {$_.sideindicator = "Added"}
     }

#Change the column name output to "Datastore + Date" and "Added or Removed" then output to file
 $Compare | 
   select @{l='Datastore' + ' - ' + (Get-Date);e={$_.InputObject}},@{l='Added or Removed';e={$_.SideIndicator}} |
   Out-File -Append $DifferenceFile

Deploy NetApp OnCommand Balance 4.2

OnCommand Balance is a virtual appliance deployed within vCenter that allows you to monitor the health of your VMware environment at the Virtual Machine, vCenter and Storage level. Having a single place that displays end-to-end performance allows you to spend less time troubleshooting performance issues and trying to correlate data and address potential issues in your environment.

I’ve been using OnCommand Balance (formerly OnCommand Insight Balance) for a few years now and it has saved countless hours finding issues in the environment. We’ve had historical data available to look at growth and performance trends, as well as increased demand on individual servers after code releases/updates. Having access to the information within the VMs (such as drive space filling up) also makes this an invaluable tool.

The following documentation will take you through the deployment process of the Virtual Appliance and initial setup. You will go through adding your vCenter hosts, storage controllers, creating saved credentials, connecting to Active Directory for authentication and provision a Windows proxy service for monitoring Windows Servers.

Prerequisites:
1. A user account with appropriate permissions to vCenter for OnCommand Balance to use
2. A domain account with permissions to access all monitored Windows machines (preferrably a Domain Admin account)
3. A separate Windows Server/VM that will be used as the Proxy service to monitor Windows machines
a. Must have latest version of Java 6 installed and User Account Control disabled
4. Username/password for the NetApps that will be monitored

Steps:
1. Download the latest version of OnCommand Balance (4.2) for this writing from the NetApp website
balance101414-step1
2. Connect to the vSphere web interface, click on “vCenter”, “Hosts and Clusters”, expand the Datacenter, and click on the Cluster/Host that will host OnCommand Balance. Right click and choose “Deploy OVF Template”
balance101414-step2
3. Click “Local file” and then “Browse”
balance101414-step3
4. Locate the OnCommand Balance OVA and click “Open” then click “Next”
balance101414-step4
5. Review the details of the OVF then click “Next”
balance101414-step5
6. Accept the EULA then click “Next”
balance101414-step6
7. Give the appliance a name and choose the folder location of the appliance (if any) and click “Next”
balance101414-step7
8. Set the virtual disk format (I prefer Thin since one of the drives is 220GB) and choose the datastore. Click “Next”
balance101414-step8
9. Choose the appropriate network and then click “Next”
balance101414-step9
10. Review the settings then click “Finish”
balance101414-step10
11. After deployment completes, locate the appliance, right click and choose “Power On”
balance101414-step11
12. Open the console of the VM (Right-click and choose “Open Console”) where you’ll see this countdown to install VMware tools prior to configuring the Balance virtual appliance (If you miss your chance to do this at this point, I was unable to install VMware tools at all on the appliance)
balance101414-step12
13. Right-click on the VM, go to “All vCenter Actions”, then “Guest OS” and then click “Install VMware Tools”
balance101414-step13
14. After the VMware tools dialog box is displayed, click “Mount”
balance101414-step14
15. The Balance virtual appliance should recognize VMware tools ISO has been mounted and proceed with the installation
balance101414-step15
16. After VMware tools install completes, press “y” then enter to configure static Network connection for the management interface
balance101414-step16
17. Enter the following information:

a. Host name
b. Host IP address
c. Netmask
d. Gateway
e. Primary DNS address
f. Secondary DNS address
g. Search domains
balance101414-step17g

18. Review the settings and then press “y” and enter if everything is correct
balance101414-step18
19. Default OnCommand Balance console login is netapp/netapp. Login to the console
balance101414-step19
20. After a few minutes (5-10) the web service will be up and running. Connect to the https://IPofAppliance/bp to begin configuration
21. Enter the name of your organization and click “Continue”
balance101414-step21
22. Choose if you want to participate in AutoSupport and click “Submit”
balance101414-step22
23. Enter the time zone, NTP Server address, the address of the primary Balance admin (preferably a distribution group), and the SMTP server address. Click “Continue” (You can choose to change the password at this time)
balance101414-step23
24. Sit around and wait a couple minutes…
balance101414-step24
25. A blank screen may appear during this time, but eventually should take you to the OnCommand Balance login page. Login with the default credentials of admin/password or whatever password was set in step 23.
balance101414-step25
26. Click the link for “Configure you storage arrays & appliances”
balance101414-step26
27. Choose the type of storage (NetApp FAS in this case), enter the management address for one of the nodes, Enter the name of the filer, enter the credentials (root in my case) and enter a nickname of these credentials as they can be modified later on during password changes. Click “Save”
balance101414-step27
28. Even though DNS is configured correctly, I usually receive this error about the other filer of this HA system not being resolvable. Click “Enter IP address instead” and then enter the IP of the other filer and click “Resolve”
balance101414-step28
29. Click the “Refresh” link on the right side of the page a few times until “Discovery Collection” status changes to “OK”
balance101414-step29
30. Click the “Add storage system” button to add additional storage arrays (Including the HA partners). Click on “Dashboard” then choose “Configure your vCenter Server”
balance101414-step30
31. Enter the FQDN/IP Address of the vCenter server. Click “Add new” next to Credentials to add the credentials for the vCenter server
balance101414-step31
32. Enter the username, password, and nickname for these credentials. Click “Next”
balance101414-step32
33. Choose what you want monitored (though I can’t imagine why you’d choose not to monitor everything) and click “Save”
balance101414-step33
34. Click the “refresh” link until “Discovery Collection” status changes to “OK”
balance101414-step34
35. Click “Add vCenter Server” button to add any additional vCenter servers. Otherwise, hover over “Discovery” and choose “Credentials”
balance101414-step35
36. To monitor the OS’s of your VMs and physical servers, you can add those credentials on this page. I’ll add domain admin credentials for monitoring my Windows domain VMs. Click “Add credentials” button
balance101414-step36
37. Choose the login method, login name (domain\username), password, nickname for the credentials, and a description. Click “Save”
balance101414-step37
38. Once added they will appear on this screen
balance101414-step38
39. Hover over “Discovery” and choose “Proxies”
balance101414-step39
40. A proxy is required to monitor the guest OS status of Windows VMs and Physical servers. This proxy runs on a windows server. Once you’ve determined (or built) the appropriate server for the proxy, enter it’s FQDN or IP address and click “Continue”. Much like the picture below says, UAC MUST be disabled. You’ll beat your head against the wall for hours trying to figure out why it fails without that.
balance101414-step40
41. Download and install the latest 32-bit Java 6 runtime on this proxy server. Then navigate to the link listed on that proxy VM to begin the installation
balance101414-step41
42. Once the Balance Proxy Installer screen appears, click “Next”
balance101414-step42
43. Locate the folder path for the 32-bit java install and click “Next”
balance101414-step43
44. Enter an admin account for the service to be run under. Check the box for “Start service immediately after install” and click “Next”
balance101414-step44
45. Select any additional components you might need for other vendors and click “Next”
balance101414-step45
46. Review the information and click “Install”
balance101414-step46
47. Click “Finish”
balance101414-step47
48. Back at the Balance web interface, click “Validate proxy setup” and if successful, click “Continue”
balance101414-step48a
balance101414-step48b
49. Hover over “Discovery” and click on “Servers”
balance101414-step49
50. Click the link on the right side for “Unmonitored Servers”
balance101414-step50
51. Click the link next the vCenter server for “# guests are not being monitored”
balance101414-step51
52. Check the box next to the VMs you wish to monitor, choose your Credentials from the dropdown box in the center and click “Monitor guest(s)”
balance101414-step52
53. Hover over “Admin” and choose “Configuration”
balance101414-step53
54. Click “Email”. In here you can set authentication for your SMTP server, choose the “From” address for Balance emails. Click “Enable alerts” and then check all the boxes for Critical, Warning, and all categories (I prefer as many alerts as I can get). Click “Update”
balance101414-step54
55. Click on “Active Directory” and click the check box for “Enable Active Directory”
balance101414-step55
56. Enter the IP/hostname of your AD server, enter the Distinguished name of the account used to search Active Directory, and enter the password for that account. Click “Test”
balance101414-step56
57. Once successful, enter the Distinguished Name of the of the OU for the user/group that will have access to login. Enter the Distinguished name of the Group that will be able to login. Enter “sAMAccount” for the search attribute. Click “Update”
balance101414-step57
58. Hover over “Admin” and click “Users”
balance101414-step58
59. Click “Add User”
balance101414-step59
60. Change “Authentication” to “Active Directory”. Enter the username and click “Lookup”. If successfully, configured, it should populate the e-mail address. Choose the appropriate user type (Admin or User) and click “Save”
balance101414-step60

You’re all setup and ready to let OnCommand Balance start collecting data in your environment. You start to receive some information within about 30 minutes, but after 3-5 days you start to get a better understanding of what is going on in your environment and have more useful metrics.