Get Host Alarms and Error Messages

vCenter alarms are good, but after you finish configuring emails for all the important alarms you want to receive that is when someone creates an email rule to ignore all these alarms. This started me down the path of adding an environment reporting email to let me know if there are any actual issues I need to worry about when I get to work in the morning.

Since the report I have runs at the same time each day I created a variable called “yesterday”. This will be used to find error messages that happened since yesterday’s run.

$yesterday = (Get-Date).AddDays(-1)

Then we need to find host error events. I’m getting every host, then getting all the error events for each host that were generated after $yesterday.

$hostErrors = Get-VMHost | Get-VIEvent -Types Error -Start $yesterday

Previously, I was filtering the start time for a CreatedTime greater than yesterday because I didn’t realize -Start and -Finish were options. I measured the difference and I saved over 2 minutes by using the -Start switch I saved over 150 seconds.

The output events have a lot of info we aren’t interested in. The only thing you need to capture is the Full Formatted Message

We’ll display the output using $hostErrors.FullFormattedMessage

Full script:

$yesterday = (Get-Date).AddDays(-1)
$hostErrors = Get-VMHost | Get-VIEvent -Types Error -Start $yesterday
$hostErrors.FullFormattedMessage

Now that’s great for error messages, but what about alarms?

First we’ll get all the hosts that have an active alarm.

$triggeredhosts = get-vmhost | get-view | where {$_.TriggeredAlarmState.OverallStatus -ne $null}

In my environment you can see that I have 1 host with an alarm, but when we view the TriggeredAlarmState we don’t see what the alarm is for or even the name of the host that has an alarm:

Now we need to do some lookups on this data to give us useful information. Because we can have more than 1 alarm at a time, we need to build a hash array and state populating data.

First we create the hash array:

$alarmReport = @()

Now we create a ForEach loop for every triggered event. We use $triggeredHosts.TriggeredAlarmState instead of just $triggeredHosts because we can have multiple alarms per host and since we’ll resolve the host name on each alarm in this loop this is an easier way to do it.

ForEach ($trigger in $triggeredHosts.TriggeredAlarmState) {

Then we resolve the host system ID to name:

$hostName = Get-VMHost | Where {$_.Id -eq $trigger.Entity}

Now we need to get the type of alarm:

$getAlarm = (Get-View -Id $trigger.Alarm).Info.Name

Then we define the columns for our array. We’re capturing host name, cluster name, the state of the host (connected, maintenance, not responding), and then the name of the alarm:

$row = "" | Select Host, Cluster, State, Alarm

Now we can populate our columns:

$row.Host = $hostName.Name
$row.Cluster = $hostName.Parent
$row.State = $hostName.State
$row.Alarm = $getAlarm

Now we add this row to the hash array and close the ForEach loop:

$alarmReport += $row }

And when we view the output of $alarmReport we see the host name, the cluster, the state, and the current active alarm.

Here’s the whole script:

$triggeredhosts = get-vmhost | get-view | where {$_.TriggeredAlarmState.OverallStatus -ne $null}
$alarmReport = @()
ForEach ($trigger in $triggeredHosts.TriggeredAlarmState) {
$hostName = Get-VMHost | Where {$_.Id -eq $trigger.Entity}
$getAlarm = (Get-View -Id $trigger.Alarm).Info.Name
$row = "" | Select Host, Cluster, State, Alarm
$row.Host = $hostName.Name
$row.Cluster = $hostName.Parent
$row.State = $hostName.State
$row.Alarm = $getAlarm
$alarmReport += $row }

Track Powered Off VMs in vCenter

Tracking powered off VMs is a struggle most people have. Depending on the number of VM Admins there are in your environment, each person can have a different way of tracking. We’ve tried renaming VMs, placing them in a “Powered Off” labeled folder, adding notes to the VM, but the real issue is that if it isn’t automated and you don’t include the date you aren’t ever sure what VMs you can safely delete.

As part of my Daily vCenter health script I now automatically add a date to each powered off VM so I know what VMs I can delete. The only prerequisite is to create a custom attribute called “OffDate” or anything else you want for the VMs.

First thing we do is get the current date in a readable and reportable format. So we use the $date variable and get today’s date then convert it to month-day-year format:

$date = (Get-Date).ToString('MM-dd-yyyy')

After that we get a list of VMs that are currently powered off and that don’t live in a folder that contains the word “template”. In our environment we have a handful of VMs that haven’t been converted to templates, but are used as templates for provisioning. I don’t want to accidentally tag and delete these so they get excluded.

$vmlist = Get-VM | Where {$_.PowerState -eq "PoweredOff" -AND $_.Folder -notlike "*template*"}

Then we create a variable of the custom attribute we’ll be searching for.

$attributeName = "OffDate"

Now we need to gather a list of VMs that are Powered off, but do not currently have their custom attribute populated. We are doing a Get-Annotation for that attribute name we just defined and looking for a value that is empty.

$noDateVM = ForEach ($vm in $vmlist) {$vm | Get-Annotation -CustomAttribute $attributeName | Where {$_.value -eq ""}}

Once we have a list of each of those VMs, we can now tag each of these VMs with the current date. I capture this as a variable so I can add the output to my daily report. I like to know what VMs are getting tagged in case something is added by mistake or an important VM was powered off that shouldn’t have been.

$offDateList = ForEach ($blankVM in $noDateVM) {$blankVM.AnnotatedEntity | Set-Annotation -CustomAttribute $attributename -Value $date}


With the tagging complete, we can now add it to our report. I do HTML formatted reports and I don’t want the section to be in the email if it’s empty. In this portion we check to see if $offDateList is empty and if it is we don’t do anything else. However, if there are VMs, we then create the output with a title and a list the VM along with today’s date.

IF (!$offDateList) {} ELSE {
$outputdateOutput = $offDateList | Select @{N="VM Name";E={$_.AnnotatedEntity}},Value | ConvertTo-Html -PreContent "<h4>New Powered Off VMs</h4>" | Out-String

Here is the complete script:

$date = (Get-Date).ToString('MM-dd-yyyy')
$vmlist = Get-VM | Where {$_.PowerState -eq "PoweredOff" -AND $_.Folder -notlike "*template*"}
$attributeName = "OffDate"
$noDateVM = ForEach ($vm in $vmlist) {$vm | Get-Annotation -CustomAttribute $attributeName | Where {$_.value -eq ""}}
$offDateList = ForEach ($blankVM in $noDateVM) {$blankVM.AnnotatedEntity | Set-Annotation -CustomAttribute $attributename -Value $date}
IF (!$offDateList) {} ELSE {
$outputdateOutput = $offDateList | Select @{N="VM Name";E={$_.AnnotatedEntity}},Value | ConvertTo-Html -PreContent "<h4>New Powered Off VMs</h4>" | Out-String }

But wait, there’s more.

Sure, we can tag VMs, but now we need to report on any VMs that have a Powered Off date, but are currently running.
First thing we do is grab a list of Powered On VMs:

$poweredOnVMs = Get-VM | Where {$_.PowerState -eq "PoweredOn"}

Now for each of these VMs we want to find those that have an OffDate that isn’t empty:

$runningVMs = ForEach ($VM in $poweredOnVMs) {
$vm | Get-Annotation -CustomAttribute $attributeName | Where {$_.value -ne ""}}

Once again, if the list is empty, I don’t want it to show up in my report so if it’s empty we don’t do anything else. If not, we create our output:

IF (!$runningVMs) {} ELSE {
$outputrunningVMReport = $runningVMs | Select @{N="VM name";E={$_.AnnotatedEntity}},Value | ConvertTo-Html -PreContent "<h4>Running VMs with Powered Off Date</h4>" | Out-String }

And here is the full script:

$poweredOnVMs = Get-VM | Where {$_.PowerState -eq "PoweredOn"}
$runningVMs = ForEach ($VM in $poweredOnVMs) {
$vm | Get-Annotation -CustomAttribute $attributeName | Where {$_.value -ne ""}}
IF (!$runningVMs) {} ELSE {
$outputrunningVMReport = $runningVMs | Select @{N="VM name";E={$_.AnnotatedEntity}},Value | ConvertTo-Html -PreContent "<h4>Running VMs with Powered Off Date</h4>" | Out-String }

Update Host DCUI with PowerCLI

With randomly generated host names keeping track of what host you’re working on can get difficult. Recently I was in the process of performing host name changes along with cluster migrations/rebalancing which made it even more difficult to keep track of what I was working.

So, of course, I decided to write a script that will update the DCUI of a host.

Normally this is what we see in the DCUI:

We can see the name, IP address, processors, RAM, version and build. A decent amount of info, but I wanted cluster, powered on VM count, and even the serial number of the server.

Now for the script itself.

First we need a list of all the hosts that are currently available. No need to try to update a host that is not currently responding:

$allHosts = Get-VMHost | Where {$_.ConnectionState -ne "NotResponding"}

Now we open a ForEach loop so these commands can be run against each of these hosts:

ForEach ($vmhost in $allHosts) {

First we’ll get a count of powered on VMs on the host:

$vmCount = ($vmhost | Get-VM | Where {$_.PowerState -eq "PoweredOn"}).Count

Now we’ll get the cluster name:

$cluster = $vmhost.Parent.Name

Then we capture the make and model of the host:

$make = "$($vmhost.Manufacturer) $($vmhost.model)"

Let’s capture the number of CPUs:

$cpuCount = $vmhost.ExtensionData.Summary.Hardware.NumCpuPkgs

Then we’ll get the model of the CPUs:

$cpuModel = $vmhost.ExtensionData.Summary.Hardware.CpuModel

And let’s get the amount of RAM in GB rounded to 2 decimal places:

$memory = [math]::round($vmhost.MemoryTotalGB,2)

This is where create our DCUI text and use the variables we’ve captured above:

$DCUIString = "{color:white}
		{esxproduct} (VMKernel Release {esxversion})
		
		$make
		
		$cpuCount x $cpuModel
		$memory GB Memory
		
		ServiceTag: {servicetag}
		{/color}
		
		{color:yellow}
		Hostname: {hostname}
		IP Address: {ip}
		Cluster: $cluster
		Powered On VMs: $vmCount
		{/color}
		
		
		
		
		<F2> Customize System/View Logs
		<F12> Shut Down/Restart
		"

Then we apply these settings to the host and close the ForEach loop:

$vmhost | Set-VMHostAdvancedConfiguration -Name Annotations.WelcomeMessage -Value $DCUIString }

The script will loop through all your hosts and now you’ll see a DCUI that looks like this:

I have this script scheduled to run daily so it is mostly up to date. In addition, I run another version every 15 minutes. In the event that a host has been placed in maintenance mode we update only that host so we can see if any powered on VMs reside on it.

For that to work, replace the first part of the script searching for all hosts with the part below:

$allHosts = Get-VMhost | where {$_.connectionstate -eq "Maintenance"}

Full script below:

$allHosts = Get-VMhost | where {$_.connectionstate -ne "NotResponding"}
ForEach ($vmhost in $allHosts) {
$vmCount = ($vmhost | Get-VM | where {$_.powerstate -eq "poweredon"}).count
$cluster = $vmhost.parent.name
$make = "$($vmhost.Manufacturer) $($vmhost.model)"
$cpuCount = $vmhost.ExtensionData.Summary.Hardware.NumCpuPkgs
$cpuModel = $vmhost.ExtensionData.Summary.Hardware.CpuModel
$memory = [math]::round($vmhost.MemoryTotalGB,2)
$DCUIString = "{color:white}
		{esxproduct} (VMKernel Release {esxversion})
		
		$make
		
		$cpuCount x $cpuModel
		$memory GB Memory
		
		ServiceTag: {servicetag}
		{/color}
		
		{color:yellow}
		Hostname: {hostname}
		IP Address: {ip}
		Cluster: $cluster
		Powered On VMs: $vmCount
		{/color}
		
		
		
		
		<F2> Customize System/View Logs
		<F12> Shut Down/Restart
		"
$vmhost | Set-VMHostAdvancedConfiguration -Name Annotations.WelcomeMessage -Value $DCUIString}

vSAN – Check VM Storage Policy & Compliance

As I continue to work with vSAN I discover there’s way more to do than just move some VMs over and you’re on your way. With multiple vSAN clusters each with different configurations I needed a way to monitor the current setup and check for changes. While creating a simple script to check which VM Storage Policy is assigned to each VM isn’t very difficult, a creating a script to check the storage policy of VMs across multiple vSAN datastores proved to be a little more difficult.

We run multiple PowerCLI scripts to check health and configuration drift (thanks to a special tool created by Nick Farmer) in our environment. In the event that a new vCenter is added or new vSAN datastore is deployed, we needed a simple script that can be run without any intervention or modification. Now we can be alerted when the proper VM storage policies isn’t assigned or the current policy is out of compliance.

To further complicate things in our setup, we create a new VM Storage Policy that contains the name of the cluster in which it’s assigned. Due to the potential differences in each vSAN cluster (stripes, failures to tolerate, replication factor, RAID, etc) having a single Storage Policy does not work for us. In the event a VM is migrated from one vSAN cluster to another we need to check that the VM storage policy matches vSAN datastore cluster policy.

What this script does is grab all the clusters in a vCenter that have vSAN enabled. For each cluster that is found with vSAN enabled, it is filtering only the VMs that live on vSAN storage (with the name of “-vsan”. Then we get the storage based policy management (Get-SpbmEntityConfiguration) of those VMs. The script then filters for a storage policy that doesn’t contain the cluster name OR a compliance status that is compliant.

$vsanClusters = Get-cluster | Where-Object {$_.vsanenabled -eq "True"}
foreach ($cluster in $vsanClusters)
{
$Cluster | get-vm |?{($_.extensiondata.config.datastoreurl|%{$_.name}) -like "*-vsan*"} |
Get-SpbmEntityConfiguration | Where-Object {$_.storagepolicy -notlike "*$Cluster*" -or $_.compliancestatus -notlike "*compliant*"} |
Select-Object Entity,storagepolicy,compliancestatus
}

Once this is run we can see the output below. I’ve obscured the names of the VMs, but we can see that there are still 12 VMs that are using the default vSAN Storage Policy instead of the cluster-specific storage policy they should be using. In addition, we see that the compliance status is currently out of date on most of these VMs. These VMs reside on 2 separate clusters and there are also 2 VMs that were filtered because they are on local storage in these clusters instead of vSAN.

storagepolicy01-12202016

Add NFS Datastore to Cluster via PowerCLI

I have been digging into more and more PowerCLI the last month or so trying to explore faster ways to accomplish common tasks. Using the NetApp VSC plugin inside vCenter I can provision a brand new NFS datastore to an entire cluster in just a few clicks, but there is no built in way to do this for mounting an existing datastore. The below script is just a simple way to mount an NFS datastore to a named cluster.

$ClusterName = "ProdCluster"
$DatastoreName = "VM_Win2003_NA5"
$DatastorePath = "/vol/VM_Win2003_NA5"
$NfsHost = "192.168.1.5"
get-cluster $ClusterName | get-vmhost | New-Datastore -NFS -Name $DatastoreName -Path $DatastorePath -NfsHost $NfsHost

Or you can replace each variable with the actual value in the script when mounting multiple datastores in the same script.

get-cluster "ProdCluster" | get-vmhost | New-Datastore -NFS -Name "VM_Win2003_NA5" -Path "/vol/VM_Win2003_NA5" -NfsHost 192.168.1.5

The next step here will be running this script from vCO and passing the variables directly from vCO. Maybe one day I’ll have the time to figure out just how to do that…