VSAN – Compliance Status is Out of Date

Occasionally the Compliance status of the performance service will go to the “out of date” status. This is not an alert that is thrown anywhere within vCenter. You will have to check this status by logging into the vSphere web client, locating your vCenter, choose the cluster, clicking on “Manage” then choosing “Health and Performance” under “Virtual SAN”
ComplianceStatus-a

As I have recently fixed this issue the above screenshot shows the “Compliant” status. Below are the steps to get to that point.

1. In the box for “Performance Service” click “Edit storage policy”
ComplianceStatus-01

2. If there is a storage policy available in the drop down, select it and click “OK”. This will apply that policy and perform the compliance check.
ComplianceStatus-02

For the lucky few where that works, that’s all you need to do. If the storage policy list is empty you’ll need to restart the vsanmgmtd service on each of the hosts.

3. Enable SSH on each of the hosts in the VSAN cluster and using an SSH client (like putty), SSH to a host and run the following command to restart the vsanmgmtd service (this is a non-impactful operation and should be able to be performed during production hours with no impact)
a. /etc/init.d/vsanmgmtd restart

4. Repeat that command on each of the hosts in the cluster until they have all restarted their services
ComplianceStatus-04

5. Wait 5 minutes and then check to see if you are able to select a storage policy for the performance service. If not, move on to step 6

6. Now we’ll need to restart the vSphere Profile-Driven Storage Service on the vCenter server. This is also non-impactful and should be able to be performed in the middle of the day. If you’re using vCenter on windows, connect to the Windows server and restart the “Vmware vSphere Profile-Driven Storage Service”. If using VCSA (like this example) you’ll need to SSH to the VCSA and run the command below
a. Service vmware-sps restart

7. After the vmware-sps service restarts, log out of the web client and wait for 5 minutes while the storage profile service completes its restart.

8. Log back in to the web client, navigate to the vCenter server, click “Manage” then choose the “Storage Providers” tab
ComplianceStatus-08

9. Click the Synchronize Providers button to resync the state of the environment
ComplianceStatus-09

10. Wait another 5 minutes while these synchronize completes. After 5 minutes, navigate to the VSAN cluster in the web client. Click on “Manage” then choose “Settings” and locate “Health and Performance” under the “Virtual SAN” section
ComplianceStatus-10

11. In the Performance Service box, click the “Edit Storage Policy” button
ComplianceStatus-11

12. From the drop down list you should be able to select the appropriate VSAN storage policy and then click “OK”
ComplianceStatus-12

13. After this is selected the compliance status should change to “Compliant” and you should be all set.

So far these are the only steps that I have needed to follow in order to fix this issue. Let me know if there are any other fixes available.

Deploy VSAN Witness Appliance

The VSAN Witness Host is a virtual appliance that is deployed into an existing vCenter server. When deploying a 2 node or stretched cluster, the witness appliance acts as a tie breaker to determine which node(s) are still available in the event the nodes lose communication with each other. The witness The witness is deployed just like any other virtual appliance, but will require access to the management network and the network you’ve designated as your VSAN network. This appliance must be run OUTSIDE your VSAN cluster. This means that you cannot add this host as a member of the existing VSAN cluster and you also should not run it as a virtual appliance inside your existing VSAN cluster.

1. Choose the cluster that will host the appliance. Click on “File” then “Deploy OVF Template”
step01

2. Browse to select the .OVA file and click “Next”
step02

3. Review the details of the appliance and click “Next”
step03

4. Review the license agreement and click “Accept” followed by “Next”
step04

5. Enter the name of the Witness Appliance and its location then click “Next”
step05

6. Choose the appropriate size of the appliance and click “Next”
step06
a. As this is a test, I’m choose the “Tiny” size. You can ignore the disk component requires for any size. As this is a virtual appliance, it will deploy the appropriately sized drives that will designated as SSD and spinning disl

7. Choose the provisioning type and click “Next”
step07
a. This is appliance is being deployed to a separate VSAN cluster than the one it will be acting as the witness for. This appliance can be deployed on shared storage, local storage, or another VSAN datastore.

8. Choose the appropriate networks for management and witness (VSAN). In this deployment, management lives on the “VM Network” and witness (VSAN) traffic is on the “VM-VSANnetwork”. This network is shared with the vMotion network and just needed an additional VM Portgroup created on each of the hosts in the cluster where this appliance is being deployed. Click “Next”
step08

9. Enter a root password for this appliance. Remember, this is a host that you will need to login to in order to administer so if there is a standard root password that you use it would be a good idea to use that here. Click “Next”
step09

10. Review the deployment settings and then click “Finish”
step10

11. Once deployed, you will need to configure the appliance like any other host. Power on the appliance and open the console, press F2 and login as root with the password you assigned in step 9
step11

12. Scroll to “Configure Management Network” and press “Enter”
step12

13. Ensure the Network Adapter assigned to your management network is “vmnic0”
step13
a. Set a VLAN (if necessary) for the management network, then assign your IPv4 and/or IPv6 settings for the management network to make it accessible on your network. Assigned DNS as needed as well. Press “ESC” and then press “Y” to configure settings and restart the management network
step13a

14. Once the host can communicate on the network, add it as a new host in vCenter.
a. Remember that this host should not be part of your VSAN cluster or any other cluster. It should be a standalone host in your datacenter.

15. Select the host in the vCenter client and configure networking for it. Locate the “witnessSwitch” and click “Properties”
step15

16. Select the “witnessPg” and click “Edit”
step16

17. On the “IP Settings” tab, enter the IP and subnet mask for the VSAN traffic network. Click “OK” at the bottom”
step17

18. Once you have confirmed that network settings are successful, login to the vSphere web client and navigate to the VSAN cluster to be configured

19. Click the “Manage” tab, then choose “Fault Domains & Stretched Cluster” under “Virtual SAN”
step19

20. In the “Streteched Cluster” box click “Configure”
step20

21. Name the fault domains and place the hosts into the appropriate fault domain. This is a 2 node cluster with 1 host in each fault domain. Click “Next”
step21

22. Locate the VSAN witness appliance host that was added to this vcenter and click “Next”
step22

23. Choose the flash drive for and the HDD for cache and capacity and click “Next”
step23

24. Review the settings and click “Finish”
step24

25. Once completed, you will now see the status of the stretched cluster as “Enabled”, the preferred fault domain and the designated witness host.
step25

VSAN – Host Not Contributing Stats

After an upgrade or maintenance on one or more of the nodes in a VSAN cluster one of the hosts can stop contributing performance stats. This is not a production down issue, but should be addressed to see the most up-to-date stats across all the nodes.

The fix for this is one of three things, but each of them involves turning off performance statistics on the cluster which will cause all historical performance stats to be removed. My hope is that VMware will fix this issue in an upcoming release because a loss of historical is not tolerable in all environments.

1. View the health of the VSAN by logging into the vCenter web client. Navigate to the appropriate vCenter and cluster, then click the “Monitor” tab, followed by “Virtual SAN” then click on “Health.” Expanded “Performance service” and click the warning for “All hosts contributing stats”
step01

2. At the bottom you will now see the list of hosts that are not contributing stats
step02

3. Now that we’ve identified the problem host, we need to disable VSAN performance service temporarily. Navigate to the “Manage” tab for this cluster then click on “Health and Performance” under “Virtual SAN”
step03

4. Click “Turn off” in the “Performance Service” box

a. Click “OK” to confirm stopping the service which will erase all existing performance data
step04a

5. Confirm Perform Service has been disabled by refreshing the page
step05

6. SSH to the affected host (using putty or similar SSH client) we identified in step 2 (you may have to enable SSH on the host before you can connect).
step06

7. Run the command below to restart the VSAN management agent. This should have no production impact so it is safe to perform outside of a maintenance window.
a. /etc/init.d/vsanmgmtd restart
step07a

8. Once the service has been restarted, go back to the vCenter web client and the click the “Edit” button for the Performance Service box
step08

9. Select the appropriate storage policy from the drop down list, ensure the “Turn ON Virtual SAN performance service” box is checked and click “OK”
step09

10. Confirm that the performance service is turned on and reporting healthy
step10

11. Navigate back to the “Monitor” tab and then “Virtual SAN” clicking the on the “Health” section. Click “Retest” to verify that all hosts are contributing stats.
step11

If this does not fix the issue, you can restart the process, but this time instead of restarting the vsanmgmt service on the one node, do it on all of the nodes in the cluster. Once the services have been restarted across all nodes then restart the performance service and all nodes should be contributing stats.

I have also seen a case where restarting the service on all nodes didn’t fix the problem. In that scenario I was able to fix the problem by entering maintenance mode on the problem node and choose “full data migration” so all the data would be removed from the cluster. After that was complete I completely rebuilt the host from scratch (including wiping the disks claimed by VSAN) then moving it back into the cluster. I haven’t heard from VMware of any other ways to fix this issue.

Cohesity – Scale-Out Secondary Storage

Backups are boring. No matter if you’re talking about swapping tapes, configuring backup jobs in your legacy agent-based  software, or spending another night restoring snapshots from your storage array; there’s just no way to make backups interesting. Cohesity aims to fix that. No, they won’t make backups sexy, but they are looking to add a bit more flash to the secondary storage market.

So what exactly is “secondary storage?” Secondary storage encompasses our backups, non-prod workloads, fileshares and the like. The secondary storage market is gaining visibility recently. With the flood of primary storage vendors, Cohesity could have been another “me-too” primary storage vendor, but they see the value in attacking an under-developed market.

The concept of Cohesity is simple. You can purchase the C2300 or C2500 models which offer 48TB or 96TB of storage respectively in each 4-node appliance (with a minimum of 3-nodes to start). Additional capactity can be added a single node at a time afterwards in 12TB or 24TB chunks depending on the model. Each node contains either 800GB or 1.6TB of flash for caching along with compute and memory. Cohesity claims they are infinitely scalable due to their distributed OASIS (Open Architecture for Scalable Intelligent Storage) architecture, though they’ve only tested up to 32 nodes at the time of this writing. Once your nodes are setup, you just point Cohesity at your vCenter Server and you now have visibility of your virtual machines.

Cohesity, leveraging VADP, is able to snapshot your configured VMs and begins ingesting all that data. The changes of these VMs are tracked (using CBT) so you’re not performing new full backups each time. All that is pretty standard in the backup world, so what sets Cohesity apart? That data is not just backed up, but it is available to actually use. Want to spin up one of these backed up VMs for testing? Space-efficient clones are created directly on the Cohesity appliance and are presented back to your ESXi hosts. Searching for a file to restore from one of these VMs? You can locate it right from the web interface and download the file without having to restore the entire VM.

The differentiator for Cohesity is not just how it scales or how simple it makes the backup process, but how it makes your backups useful. Enabling developers to access clones of your production systems to test deployments and hotfixes without impacting your production storage. Integrated QoS preventing your dev/test workloads from consuming all your resources and causing backup performance to suffer. Utilizing the onboard flash combined with global deduplication, performance of these workloads can mimic production without the cost of an all flash array.

An all-inclusive secondary storage appliance that provides visibility of data sprawl adds to the value. Often times, as production systems are backed up and cloned and cloned again, you lose sight of the origin of that data. Migrating data from one storage array to another you lose that deduplication and you’re now increasing capacity across systems to accommodate your storage footprint. By providing an all-in-one solution for your backups and dev/test workloads, you’re able to maximize your investment without the need for multiple arrays and storage targets.

The backup market is a crowded one. There are more feature rich backup software providers in the space, but many of them require the purchase of additional storage that doesn’t have the capabilities of what Cohesity provides. Having just released Version 1 in mid-October, Cohesity has a lot of capabilities in their software with what appears to be a great vision for the future. The product is still in need of refinement to simplify the process of searches, reporting, and scheduling, but the foundation of what the Cohesity team has built has me excited to see where they’ll be able to take their product.

__________

Watch all the videos from Cohesity at Storage Field Day 8 here.

Disclaimer: During Storage Field Day 8, my expenses (flight, hotel, etc) were paid for by Tech Field Day. I am under no obligation to write about any of the presented content nor am I compensated by any of the presenting companies for such writing.

VeeamON 2015

As exciting as VMworld is each and every year, it appears to have lost its charm. Yes, it’s the largest gathering of virtualization people (my people) each and every year, but that is part of the problem. 20,000+ of my closest friends isn’t quite as intimate as it sounds. Oh, we’re close, but that’s more about proximity than the strength of our friendships.

The value of VMworld is the sessions, labs and networking opportunities. With sessions and labs being available after the conference ends, it’s important to maximize those networking opportunities as best you can. Trying to find someone in a crowd of 20,000 is difficult. If you’re not stalking your favorite virtualization professionals on Twitter you’ll likely miss them. Enter VeeamON.

This is the second year for VeeamON and my first year in attendance. Veeam, everyone’s favorite data protection & availability company (or at least mine), holds their annual gathering of data availability experts in Las Vegas (at the Aria hotel this year). What is normally just a function of our jobs is brought to the forefront in this conference. The size of VeeamON which has an expected attendance of 2500+ people will see many more opportunities to get one-on-one time with those in attendance.

VeeamON focuses on community, much like Veeam itself has done over the years. This dedication to educating and enabling the community is what has made Veeam Software successful and respected in the industry. Veeam is bringing out the vBrownbag crew for 19 sessions that will be available live during the conference as well. Its users and the community have helped shaped Veeam and built very passionate advocates.

With vendor sponsored sessions that consist of integration with the various aspects of the Veeam software features (such as Veeam Explorer for Storage Snapshots) as well as a focus on customer and user sessions that describe their use of Veeam solutions in their environment. That’s just the beginning. Other sessions talk about data protection as a whole and the concept of data availability in our modern datacenters. Being a Veeam user or customer isn’t required to walk away with new concepts, ideas and knowledge. Click here to view the agenda.

We’re past the point of just needing backups, we need our data readily available in many different and evolving scenarios. From backups, to disaster recovery, to sandboxes for testing, data availability gets reduced to just having a copy of your data. Veeam is pioneering the data availability movement and has the tools in place to bring data availability to the always-on enterprise. Above all else, Veeam is on your side.

Track Datastore Add & Removes With PowerCLI

While working with the data protection team at my job I was asked if there was any way to track new datastores being added to a vSphere cluster. When new LUNs are allocated to our vSphere clusters, the data protection team isn’t always made aware ahead of time. Normally this isn’t a big deal, but in our case we have a product that requires access to specified datastores for backups. In order to maintain access to these virtual machines for backup purposes, we need to be notified when new datastores are added.

As I sat and thought about how I could accomplish this task I came up with a couple ideas, but figured a scheduled task with PowerCLI/PowerShell would be the easiest to implement. In this script we will connect to the vCenter server, get all the datastores in the cluster, write a file daily with a date stamp, then compare the current and previous day’s datastore output files and write that to a third file that will only display the new datastores that have been added or the datastores that have been removed.

I’ve broken down the script so I can explain each section making it easy to understand. Before I had any knowledge of PowerShell/PowerCLI, modifying something to fit my environment when I didn’t understand what was happening at each step was time consuming and frustrating.

1. This is where we define the name of the vCenter instance we’ll be connecting to and the name of the cluster we’re interested in.

$vCenter = "LabvCenter.domain.com"
$Cluster = "LabCluster"

2. This is where we define the output location for our datastores and difference file. I chose to drop it into a folder named for the cluster, but that can be removed.

$filePath = "C:\test\" + $Cluster + "\"

3. This is where we connect to vCenter and then immediately wait 15 seconds which can fix issues of commands running before security warnings are displayed

Connect-VIserver $vCenter
Start-Sleep -s 15

4. This will gather all the datastores in the cluster and exclude any datastore that has a name containing “*-local”. The wildcard is important because the local datastores contain “servername + -local” and if the wildcard wasn’t there all of the local datastores would be included because no datastore is named exactly “-local”

$Datastores = Get-Cluster -Name $Cluster | Get-Datastore | Where {$_.Name -notlike "*-local"}

5. I prefer the format of 2 digit month, 2 digit day, 2 digit year. This will get the current date of the system running this script, then convert it to this format of 051415 for example.

$today = (Get-Date).ToString("MMddyy")
$yesterday = (Get-Date).AddDays(-1).ToString("MMddyy")
$2DaysAgo = (Get-Date).AddDays(-2).ToString("MMddyy")

6. This will set the file name and location for the output from 2 days ago. If that file exists, it will be removed. Rather than have an output from every day saved until I manually remove it, this process seemed better. I chose to delete the file from 2 days ago as opposed to deleting yesterday’s file after we run the comparison in case we see a huge change in the difference file we can manually compare the 2 files to try to find the error.

$2DayOldFile = $filepath + $Cluster + $2DaysAgo + ".txt"
If (Test-Path $2DayOldFile){Remove-Item $2DayOldFile}

7. This will set the file path and name to the file path defined at the top, plus the cluster name, plus the date and add .txt to the end.

$CurrentFile = $filePath + $Cluster + $today + ".txt"
$YesterdaysFile = $filePath + $Cluster + $yesterday + ".txt"

8. Here we are exporting all the datastores from Step 4 by name and outputting to the file name/path defined in Step 7.

$Datastores | Select Name | Out-File $CurrentFile

9. This is where we set the name and path for the difference file that will track the datastore add/remove.

$DifferenceFile = $filePath + "Datastore-Changes" + ".txt"

10. This will read the content from today’s content and yesterday’s content.

$YesterdaysContent = Get-Content $YesterdaysFile
$CurrentContent = Get-Content $CurrentFile

11. Here we are comparing the content we just read in step 10.

$Compare = Compare-Object $YesterdaysContent $CurrentContent

12. The standard way “Compare-Object” outputs this data shows difference with a side indicator of <= or => depending on where the difference exists. Rather than remember which file was read first to determine whether a datastore was added or removed, we change the column names. If a datastore existed yesterday, but is missing today it is labeled as “Removed”. If a datastore didn’t exist yesterday, but does today it is labeled as “Added”.

$compare | foreach {
if ($_.sideindicator -eq '<=')
{$_.sideindicator = "Removed"}

if ($_.sideindicator -eq '=>')
{$_.sideindicator = "Added"}
}

13. This will take the results from step 11 with formatting of step 12 then change the column names. The list of objects compared is normally named “InputObject” and then “Added or Removed” is normally “SideIndicator”. Maybe that’s fine, but I prefer something a little easier to read. I’ve renamed “InputObject” to “Datastore” but also I add the current date and we change “SideIndicator” to “Added or Removed”. Once that is done, we output that file to the path and name defined in Step 9. The reason why we include the current date in the “Datastore” column is because we are using “-Append” with the “Out-File” command. This will add a dated entry of changes that occurred to the bottom of the existing (or new) output file. This means we aren’t overwriting the same file every day, we are just adding to it. In case you forget to check this file for a few days you won’t lose that data.

$Compare |
select @{l='Datastore' + ' - ' + (Get-Date);e={$_.InputObject}},@{l='Added or Removed';e={$_.SideIndicator}} |
Out-File -Append $DifferenceFile

Now that we know what this thing does, let’s see it in action. I have run the output over 3 days and this is how the output file is displayed. We can see that on 05-14-15 we added Lab-Datastore-10 which didn’t exist on 05-13-15. Then on 05-15-15 we removed Lab-Datastore-03 and we added -11 and -12.
image

When running the script I commented out the removal of the 2 day old file so we could compare manually. Now we have an output file created (Datastore-Changes.txt) that should show the differences.
image

Inside Datastore-Changes.txt we see that on 5/14 the datastore “Lab-Datastore-10” was added and on 5/15 we lost Lab-Datastore-03, but added 11 and 12.

image

We can delete this file at any time and the next time this script runs we’ll create a brand new file. This means there is no dependency on this file already existing in order for the script to run and doesn’t require us to keep a long list of all the datastore add/removes for all eternity. Now you’ll just need to save the script schedule it to run using Windows Task Scheduler.

Below is the full scripts with comments.

#Define the vCenter Server and Cluster
$vCenter = "LabvCenter.domain.com"
$Cluster = "LabCluster"

#Set the path location for the output files
$filePath = "C:\test\" + $Cluster + "\"

#Connect to the vCenter Server and sleep for 15 seconds (necessary for security warnings)
Connect-VIserver $vCenter
Start-Sleep -s 15

#Get a list of all the datastores
$Datastores = Get-Cluster -Name $Cluster | Get-Datastore | Where {$_.Name -notlike "*-local"}

#Get the current date in the correct format
$today = (Get-Date).ToString("MMddyy")
$yesterday = (Get-Date).AddDays(-1).ToString("MMddyy")
$2DaysAgo = (Get-Date).AddDays(-2).ToString("MMddyy")

#Delete the output from 2 days ago (Remove this section if you want to keep the history)
$2DayOldFile = $filepath + $Cluster + $2DaysAgo + ".txt"
If (Test-Path $2DayOldFile){Remove-Item $2DayOldFile}

#Set the filename to include today's date
$CurrentFile = $filePath + $Cluster + $today + ".txt"
$YesterdaysFile = $filePath + $Cluster + $yesterday + ".txt"

#Export those datastores to a TXT file
$Datastores | Select Name | Out-File $CurrentFile

#Set file name & path for difference file
$DifferenceFile = $filePath + "Datastore-Changes" + ".txt"

#Get the content for yesterday and today's files
$YesterdaysContent = Get-Content $YesterdaysFile
$CurrentContent = Get-Content $CurrentFile

#Compare yesterday's and today's files
$Compare = Compare-Object $YesterdaysContent $CurrentContent

#Change the source/target column to "Removed" and "Added"
$compare | foreach { 
      if ($_.sideindicator -eq '')
        {$_.sideindicator = "Added"}
     }

#Change the column name output to "Datastore + Date" and "Added or Removed" then output to file
 $Compare | 
   select @{l='Datastore' + ' - ' + (Get-Date);e={$_.InputObject}},@{l='Added or Removed';e={$_.SideIndicator}} |
   Out-File -Append $DifferenceFile

Create New NFS Project on Tegile

The basis of a Project on the Tegile array is applying permissions and policies to a single volume or group of volumes. This means that changes made at the Project level can propagate to the volumes that live inside that project. If new IP addresses need to be added for read/write and root access for all the volumes, that can be handled at the Project-level instead of having to modify each export. However, you still have the ability to make changes at the individual volume level if that’s required.

In this setup, I’ll create a new project to host my Windows workloads in VMware. I’ll create a volume for Windows 2012 Operating System files and allow all the host on my NFS network read/write and root access to this volume.

1. Login to the web interface of the Tegile array
tegileproject020415-step1
2. Click on “Data”
tegileproject020415-step2
3. Click on the Pool that will host this new project
tegileproject020415-step3
4. In the “Project” window, click “Add Project”
tegileproject020415-step4
5. Enter the name of the Project , choose the Purpose, and select “NFS” for access type. Click “Next”
tegileproject020415-step5
6. Enter the Share Name, enter the number of mount points (more can be added later), and enter any Share limits or reservations. Click “Next”
tegileproject020415-step6
7. Set “NFS Sharing” to “on”. Set “Access Mode” to “Read-Write”, set “Access Type” to “IP” and enter the individual IP addresses or the subnet that will have access to this share. Check the box for “Root Access” then click “Add”. Repeat for each IP/Subnet then click “Next”
tegileproject020415-step7
8. Set your snapshot policy (if required). This can be configured at a later time as well. Click “Next”
tegileproject020415-step8
9. Review your settings and click “Finish”
tegileproject020415-step9
10. Click on the newly created Project and then you will see the volume share name and the mountpoint
tegileproject020415-step10

At this point, we just need to mount this new Volume on our ESXi hosts which can be done manually through the vSphere client on each host or we can do it through PowerCLI which will do it a lot faster.

In the example above, the name of our Volume is “2012_OS”, the path is our share mountpoint (/export/PDX_Windows/2012_OS) and the IP of the Tegile is 192.168.1.15. You’ll need to define the hostname as it appears in vCenter. To mount to a single host we can use the following PowerCLI command:

New-Datastore -NFS -VMHost "Hostname" -Name "2012_OS" -Path /export/PDX_Windows/2012_OS -NfsHost 192.168.1.15

To mount to an entire cluster, we can use this command after defining the name of the cluster as it appears in vCenter:

Get-Cluster "ClusterName" | Get-VMHost | New-Datastore -NFS -Name "2012_OS" -Path /export/PDX_Windows/2012_OS -NfsHost 192.168.1.15

Change IP of vCSA

While changing the IP address of my vCenter Server is not something I’ve ever had to do before that changed this week. In my quest to separate networks into more logical groupings instead of everything living on the same subnet I had to change the IP address of my vCenter Server Appliance to place it on a new network along with the hosts it was managing. There is apparently a right way and a wrong way to do this.

I logged into the vCSA web interface (vCenterIP:5480), clicked on the “Network” tab and then click on “Address” and assumed this would be the correct place. So I changed the IP address and clicked “Save Settings” then rebooted the appliance.

changeip012315-step1

Yeah…that wasn’t right. As I watched the appliance boot from the console I saw a lot of errors being thrown trying to access services running on the old address and failing. Then I decided to shut down (not reboot) the vCSA and try a different method. This is a pretty simple process, but in case you’re looking for the right way of doing it, this is what worked for me.

Once the appliance is powered off, right click and choose “Edit Settings”
changeip012315-step2

Click the “Options” tab then choose “Properties” under “vApp Options”
changeip012315-step3

Enter the new IP address, gateway, and any other information that is changing. If you’re moving it to a new portgroup, update that now as well and click “OK”
changeip012315-step4

Once the changes have been made, power on the appliance and you should see the new addresses being referenced during start up.
changeip012315-step5

And now that start up is complete, we see the new IPs listed for managing the appliance and you should be able to connect on the new IP.
changeip012315-step6

Like I said, this is a very simple process. Once the vCSA was running, my hosts were notified of the change and were still in their cluster. Nothing bad happened and the lab continued to function as expected.

Add NFS Datastore to Cluster via PowerCLI

I have been digging into more and more PowerCLI the last month or so trying to explore faster ways to accomplish common tasks. Using the NetApp VSC plugin inside vCenter I can provision a brand new NFS datastore to an entire cluster in just a few clicks, but there is no built in way to do this for mounting an existing datastore. The below script is just a simple way to mount an NFS datastore to a named cluster.

$ClusterName = "ProdCluster"
$DatastoreName = "VM_Win2003_NA5"
$DatastorePath = "/vol/VM_Win2003_NA5"
$NfsHost = "192.168.1.5"
get-cluster $ClusterName | get-vmhost | New-Datastore -NFS -Name $DatastoreName -Path $DatastorePath -NfsHost $NfsHost

Or you can replace each variable with the actual value in the script when mounting multiple datastores in the same script.

get-cluster "ProdCluster" | get-vmhost | New-Datastore -NFS -Name "VM_Win2003_NA5" -Path "/vol/VM_Win2003_NA5" -NfsHost 192.168.1.5

The next step here will be running this script from vCO and passing the variables directly from vCO. Maybe one day I’ll have the time to figure out just how to do that…

Invalid Virtual Machine Configuration

When a Snapshot of a VM is created and one of the disks is removed prior to removal of the Snapshot, the error “Invalid Virtual Machine Configuration” will appear when attempting to delete that snapshot. This will also prevent any additional snapshots from being created.

In our situation, a snapshot was taken using NetApp Virtual Storage Console plugin during a schedule backup job. At the time of snapshot removal an Oracle load test was being performed on the same storage system. This caused excessive latency and prevented the snapshot from being removed. Follow the steps below to fix this issue.

1. Locate the Virtual machine in vCenter that is throwing this error and select it
2. Click on the “Summary” tab for the VM
invalidVM121214-step2
3. Under “Storage, Right-click on the OS data drive and click “Browse datastore”
invalidVM121214-step3
4. Locate the Folder for this Virtual Machine and open it
5. Locate the file <vmname>.vmsd
invalidVM121214-step5
6. Right click on the .VMSD file and choose “rename”
invalidVM121214-step6
7. Change the name to <vmname>.vmsd.old
invalidVM121214-step7
8. Right click on the Virtual Machine, hover over “Snapshot” then choose “Take Snapshot”
invalidVM121214-step8
9. Enter a name and ensure both boxes are unchecked and click “OK”
invalidVM121214-step9
10. The VM snapshot might fail with a message “A general system error occurred”, this is normal.
11. Right click on the Virtual Machine, hover over “Snapshot”, then choose “Snapshot Manager”
invalidVM121214-step11
12. The previous Snapshot that was there will be gone, but the recent snapshot will remain (this is normal). Click the Snapshot and click “Delete All” and “Yes” to confirm delete
invalidVM121214-step12
13. Try taking a new snapshot and ensure it works
14. As a matter of clean, ensure that you delete the <vmname>.vmsd.old file once you’re finished. No need to leave stale files laying around