Invalid Virtual Machine Configuration

When a Snapshot of a VM is created and one of the disks is removed prior to removal of the Snapshot, the error “Invalid Virtual Machine Configuration” will appear when attempting to delete that snapshot. This will also prevent any additional snapshots from being created.

In our situation, a snapshot was taken using NetApp Virtual Storage Console plugin during a schedule backup job. At the time of snapshot removal an Oracle load test was being performed on the same storage system. This caused excessive latency and prevented the snapshot from being removed. Follow the steps below to fix this issue.

1. Locate the Virtual machine in vCenter that is throwing this error and select it
2. Click on the “Summary” tab for the VM
invalidVM121214-step2
3. Under “Storage, Right-click on the OS data drive and click “Browse datastore”
invalidVM121214-step3
4. Locate the Folder for this Virtual Machine and open it
5. Locate the file <vmname>.vmsd
invalidVM121214-step5
6. Right click on the .VMSD file and choose “rename”
invalidVM121214-step6
7. Change the name to <vmname>.vmsd.old
invalidVM121214-step7
8. Right click on the Virtual Machine, hover over “Snapshot” then choose “Take Snapshot”
invalidVM121214-step8
9. Enter a name and ensure both boxes are unchecked and click “OK”
invalidVM121214-step9
10. The VM snapshot might fail with a message “A general system error occurred”, this is normal.
11. Right click on the Virtual Machine, hover over “Snapshot”, then choose “Snapshot Manager”
invalidVM121214-step11
12. The previous Snapshot that was there will be gone, but the recent snapshot will remain (this is normal). Click the Snapshot and click “Delete All” and “Yes” to confirm delete
invalidVM121214-step12
13. Try taking a new snapshot and ensure it works
14. As a matter of clean, ensure that you delete the <vmname>.vmsd.old file once you’re finished. No need to leave stale files laying around

Tegile NFS Datastore Management in vCenter

As the primary VMware and storage admin, I try to minimize the number of tools I have to use to accomplish my tasks. When it comes to provisioning and managing volumes for VMware, I prefer to do it all from within the vSphere if possible. The VSC console for my NetApp filers has saved a lot of time over the years, but as we continue to explore our Tegile array we can see what their software has to offer.

My last post was about registering the Tegile plugin with vCenter to have this functionality available in the vSphere client. This post goes into the basic administration of NFS volumes from within the vSphere client.

Prerequisites:
1. Credentials to the Tegile web interface (default is admin/tegile)
2. Registered the Tegile plugin on your vCenter server. Click here for those steps.

Steps:
1. Login to the vSphere thick client then click on “Home” and choose “Tegile Management” under “Solutions and Applications”
tegilenfs092214-step1
2. Proceed through any security warnings and login to the Tegile interface
tegilenfs092214-step2
3. On the left you’ll see a list of all the datastores on the Tegile that have been mounted on the ESXi hosts in this vCenter. Towards the bottom, click on “Add Datastore”
tegilenfs092214-step3
4. Enter the following information and click “Create”

a.Name: Name of the datastore
b. Type: Whether block or file based (SAN or NAS)
c. Protocol: NFS, iSCSI
d. Quota: Check this box to set a max size of the volume
e. ESX/ESXi Server (Version): Check the hosts that this datastore will be provisioned to
f. Pool: The disk pool for this datastore (if multiple are available)
g. Project: The project that this datastore will be associated with
h. Purpose: The type of workload hosted on this datastore (important for block size assignment)
i. Zebi Floating IP Address: The IP each ESXi host will connect to
tegilenfs092214-step4i

5. Once the operation is complete, click “OK”
tegilenfs092214-step5
6. The new datastore has been created and mounted and appears in the list of Zebi datastores
tegilenfs092214-step6
7. Click the “More Details” button for the newly created datastore to see all the details of this volume
tegilenfs092214-step7
8. In order to resize this volume, click the “resize” button
tegilenfs092214-step8

a. Check the box for “New Share Quota” and enter the new size and press “Submit”
tegilenfs092214-step8a

9. This view will refresh and the new size will be reflected
tegilenfs092214-step9
10. I have moved a virtual machine into this datastore to test the snapshot function with quiesce enabled. Click the “Snapshot” button for the datastore
tegilenfs092214-step10
11. Enter the name of the snapshot, change “Quiesce” to “on” and click “Create”
tegilenfs092214-step11
12. You’ll receive a message that snapshot creation has been triggered. Click “OK”
tegilenfs092214-step12

a. A new task will be created to snapshot all VMs that are in that datastore
tegilenfs092214-step12a

13. Once the task to remove the virtual machine snapshot completes, click the “Refresh” button on the snapshot screen to see the new snapshot
tegilenfs092214-step13
14. To delete the snapshot, check the box to the snapshot and press the “Delete” button
tegilenfs092214-step14

a. Click “Yes” to confirm deletion
tegilenfs092214-step14a
b. After this box disappears the snapshot is deleted
tegilenfs092214-step14b

i. *UPDATED 10/9/14* There was a bug in version 2.1.2.4.140802 of the Zebi software that stopped the confirmation box was going away after the snapshot deletion completed. Clicking “No” would allow you to return to the snapshot list without any errors. In version 2.1.2.5.140925 this has been fixed and now the confirmation box disappears after the snapshot deletion completes.

Those are the basic functions you can perform from within the plugin. In a future release I would like to see the ability to create full snapshot schedules from the plugin. Since I am the one who is responsible for VMware and storage in our environment it’s simple for me to create the schedule on the web interface of the Tegile array, but that is not always the case. Another function I would like to see is mounting existing datastores on new hosts without having to go through the “Add Storage” process in vCenter for each host.

I’m confident the functionality will get there and I’ll continue to build my list of feature requests for the Tegile team.

Register vCenter Server on Tegile

After 7 years of NetApp administration and implementation I have started looking for a new storage vendor that can “do it all” like NetApp has been able to do. Protocol support is a big deal in each of the environments I’ve worked in, but performance (IOPs and low-latency) are 2 things my existing NetApps haven’t been able to provide. The idea of adding capacity just to add performance is an antiquated way of thinking and NetApp just hasn’t been able to keep up with the evolving storage market.

I am starting a short series on Tegile setup and administration. Tegile came to us a couple of months ago and has impressed us from the very first conversation and all throughout our sizing and implementation. The box is simple to setup and administer and its performance is crushing our current NetApp.

This guide walks you through connecting the Tegile array to your vCenter server, installing the NFS VAAI Plugin, and setting the Tegile recommended values on the ESXi hosts. Once this is completed, you’ll be able to provision new volumes, resize existing volumes, create VM-aware storage snapshots as well as view storage performance of your VMs all from within the vSphere client.

Prerequisites:
1. Admin credentials to the Tegile and vCenter server
2. Dedicated service account in vCenter (I created an account called “ZebiAdmin”)
3. Root password for the ESXi hosts (required to set recommended values)

 

Steps:
1. Connect to the web interface of the storage array and login with Admin credentials

a. Default username: admin
b. Default password: tegile

vctegile091614-step1
2. Click on “Settings” then choose “App-Aware”
vctegile091614-step2
3. Click “Add vCenter/ESXi Host” towards the bottom
vctegile091614-step3
4. Enter the following information:

a. Host Name/IP address: Host name or IP of the vCenter server
b. Username: User account with admin access to vCenter
c. Password: Password for user account
d. Enable Quiesce: This needs to be checked if quiescing will be used at all (a VMware snapshot is taken during thestorage snapshot process for OS consistency). Can be toggled per snapshot job

vctegile091614-step4d
5. Click “Test” to see if the connection is successful. If it is, the “Save” button will turn solid blue and can be clicked
vctegile091614-step5
6. Click “OK” to confirm enabling of quiesce on VMware
vctegile091614-step6
7. Once saved, click the green “Register” button to add the Tegile plugin to vCenter
vctegile091614-step7
8. Once the registration is successful, click “OK”
vctegile091614-step8
9. Login to the vSphere thick client (not the web client). Click the “Home” button then click on “Tegile Management” under “Solutions and Applications” (Click yes to proceed through any certificate warnings)
vctegile091614-step9
10. Login to the Tegile web interface (Likely the same username and password as in step 1)
vctegile091614-step1
11. In this interface you’ll see a list of Datastores on the Tegile that are mounted on your ESXi hosts as well as real-time stats of your array, datastores, and VMs.
vctegile091614-step11
vctegile091614-step11-2
12. Click on “ESX Settings”
vctegile091614-step12
13. Select all the ESXi hosts and then click the Green Arrow icon to install/upgrade the VAAI NFS plugin on these hosts
vctegile091614-step13
14. After the install completes (may take 2-3 minutes), click the “Configuration” button for each host
vctegile091614-step14
15. Login to the ESXi host (likely “root” credentials)
vctegile091614-step15

a. Click “Yes” to enable SSH on this host if it isn’t already enabled
vctegile091614-step15a

16. NFS.MaxQueueDepth should be set to “32” and the rules for iSCSI and FC can be installed in this location. Click “Save” to enable these changes

17. After the NFS VAAI plugin has been installed and settings saved, reboot the host. Repeat for each host in vCenter.

a. The settings changes are immediate, but the NFS VAAI plugin requires a host reboot

 

The process is simple and straight forward. This same process on the NetApp requires the Virtual Storage Console plugin to be installed on a separate server and configured then registered on the vCenter side with much more configuration. Also, installing the NetApp NFS VAAI plugin on the hosts is done through vCenter Update Manager and has been downloaded separately from the NetApp support site. That being said, the Tegile solution is lacking some of the polish that NetApp provides. I would like to see recommended values of the ESXi hosts set all at once, as opposed to one host at a time. In addition, I’d like the Tegile to change NFS.MaxVolumes default value from 8 to something much higher like the NetApp (256).

NetApp VSC 4.2.1 Install on vCenter 5.5

I’m open to change, usually. New technology, new releases, new features, new options; I just like new. Though I do like new there is some “new” I struggle to get behind and that “new” is the vSphere Web Client. Even though it has been around awhile, with vSphere 5.5 there are things you have to use the web client for in order to administer your VMs (assuming they’re Harware Version 10). As a man not quite ready to make the leap to full-blown Web Client administration (and how can you since you still need the thick client to run Update Manager), I decided to install NetApp’s Virtual Storage Console 4.2.1 so I can continue to perform NetApp administration in the vSphere thick client.

As the primary VMware and NetApp engineer, having one place to create, mount, and resize volumes is a time saver. Plus the ability to create volume-level snapshots that quiesce the guest operating systems of your VMs provides a quick, point-in-time recovery time for your infrastructure. The steps below show how to install and configure the VSC 4.2.1 plugin in vSphere 5.5 connecting to an HA-pair of NetApps running ONTAP 8.1.2 in 7-mode.

Prerequisites:

1. A server that will run the VSC service (I usually install it on the Windows Server that hosts the vCenter Service or Update Manager as there are no port conflicts)
2. A domain account with “Administrator” rights to vCenter and local administrator on the Windows Server (this account will run the VSC service)
3. Credentials for the NetApps (I use ‘root’ for this, but a new user can be created on the NetApps with appropriate permissions)

 

Steps:

1. Run VSC-4.2.1-win64.exe as administrator
2. Click “Next”
VSC4052814-step2
3. Read through the “Shared Credentials” notes, click “I Understand” then click “Next”
VSC4052814-step3
4. Check the box for “Backup and Recovery” (if licensed) and click “Next”
VSC4052814-step4
5. Set the installation directory (I prefer to install to a non-OS drive) and click “Next”
VSC4052814-step5
6. Note the URL and click the “Install” button
VSC4052814-step6
7. When the installation completes click “Finish” and you should have a browser pop-up to the URL above
VSC4052814-step7
8. On the browser page, continue through any security warnings
VSC4052814-step8
9. Choose the IP of the local service the plugin will use to communicate with vCenter (This is the IP of the server you are installing VSC to)

  • a. Enter the IP of the vCenter server
  • b. Enter a service account username and password (this should not be your own credentials)
  • c. Click “Register”
    VSC4052814-step9c
  • d. You should see this message below if successful
    VSC4052814-step9d

10. On the server, go to “Start”, “Administrative Tools”, and click on “Services”
VSC4052814-step10
11. Locate the “Virtual Storage Console for VMware vSphere Server” service, right click and choose “Properties”
VSC4052814-step11
12. Click the “Log On” tab then click “This account”

  • a. Enter the account username and password used to connect to vCenter then click “OK”
    VSC4052814-step12a
  • b. Click “OK” for the message about granting Log On As A Service rights
  • c. Click “OK” about not taking affect until the service is restarted

13. Right-click on the service and click “Restart”
VSC4052814-step13
14. Login to the vSphere thick client to the vCenter server

  • a. You should receive a Security Warning pop up, click the box to install the certificate and click “Ignore”
    VSC4052814-step14a

15. Click “Plug-ins” then “Manage Plug-ins” at the top
VSC4052814-step15
16. Locate the “Virtual Storage Console” plugin, right click it and click “Enable” then click “Close”
VSC4052814-step16

17. Click the “Home” button towards the top left then click “NetApp” under “Solutions and Applications”
VSC4052814-step21

  • a. If you receive a Security Alert click “Yes” to proceed

18. Right-click on one of the storage controllers listed at the top and click “Modify Credentials”
VSC4052814-step22
19. Enter the management IP address, username (likely root) and password (try with SSL, but if it doesn’t work try without SSL) and click “OK”
VSC4052814-step23

  • a. Click “OK” for the controller privileges summary
  • b. Repeat for any additional Storage Controllers
  • c. Once completed this is what you should see
    VSC4052814-step23c

20. Click on “Provisioning and Cloning” towards to the bottom left
VSC4052814-step24
21. Click the link for “Storage controllers” and click the “Refresh” link towards the top right
22. Right click on one of the controllers and click “Resources”
VSC4052814-step26
23. Move the NFS/iSCSI network interface(s) to the right column, move any volumes that VMware will manage to the right column and move any aggregates to the right column to be managed as well. Click “Save” (The 10.32.22.x network is for management of the NetApp while the 192.168.10.x network is a private, non-routable network for NFS traffic. All volumes should be mounted on that network.)
VSC4052814-step27

  • a. Repeat for any other controllers

 

*The original documentation I posted showed changing the role in vCenter for the “netappvsc” user to “VSC Administrator” instead of just “Administrator”. Turns out this breaks the Virtual Storage Console. When you attempt to mount or provision datastores, you receive the following error; “HTTP ERROR 403. Problem accessing /kamino/index.html. Reason: Forbidden”
VSC4052814-error

At this point you are ready to manage your NetApp filers from the vSphere thick client. By right-clicking on your cluster in vCenter you can provision volumes to all the hosts in a cluster saving so much time of provisioning a new volume and then mounting it one host at time. Too bad you can mount a volume that’s already been created at the cluster level; it can only be done one host at a time.