Invalid Virtual Machine Configuration

When a Snapshot of a VM is created and one of the disks is removed prior to removal of the Snapshot, the error “Invalid Virtual Machine Configuration” will appear when attempting to delete that snapshot. This will also prevent any additional snapshots from being created.

In our situation, a snapshot was taken using NetApp Virtual Storage Console plugin during a schedule backup job. At the time of snapshot removal an Oracle load test was being performed on the same storage system. This caused excessive latency and prevented the snapshot from being removed. Follow the steps below to fix this issue.

1. Locate the Virtual machine in vCenter that is throwing this error and select it
2. Click on the “Summary” tab for the VM
invalidVM121214-step2
3. Under “Storage, Right-click on the OS data drive and click “Browse datastore”
invalidVM121214-step3
4. Locate the Folder for this Virtual Machine and open it
5. Locate the file <vmname>.vmsd
invalidVM121214-step5
6. Right click on the .VMSD file and choose “rename”
invalidVM121214-step6
7. Change the name to <vmname>.vmsd.old
invalidVM121214-step7
8. Right click on the Virtual Machine, hover over “Snapshot” then choose “Take Snapshot”
invalidVM121214-step8
9. Enter a name and ensure both boxes are unchecked and click “OK”
invalidVM121214-step9
10. The VM snapshot might fail with a message “A general system error occurred”, this is normal.
11. Right click on the Virtual Machine, hover over “Snapshot”, then choose “Snapshot Manager”
invalidVM121214-step11
12. The previous Snapshot that was there will be gone, but the recent snapshot will remain (this is normal). Click the Snapshot and click “Delete All” and “Yes” to confirm delete
invalidVM121214-step12
13. Try taking a new snapshot and ensure it works
14. As a matter of clean, ensure that you delete the <vmname>.vmsd.old file once you’re finished. No need to leave stale files laying around

Tegile Array Replication and Restore

These days most of my replication is handled at the VM-level by software design for virtualization. While that is the case for most of my evironment, I still have a few non-virtualized workloads that run on shared storage that need to be replicated in the event of a disaster at my primary location. This process has never been too complex from my days of working with NetApp and now as I continue exploring the Tegile I’m happy to say that it’s just as easy through the GUI.

Documenting this process for my non-virtual workloads would be a little difficult so I’ve decided to document this process using an NFS datastore containing a few virtual machines. The first half of this guide is setting up the replication relationship and replicating the data. The second-half is the process to actually restore that data and make it usable at your DR site.

 

1. Login to the web interface of the Tegile that is the replication source
2. Click on “Settings” then “App-Aware”
tegiledr111214-step2
3. Click on “Zebi Replication” on the left column
tegiledr111214-step3
4. Under the tab “Replication Target” click the “Add” button (This is adding the DR Tegile as the target array)
tegiledr111214-step4
5. Enter the name or IP of the array (the shared Management IP address) and the username/password (Optionally you can specify a port range for replication which we won’t be doing for this documentation) and click “Add”
tegiledr111214-step5
6. Once it has been successfully added it will appear in the “Replication Target” list
tegiledr111214-step6
7. Login to the web interface of the DR target Tegile, click on “Settings” then “App-Aware”, choose “Zebi Replication” on the left column and then click on “Replication Source” tab. You should see your other array listed here (The IP address will be the “management” IPs of each controller, not the shared management IP for both arrays)
tegiledr111214-step7
8. Back on the Primary Tegile (Replication source) click on “Data”
tegiledr111214-step8
9. Click on the disk pool then then project that will be replicated
tegiledr111214-step9
10. For this documentation I’ve created a Project named “NFS_Replication” with a volume named DR_Windows with 4 VMs inside. Click on the project that will be replicated and click on the “Edit” button
tegiledr111214-step10
11. Click on “Replication” on the left column
tegiledr111214-step11
12. Click the “Add Replication” button
tegiledr111214-step12

a. Select the Target System and click “Next”
tegiledr111214-step12a
b. Select the “Target Pool” and enter a name for the “Replication Project”. Click “Next”
tegiledr111214-step12b
c. Choose what options are required and which volumes will be replicated (This test only has one volume, DR_Windows, but you can include or exclude any volumes that exist in this project. We’ll choose quiesce which will perform a VMware snapshot to put the OS in a consistent state. Click “Next”
tegiledr111214-step12c
d. Choose your schedule (manual or automatic), frequency, and how many additional snapshots (restore points) will be saved on the target array. For this example we’ll do daily replication that happens at 10:49 am and we’ll keep 14 snapshots. Click “Finish”
tegiledr111214-step12d
e. Once it’s all setup, you’ll see your target array, the target pool, and the target project
tegiledr111214-step12e

13. I have 4 VMs in that datastore (DR-Test01-04). Once the time hits, we can see that snapshots are taken, then removed, for each of the VMs in that datastore.
tegiledr111214-step13
14. On the DR target array, we can see we now have snapshots available for this project. (The reason there are 2 is because I initiated a manual replication sync for testing first)
tegiledr111214-step14

a. To manually kick off a replica snapshot, on the source array, find the project, click on “Replication” and then click the “Play” button that says “Replicate”
tegiledr111214-step14a

 

That is how simple it is to setup replication. Now let’s imagine we need to spin up those replicated VMs in this volume. Here is how we do that.

 

1. On the DR target array, click on Data, select the pool, then click on “Replica (1)” to view the replica project
tegilerest111214-step1
2. Click the “Edit” button for the NFS volume
tegilerest111214-step2
3. Click on “Snapshots” and find the snapshot you want to bring live (We’ll choose the latest version). Click the “Clone” button
tegilerest111214-step3

a. Cloning the snapshot will allow us to create a new project and NFS Volume from this snapshot and spin up these VMs in DR. By doing a clone, we’re able to continue to replicate data in the event you are testing replication instead of having an actual DR event.

4. Enter a name for the new Project (DR_NFS_Replication for this writing) and a name for the mount point (/export/DR_NFS_Replication for this writing) and click “Clone”
tegilerest111214-step4
5. If successful, you’ll receive this message about the new project being created. Click “OK”
tegilerest111214-step5
6. Close the window for “Share Configuration” and click on “Local (1)” under “Projects”
tegilerest111214-step6
7. Click on the “DR_NFS_Replication” project then view the Mountpoint of the Share (/export/DR_NFS_Replication/DR_Windows). Note the “c” before the share name which denotes it was a clone from another projects
tegilerest111214-step7
8. Click the “Edit” button for the project and then click on “Sharing”
tegilerest111214-step8
9. This is where you will add the IP addresses or range of IPs that need read/write and root access to the shares in this project. The IP addresses/ranges will carry over from the source array. Our IP range is the same in DR as our lab so we’ll leave this alone.
tegilerest111214-step9
10. Connect to your DR vCenter server or ESXi hosts. Click on the host, then “Configuration”, then “Storage”
tegilerest111214-step10
11. Click “Add Storage” towards the top right
tegilerest111214-step11
12. Choose “Network File System” and click “Next”
tegilerest111214-step12
13. Enter the NFS IP address of the DR Tegile, enter the folder path (/export/DR_NFS_Replication/DR_Windows) and then enter the name of the Datastore (DR_Windows). Click “Next”
tegilerest111214-step13
14. Review the summary info and click “Finish”
tegilerest111214-step14
15. Repeat for each host that needs access to this datastore. Afterwards, right click the datastore and click “Browse Datastore”
tegilerest111214-step15
16. Inside you’ll see the 4 VMs we that were located in here before. Open each folder, right click the VM name.vmx file and choose “Add to inventory”
tegilerest111214-step16
17. Enter the name and location for the VM and click “Next”
tegilerest111214-step17
18. Choose the Cluster or host and click “Next”
tegilerest111214-step18
19. Review the settings and click “Finish.” Repeat for each VM that needs to be added.
tegilerest111214-step19
20. Power on all the VMs and now you can run any validation tests or bring these VMs live in a DR event
tegilerest111214-step20

 

Obviously, the process of mounting the datastore in your DR vCenter Server and re-adding the VMs one by one would be time consuming and tedious. When developing your DR plans, having this process scripted (easy enough in something like PowerCLI) ahead of time on the vCenter side of things would ease that burden. From the standpoint of the Tegile, this process is fairly intuitive and simple to setup. One of the things I love is that by default the data you are bringing live on the DR site is a clone and replication continues running without being affected.

vCenter Orchestrator Install and Config

I have wanted to get started with vCO for awhile now, but I have not had much of use for it. Justifying the time to deploy and learn a new tool when you don’t have a glaring need for it proves tricky, but recently I was able to carve out some time to learn. One of the biggest hurdles was finding step-by-step deployment guide that worked so I decided to document this process.

The following documentation is for installing the vCenter Orchestrator (vCO) Appliance v5.5.1 with an already deployed vCenter 5.5 server (vCSA in my case). The appliance allows you to run vCO without installing it on a dedicated Windows Server.

1. Search for VMware-vCO-Appliance and download the latest version (VMware-vCO-Appliance-5.5.1.0-1617225_OVF10.ova for this writing)
VCO080414-step1
2. Accept the license terms and save the file locally
3. Connect the vSphere client to your vCenter Server then choose File -> Deploy OVF Template
VCO080414-step3
4. Click the “Browse” button, locate the .OVF downloaded previously and click “Open” then click “Next”
VCO080414-step4
5. Review the template details and click “Next”
VCO080414-step5
6. Accept the license agreement and click “Next”
7. Choose a name and location for this appliance and click “Next”
VCO080414-step7
8. Choose a datastore for the appliance and click “Next”
VCO080414-step8
9. Choose the appropriate disk format (I prefer thin provisioned) and click “Next”
VCO080414-step9
10. Choose the appropriate Destination Network (VM Port Group) and click “Next”
VCO080414-step10
11. Enter passwords for both the root user of the appliance and the password for the configuration interface (‘vmware’ is the username)
VCO080414-step11

  • Enter the Hostname, gateway, DNS, IP and subnet mask for the appliance and click “Next”
    VCO080414-step11a

12. Review the details of the configuration and then click “Finish”
VCO080414-step12
13. Once the appliance has been deployed successfully, click “Close”

VCO080414-step13
14. Right click on the appliance and choose “Open Console”
VCO080414-step14
15. Click the Power button to turn on the VM
VCO080414-step15
16. Boot to “VMware vCenter Orchestrator Appliance”
VCO080414-step16
17. Note the URLs for each function
VCO080414-step17
18. Open a web browser and connect to the URL for Orchestrator Configuration (Port 8283)
19. Login with the username “vmware” and the password entered for the vCO configuration during appliance deployment
VCO080414-step19
20. Click on “Network”
VCO080414-step20
21. Change the “IP address” to the IP used to access vCO and click “Apply changes” in the bottom right corner
VCO080414-step21
22. Click the “SSL Trust Manager” tab, enter the IP or hostname of your vCenter server and click “Import”
VCO080414-step22
23. Once the cert information is displayed, click the “import” link
VCO080414-step23
24.Repeat this process again, this time importing the certificate for SSO. Enter the FQDN of the SSO server with port 7444 and click “Import” then “Import” again once the certificate details are displayed
VCO080414-step24new
25. Click on “Authentication” to configure user access

VCO080414-step24
26. For this writing we will use the SSO Authentication method, so change Authentication mode to “SSO Authentication” and click “Advanced settings”
VCO080414-step25
27. Enter the Token and Admin service URLs, the SSO admin username and passwords. Click “Register Orchestrator”

  • Token service URL: https://vCenterIPaddress:7444/ims/STSService
  • Admin service URL: https://vCenterIPaddress:7444/sso-adminserver/sdk
  • Admin user name: administrator@vsphere.local
  • Admin password: Password for admin account
    VCO080414-step26d

28. Once registration completes, choose the vCO Admin – domain and group from the list (These are populated based on your SSO config). Click “Accept Orchestrator Configuration”
VCO080414-step27
29. Click on “Startup Options”
VCO080414-step28
30. Click “Restart the vCO configuration server”
VCO080414-step29
31. Log back in once the server has finished restarting and click “Licenses”
VCO080414-step30
32. Choose “Use vCenter Server license” and enter the host name of the vCenter server, port should be 443, path is /sdk, and for username and password I used the SSO admin. Click “Apply changes” towards the bottom right of the screen
VCO080414-step31
33. Click on “vCenter Server (5.5.1)”
VCO080414-step32
34. Click “New vCenter Server Host” and enter the hostname of the vCenter server, port is 443, path is /sdk, I chose “Session per user” and the username and password for the SSO admin account. Click “Apply changes”
VCO080414-step33
35. Click on “Mail (5.5.1)”
VCO080414-step34
36. Click the check box for “define default values” and enter in the following information and click “Apply changes”

  • SMTP host: The address for your mail server
  • SMTP Port: Usually 25
  • Username and password: If your mail server requires authentication
  • From name: Name that vCO emails will appear from
  • From address: Email address that vCO emails will appear from
    VCO080414-step35e

37. Open a new browser window/tab and navigate to https://vCOIPaddress:8281/vco/client/client.jnlp to access the Java web client for vCO. Login as user that is a member of whatever group was chosen in step 27 as a vCO Admin
VCO080414-step36

  • At first this did not work and kept reporting “No vCO license available” when I attempted to login. After restarting the service and configuration server through the web interface, I ended up restarting the vCO appliance within vCenter and then I was able to login without issue

38. At this point you’re all setup and ready to start creating workflows
VCO080414-step37