Veeam v9 – New Feature Annoucements

While the need for backups hasn’t changed, how you use these backups has. Not only that, the speed at which we can recover our data is changing as well. As the cost of downtime continues to grow, having to restore an entire server just to recover one file or a small number of files just won’t cut it. Your backup needs to be backup quickly and restore even faster.

The improvements in Veeam v9 are doing just that. Veeam has been introducing faster and faster ways to backup and restore (and limit the impact on production virtual machines during backups as well) for years and v9 is no exception. There are a few new options I want to touch on that are pain points I’ve experienced in my environments.

1. Backups from Snapmirror/Snapvault destinations.
As a former NetApp admin, I love the idea of minimizing the effect of backups on my virtual machines. By enabling backup from snapmirror destinations, you can get your VMs offsite using built in software on your NetApp array, and then create off-SAN backups that aren’t limited by your snapmirror rentention schedule due to space constraints.

2. Direct NFS Backup Mode
Direct SAN access has been in Veeam Backup & Replication forever, but backing up VMs on your NFS datastores was a different story. A proxied connection was required through an ESXi host to backup these VMs. In v9, a brand new NFS client was written by the engineers at Veeam to connect directly to your NFS volumes and backup VMs without additional host impact, latency, or speed constraints.

3. Per VM-backup File chain
As the size of your backup job grows, the managing of that file gets to be painful. As your backup repository begins to fill up you’re left having to migrate the entire backup file to a new repository. By creating a Per-VM backup file chain, one job can be created for all of your virtual machines, but each VM has its own file chain. This feature is especially useful with the next feature I’ll talk about.

4. Scale-out Backup Repository
Backup repository management has always been one of the largest pain points when managing Veeam backup jobs. I remember my first Veeam setup I was limited to 2TB LUNs on my backup server and I had to create 8 of them to store my backups. As backup jobs couldn’t span repositories, this meant I was creating individual jobs tied to repositories and then rebalancing as repositories began to fill. The Scale-out backup repository feature allows a virtual backup repository to be create on top of your current physical repositories. Now fewer jobs need to be created and you’re able to take advantage of all the space in each repository. Thanks to Luca Dell’Oca for clarifying that maintenance mode and evacuation are also supported. This mean if a repository needs to be taken down (due to SAN maintenance for example) it can be marked as maintenance mode and excluded from the repository during maintenance operations.

For me, these are the 3 big features I’m happy to see in Veeam v9. There are additional features such as explorers for Oracle, Active Directory (support for AD-integrated DNS and GPO restoration!), SQL Server and SharePoint. The entire list of new features can be found at the link below.

Click here for all the feature announcements.

Add NFS Datastore to Cluster via PowerCLI

I have been digging into more and more PowerCLI the last month or so trying to explore faster ways to accomplish common tasks. Using the NetApp VSC plugin inside vCenter I can provision a brand new NFS datastore to an entire cluster in just a few clicks, but there is no built in way to do this for mounting an existing datastore. The below script is just a simple way to mount an NFS datastore to a named cluster.

$ClusterName = "ProdCluster"
$DatastoreName = "VM_Win2003_NA5"
$DatastorePath = "/vol/VM_Win2003_NA5"
$NfsHost = "192.168.1.5"
get-cluster $ClusterName | get-vmhost | New-Datastore -NFS -Name $DatastoreName -Path $DatastorePath -NfsHost $NfsHost

Or you can replace each variable with the actual value in the script when mounting multiple datastores in the same script.

get-cluster "ProdCluster" | get-vmhost | New-Datastore -NFS -Name "VM_Win2003_NA5" -Path "/vol/VM_Win2003_NA5" -NfsHost 192.168.1.5

The next step here will be running this script from vCO and passing the variables directly from vCO. Maybe one day I’ll have the time to figure out just how to do that…

Invalid Virtual Machine Configuration

When a Snapshot of a VM is created and one of the disks is removed prior to removal of the Snapshot, the error “Invalid Virtual Machine Configuration” will appear when attempting to delete that snapshot. This will also prevent any additional snapshots from being created.

In our situation, a snapshot was taken using NetApp Virtual Storage Console plugin during a schedule backup job. At the time of snapshot removal an Oracle load test was being performed on the same storage system. This caused excessive latency and prevented the snapshot from being removed. Follow the steps below to fix this issue.

1. Locate the Virtual machine in vCenter that is throwing this error and select it
2. Click on the “Summary” tab for the VM
invalidVM121214-step2
3. Under “Storage, Right-click on the OS data drive and click “Browse datastore”
invalidVM121214-step3
4. Locate the Folder for this Virtual Machine and open it
5. Locate the file <vmname>.vmsd
invalidVM121214-step5
6. Right click on the .VMSD file and choose “rename”
invalidVM121214-step6
7. Change the name to <vmname>.vmsd.old
invalidVM121214-step7
8. Right click on the Virtual Machine, hover over “Snapshot” then choose “Take Snapshot”
invalidVM121214-step8
9. Enter a name and ensure both boxes are unchecked and click “OK”
invalidVM121214-step9
10. The VM snapshot might fail with a message “A general system error occurred”, this is normal.
11. Right click on the Virtual Machine, hover over “Snapshot”, then choose “Snapshot Manager”
invalidVM121214-step11
12. The previous Snapshot that was there will be gone, but the recent snapshot will remain (this is normal). Click the Snapshot and click “Delete All” and “Yes” to confirm delete
invalidVM121214-step12
13. Try taking a new snapshot and ensure it works
14. As a matter of clean, ensure that you delete the <vmname>.vmsd.old file once you’re finished. No need to leave stale files laying around

Restore Files & AD Objects From NetApp & Veeam v8

With the release of Veeam Backup & Replication v8 we can restore directly from NetApp Snapshots. Whether it’s an entire VM, individual files, or just some objects in Active Directory, you can do it all from the Veeam console. For a guide on installing and configuring Veeam v8 with NetApp storage, click here

We’ll be testing the restore of individual files and some Active Directory objects for this blog post. In this scenario we have a couple Domain Controllers (2008 R2) and a couple of member servers with some files that we’ll delete. We also have an OU with a couple users, a member server, and a group.

Each of these VMs sit on either of these two volumes, Win_2008 and Win_2012. If you click on “Storage Infrastructure” in the Veeam Backup and Replication console, then expand your NetApp storage you’ll see a list of all the volumes available and their snapshots.
veeamrest120114-part1

1. I’ve taken a snapshot in NetApp System Manager of these volumes. To list these snaps, refresh the volume by right-clicking on the volume and choosing “Rescan volume” or right click on the storage array and choose “Rescan Storage” (Since we have 2 volumes to refresh, we’ll rescan storage.
veeamrest120114-step1
2. A new window will popup showing the progress
veeamrest120114-step2
3. Once completed, we now see the new snapshot I created called “Pre-delete”
veeamrest120114-step3
4.I’m going to delete a file from the server “Lab2008” (on the Win_2008 datastore) and “Lab2012” (on the Win_2012 datastore) that are sitting on my desktop.
veeamrest120114-step4a
veeamrest120114-step4b
5. And let’s also delete the OU “Delete Test” which contains a couple test users, a group they are apart of and the VM “Lab2008”
veeamrest120114-step5
6. Now that those files and OU\objects have been delete, let’s go back to the Veeam console and see what we can recover. We’ll start with the files for the “Lab2012” VM.
7. Expanding “Win_2012” datastore in “Storage Infrastructure” view, click on the name of the snapshot I created earlier and we see the “Lab2012” VM
veeamrest120114-step7
8. We right-click on “Lab2012”, hover over “Restore guest files” and then choose “Microsoft Windows”
veeamrest120114-step8
9. Under the “File Level Restore” screen, click “Customize” in the bottom right corner
veeamrest120114-step9
10. As long as you’re restoring to a vCenter/Host that’s already been added to Veeam, choose the host, resource pool (if any) and folder. Click “OK” then click “Next”
veeamrest120114-step10
11. Enter a reason for the restore and click “Next”
veeamrest120114-step11
12. Click “Finish”
veeamrest120114-step12
13. The restore session will open and mount the snapshot/VM to the chosen host
veeamrest120114-step13
14. In vCenter, we see these 2 tasks of creating a datastore and registering the virtual machine.
veeamrest120114-step14
15. On the host, we see a new powered off VM with the name of “Lab2012” followed by a GUID.
veeamrest120114-step15
16. Back at the Veeam console, the Backup Browser window appears and we can browse to the location of the deleted file
veeamrest120114-step16
17. From here, we can copy the file to our local machine or restore it directly to the Virtual Machine. Right click on the file and choose “Restore” then “Overwrite”
veeamrest120114-step17
18. We’ll pick “Use the following account” and choose my Lab Domain credentials and click “OK”
veeamrest120114-step18
19. The restore process will start and you’ll see this output if you click “Show Details”
veeamrest120114-step19
20. Logging back in to “Lab2012” we can see the file has been restored
veeamrest120114-step20
21. Close the “Restoring files” window in the Veeam console and the “Backup Browser” window. After they’re closed, the VM will be unregistered on the host and the datastore will be unmounted.
22. I’m doing a restore from “Lab2008” but this time I will just copy the file to my local computer instead of restoring to the guest VM. After browsing the datastore snapshots and choosing “Restore Guest Files”, we’ll browse the directory structure, locate the file, right-click and choose “Copy To”
veeamrest120114-step22
23. A window will pop up to choose the folder location on your machine and whether to preserve permissions and ownership. Then click “OK”
veeamrest120114-step23
24. Now in the root of the C: drive we have the “Lab2008-txt” file
veeamrest120114-step24
25. Let’s look at the “Lab2008” VM now. It was in that OU we deleted and after rebooting it and trying to login we receive the message “The security database on the server does not have a computer account for this workstation trust relationship”. We can fix that.
veeamrest120114-step25
26. Back in the Veeam console and the “Pre-delete” snapshot for the “Win_2008” datastore, we’ll locate the “Lab-DC01” VM. Right click on the VM, hover over “Restore application items” and then click “Microsoft Active Directory objects”
veeamrest120114-step26
27. Our host settings are saved from the last restore we did, so click “Next”
veeamrest120114-step27
28. Enter a restore reason and click “Next”
veeamrest120114-step28
29. Review the summary and click “Finish”
veeamrest120114-step29
30. The Veeam Explorer for Microsoft Active Directory window will appear
veeamrest120114-step30
31. Then the VM will be mounted in vCenter
veeamrest120114-step31
32. Once the Veeam Explorer window for AD opens, you’ll be able to browse your Domain object. We’ll expand the “LabOU” object where we see “Delete Test” with the same 2 test users, “Lab2008” server and the group those users belong to.
veeamrest120114-step32
33. Right click the “Delete Test” OU and choose “Restore container to LabDC.local”
veeamrest120114-step33
34. Enter the credentials for the account with access to add objects to the domain and click “OK”
veeamrest120114-step34
35. You’ll see the progress of the restore and then the summary of how many objects were restored
veeamrest120114-step35

(In order for this to work your Veeam server will need network access to the live domain controller)

36. If we refresh the screen for Active Directory Users and Computers on “Lab-DC01” we’ll see the OU is back with all of it’s objects
veeamrest120114-step36
37. In the properties for the users, we can see that group membership was retained. The group “Email Group” is located in another OU and that membership was restored as well
veeamrest120114-step37
38. And now when we try to login to “Lab2008” with domain credentials it works with no issues.

 

How fast can this restore happen? From the time I opened the Veeam console until the time the OU reported as being restored took 3 minutes and 34 seconds. In an emergency where someone accidentally deletes an entire OU, a user account, a server, or anything else, they can all be restored in under 5 minutes time without the need to reset any passwords and everything will work without anyone ever noticing. Veeam is awesome and just keeps getting better and better.

Veeam v8 Install With NetApp Config

Veeam has released v8 of it’s Backup and Replication software. As a long time Veeam user this is a release I have been waiting for. Previously, Veeam had released support for storage snapshots on HP storage arrays, but with my environments being primarily NetApp over the last few years I wasn’t able to take advantage. Now in v8, we can restore and backup directly from snapshot. This speeds up the process and limits the impact on the Virtual Machines in the environment.

This guide walks you through a brand new installation of Veeam Backup & Replication v8 on Server 2012 and how to add your NetApp storage array as an object to browse existing snapshots. This is a high-level guide and in the future I’ll do a more in-depth backup/restore from Storage. For my guide on installing Veeam v7 with Windows 2012 R2 Data Deduplication, click here.

If you’re not interested in a custom SQL Express installation as well, pick up the guide at step 15. Steps 1-15 show how to install SQL Express to the secondary drive to prevent growing databases from affecting the main OS partition.

Prerequisites:

1. Dedicated server for installing Veeam
2. License file for Veeam (copied out to the server)
3. Latest version of Veeam v8 downloaded and mounted on the server (the installer is in an .ISO)
4. A service account for running the Veeam services (Optional, but my preferred method)
5. Username/password with admin rights to vCenter
6. Username/password for NetApp array (for this post I’ll be using the ‘root’ account)

Steps:

1. Right click the DVD drive and click “Open”
veeamv8111714-step1
2. Navigate to Redistr -> x64. Locate SQLEXPRx64.exe, right click and choose “Run as administrator”
veeamv8111714-step2
3. Click “Yes” to run the installer if prompted
veeamv8111714-step3
4. Under the “Installation” section, click “New SQL Server stand-alone installation”
veeamv8111714-step4
5. Click the check box for “I accept the license terms” and decide if you want to send feature usage data to Microsoft then click “Next”
veeamv8111714-step5
6. Ensure the check box for “Include SQL Server product updates” is checked and click “Next”
veeamv8111714-step6
7. Updates and setup files will install…
veeamv8111714-step7
8. Choose the features to install (Database Engine Services is the only thing required). Choose the install directory (I always choose the secondary drive of the machine and click “Next”
veeamv8111714-step8
9. Choose a name for the instance or leave as default (SQLExpress), choose the instance root directory (secondary drive again) and click “Next”
veeamv8111714-step9
10. Enter a service account for running the SQL DB engine (or leave it as local system) and click “Next”
veeamv8111714-step10
11. Choose “Mixed mode” for the authentication type then enter a password for the “sa” account (Immediately save this password somewhere). Choose the groups/users that will be SQL Server administrators
veeamv8111714-step11a

a. Be default, only users/groups added here will have access to the Veeam console. If you don’t want to grant permissions to the SQL instance, you can grant access to these users/groups for the Veeam database after it has been created

12. Click on the “Data Directories” tab and ensure all the directories are pointing to the secondary drive and click “Next”
veeamv8111714-step12
13. Choose whether to send error reports and click “Next” and the installation will begin
veeamv8111714-step13
14. Once the installation completes, click “Close”
veeamv8111714-step14
15. Close the “SQL Server Installation Center” window. Navigate back to the root of the DVD drive. Right click on “Setup.exe” and choose “Run as administrator”
veeamv8111714-step15
16. Click “Yes” to run the installer if prompted
veeamv8111714-step16
17. Click “install” for “Veeam Backup & Replication”
veeamv8111714-step17
18. Click “Next”
veeamv8111714-step18
19. Read and accept the license terms and click “Next”
veeamv8111714-step19
20. Click “Browse” and locate your license file then click “Next”
veeamv8111714-step20
21. Choose the features to install and the install directory then click “Next”

a. To install to a different location (like a secondary drive), the folders need to be created ahead of time
veeamv8111714-step21

22. If any features are missing, click “Install”
veeamv8111714-step22
23. Once the system configuration check passes, click “Next”
veeamv8111714-step23
24. Review the default configuration and if no changes need to be made, click “Install”
veeamv8111714-step24
25. Once the install completes, click “Finish”
veeamv8111714-step25
26. Close the setup window and restart the server
27. After the server finishes rebooting, login and view the services to ensure the Veeam and SQL services that are “Automatic” have started
veeamv8111714-step27
28. Open “Veeam Backup & Replication”
veeamv8111714-step28
29. Click “Managed servers” on the left side and then click “VMware vSphere”
veeamv8111714-step29
30. Enter the name or IP of the vCenter Server and click “Next”
veeamv8111714-step30
31. Click the “Add” button and then enter the username/password of an account with permissions on the vCenter server. Click “OK” then click “Next”
veeamv8111714-step31
32. Click “Finish”
veeamv8111714-step32
33. To add your NetApp storage systems to Veeam, click on “Storage Infrastructure” and then click the “Add Storage” button
veeamv8111714-step33
34. Click “NetApp Data ONTAP”
veeamv8111714-step34
35. Enter the Name or IP of the storage system and click “Next”
veeamv8111714-step35
36. Click “Add” to add credentials to connect to the NetApp then choose the protocol and port. Click “Next”
veeamv8111714-step36
37. If the name/IP and credentials work, click finish and discovery of VMs and LUNs/Volumes will begin.
veeamv8111714-step37
38. Once storage and VMs have been discovered, click “Close”
veeamv8111714-step38
39. In the “Storage Infrastructure” view, expand “NetApp”, then the storage system. Choose a volume with virtual machines and current volume snapshots. Expand the volume, choose a snapshot and see what VMs are inside.
veeamv8111714-step39

a. From this view you can delete existing snapshots, create new storage snapshots, and rescan the volume for new snapshots. At the VM-level, you can instantly recover the VM from snapshot, restore guest-OS files, and even restore objects from Active Directory, Exchange, SQL or SharePoint.
veeamv8111714-step39a

40. Click on “Backup & Replication” then expand “Backups” and click on “Storage snapshots.” You’ll see a list of all the volumes that have snapshots, what VM’s are in those snapshots, and how many restore points are available.
veeamv8111714-step40

This is the basics of installing Veeam v8 and connecting to your vCenter Server and NetApp Storage. The process is incredibly simple and like every else from Veeam it just works. In the future I intend to add more restore scenarios such as application item recovery and VM recovery from storage snapshots.

Deploy NetApp OnCommand Balance 4.2

OnCommand Balance is a virtual appliance deployed within vCenter that allows you to monitor the health of your VMware environment at the Virtual Machine, vCenter and Storage level. Having a single place that displays end-to-end performance allows you to spend less time troubleshooting performance issues and trying to correlate data and address potential issues in your environment.

I’ve been using OnCommand Balance (formerly OnCommand Insight Balance) for a few years now and it has saved countless hours finding issues in the environment. We’ve had historical data available to look at growth and performance trends, as well as increased demand on individual servers after code releases/updates. Having access to the information within the VMs (such as drive space filling up) also makes this an invaluable tool.

The following documentation will take you through the deployment process of the Virtual Appliance and initial setup. You will go through adding your vCenter hosts, storage controllers, creating saved credentials, connecting to Active Directory for authentication and provision a Windows proxy service for monitoring Windows Servers.

Prerequisites:
1. A user account with appropriate permissions to vCenter for OnCommand Balance to use
2. A domain account with permissions to access all monitored Windows machines (preferrably a Domain Admin account)
3. A separate Windows Server/VM that will be used as the Proxy service to monitor Windows machines
a. Must have latest version of Java 6 installed and User Account Control disabled
4. Username/password for the NetApps that will be monitored

Steps:
1. Download the latest version of OnCommand Balance (4.2) for this writing from the NetApp website
balance101414-step1
2. Connect to the vSphere web interface, click on “vCenter”, “Hosts and Clusters”, expand the Datacenter, and click on the Cluster/Host that will host OnCommand Balance. Right click and choose “Deploy OVF Template”
balance101414-step2
3. Click “Local file” and then “Browse”
balance101414-step3
4. Locate the OnCommand Balance OVA and click “Open” then click “Next”
balance101414-step4
5. Review the details of the OVF then click “Next”
balance101414-step5
6. Accept the EULA then click “Next”
balance101414-step6
7. Give the appliance a name and choose the folder location of the appliance (if any) and click “Next”
balance101414-step7
8. Set the virtual disk format (I prefer Thin since one of the drives is 220GB) and choose the datastore. Click “Next”
balance101414-step8
9. Choose the appropriate network and then click “Next”
balance101414-step9
10. Review the settings then click “Finish”
balance101414-step10
11. After deployment completes, locate the appliance, right click and choose “Power On”
balance101414-step11
12. Open the console of the VM (Right-click and choose “Open Console”) where you’ll see this countdown to install VMware tools prior to configuring the Balance virtual appliance (If you miss your chance to do this at this point, I was unable to install VMware tools at all on the appliance)
balance101414-step12
13. Right-click on the VM, go to “All vCenter Actions”, then “Guest OS” and then click “Install VMware Tools”
balance101414-step13
14. After the VMware tools dialog box is displayed, click “Mount”
balance101414-step14
15. The Balance virtual appliance should recognize VMware tools ISO has been mounted and proceed with the installation
balance101414-step15
16. After VMware tools install completes, press “y” then enter to configure static Network connection for the management interface
balance101414-step16
17. Enter the following information:

a. Host name
b. Host IP address
c. Netmask
d. Gateway
e. Primary DNS address
f. Secondary DNS address
g. Search domains
balance101414-step17g

18. Review the settings and then press “y” and enter if everything is correct
balance101414-step18
19. Default OnCommand Balance console login is netapp/netapp. Login to the console
balance101414-step19
20. After a few minutes (5-10) the web service will be up and running. Connect to the https://IPofAppliance/bp to begin configuration
21. Enter the name of your organization and click “Continue”
balance101414-step21
22. Choose if you want to participate in AutoSupport and click “Submit”
balance101414-step22
23. Enter the time zone, NTP Server address, the address of the primary Balance admin (preferably a distribution group), and the SMTP server address. Click “Continue” (You can choose to change the password at this time)
balance101414-step23
24. Sit around and wait a couple minutes…
balance101414-step24
25. A blank screen may appear during this time, but eventually should take you to the OnCommand Balance login page. Login with the default credentials of admin/password or whatever password was set in step 23.
balance101414-step25
26. Click the link for “Configure you storage arrays & appliances”
balance101414-step26
27. Choose the type of storage (NetApp FAS in this case), enter the management address for one of the nodes, Enter the name of the filer, enter the credentials (root in my case) and enter a nickname of these credentials as they can be modified later on during password changes. Click “Save”
balance101414-step27
28. Even though DNS is configured correctly, I usually receive this error about the other filer of this HA system not being resolvable. Click “Enter IP address instead” and then enter the IP of the other filer and click “Resolve”
balance101414-step28
29. Click the “Refresh” link on the right side of the page a few times until “Discovery Collection” status changes to “OK”
balance101414-step29
30. Click the “Add storage system” button to add additional storage arrays (Including the HA partners). Click on “Dashboard” then choose “Configure your vCenter Server”
balance101414-step30
31. Enter the FQDN/IP Address of the vCenter server. Click “Add new” next to Credentials to add the credentials for the vCenter server
balance101414-step31
32. Enter the username, password, and nickname for these credentials. Click “Next”
balance101414-step32
33. Choose what you want monitored (though I can’t imagine why you’d choose not to monitor everything) and click “Save”
balance101414-step33
34. Click the “refresh” link until “Discovery Collection” status changes to “OK”
balance101414-step34
35. Click “Add vCenter Server” button to add any additional vCenter servers. Otherwise, hover over “Discovery” and choose “Credentials”
balance101414-step35
36. To monitor the OS’s of your VMs and physical servers, you can add those credentials on this page. I’ll add domain admin credentials for monitoring my Windows domain VMs. Click “Add credentials” button
balance101414-step36
37. Choose the login method, login name (domain\username), password, nickname for the credentials, and a description. Click “Save”
balance101414-step37
38. Once added they will appear on this screen
balance101414-step38
39. Hover over “Discovery” and choose “Proxies”
balance101414-step39
40. A proxy is required to monitor the guest OS status of Windows VMs and Physical servers. This proxy runs on a windows server. Once you’ve determined (or built) the appropriate server for the proxy, enter it’s FQDN or IP address and click “Continue”. Much like the picture below says, UAC MUST be disabled. You’ll beat your head against the wall for hours trying to figure out why it fails without that.
balance101414-step40
41. Download and install the latest 32-bit Java 6 runtime on this proxy server. Then navigate to the link listed on that proxy VM to begin the installation
balance101414-step41
42. Once the Balance Proxy Installer screen appears, click “Next”
balance101414-step42
43. Locate the folder path for the 32-bit java install and click “Next”
balance101414-step43
44. Enter an admin account for the service to be run under. Check the box for “Start service immediately after install” and click “Next”
balance101414-step44
45. Select any additional components you might need for other vendors and click “Next”
balance101414-step45
46. Review the information and click “Install”
balance101414-step46
47. Click “Finish”
balance101414-step47
48. Back at the Balance web interface, click “Validate proxy setup” and if successful, click “Continue”
balance101414-step48a
balance101414-step48b
49. Hover over “Discovery” and click on “Servers”
balance101414-step49
50. Click the link on the right side for “Unmonitored Servers”
balance101414-step50
51. Click the link next the vCenter server for “# guests are not being monitored”
balance101414-step51
52. Check the box next to the VMs you wish to monitor, choose your Credentials from the dropdown box in the center and click “Monitor guest(s)”
balance101414-step52
53. Hover over “Admin” and choose “Configuration”
balance101414-step53
54. Click “Email”. In here you can set authentication for your SMTP server, choose the “From” address for Balance emails. Click “Enable alerts” and then check all the boxes for Critical, Warning, and all categories (I prefer as many alerts as I can get). Click “Update”
balance101414-step54
55. Click on “Active Directory” and click the check box for “Enable Active Directory”
balance101414-step55
56. Enter the IP/hostname of your AD server, enter the Distinguished name of the account used to search Active Directory, and enter the password for that account. Click “Test”
balance101414-step56
57. Once successful, enter the Distinguished Name of the of the OU for the user/group that will have access to login. Enter the Distinguished name of the Group that will be able to login. Enter “sAMAccount” for the search attribute. Click “Update”
balance101414-step57
58. Hover over “Admin” and click “Users”
balance101414-step58
59. Click “Add User”
balance101414-step59
60. Change “Authentication” to “Active Directory”. Enter the username and click “Lookup”. If successfully, configured, it should populate the e-mail address. Choose the appropriate user type (Admin or User) and click “Save”
balance101414-step60

You’re all setup and ready to let OnCommand Balance start collecting data in your environment. You start to receive some information within about 30 minutes, but after 3-5 days you start to get a better understanding of what is going on in your environment and have more useful metrics.

Tegile NFS Datastore Management in vCenter

As the primary VMware and storage admin, I try to minimize the number of tools I have to use to accomplish my tasks. When it comes to provisioning and managing volumes for VMware, I prefer to do it all from within the vSphere if possible. The VSC console for my NetApp filers has saved a lot of time over the years, but as we continue to explore our Tegile array we can see what their software has to offer.

My last post was about registering the Tegile plugin with vCenter to have this functionality available in the vSphere client. This post goes into the basic administration of NFS volumes from within the vSphere client.

Prerequisites:
1. Credentials to the Tegile web interface (default is admin/tegile)
2. Registered the Tegile plugin on your vCenter server. Click here for those steps.

Steps:
1. Login to the vSphere thick client then click on “Home” and choose “Tegile Management” under “Solutions and Applications”
tegilenfs092214-step1
2. Proceed through any security warnings and login to the Tegile interface
tegilenfs092214-step2
3. On the left you’ll see a list of all the datastores on the Tegile that have been mounted on the ESXi hosts in this vCenter. Towards the bottom, click on “Add Datastore”
tegilenfs092214-step3
4. Enter the following information and click “Create”

a.Name: Name of the datastore
b. Type: Whether block or file based (SAN or NAS)
c. Protocol: NFS, iSCSI
d. Quota: Check this box to set a max size of the volume
e. ESX/ESXi Server (Version): Check the hosts that this datastore will be provisioned to
f. Pool: The disk pool for this datastore (if multiple are available)
g. Project: The project that this datastore will be associated with
h. Purpose: The type of workload hosted on this datastore (important for block size assignment)
i. Zebi Floating IP Address: The IP each ESXi host will connect to
tegilenfs092214-step4i

5. Once the operation is complete, click “OK”
tegilenfs092214-step5
6. The new datastore has been created and mounted and appears in the list of Zebi datastores
tegilenfs092214-step6
7. Click the “More Details” button for the newly created datastore to see all the details of this volume
tegilenfs092214-step7
8. In order to resize this volume, click the “resize” button
tegilenfs092214-step8

a. Check the box for “New Share Quota” and enter the new size and press “Submit”
tegilenfs092214-step8a

9. This view will refresh and the new size will be reflected
tegilenfs092214-step9
10. I have moved a virtual machine into this datastore to test the snapshot function with quiesce enabled. Click the “Snapshot” button for the datastore
tegilenfs092214-step10
11. Enter the name of the snapshot, change “Quiesce” to “on” and click “Create”
tegilenfs092214-step11
12. You’ll receive a message that snapshot creation has been triggered. Click “OK”
tegilenfs092214-step12

a. A new task will be created to snapshot all VMs that are in that datastore
tegilenfs092214-step12a

13. Once the task to remove the virtual machine snapshot completes, click the “Refresh” button on the snapshot screen to see the new snapshot
tegilenfs092214-step13
14. To delete the snapshot, check the box to the snapshot and press the “Delete” button
tegilenfs092214-step14

a. Click “Yes” to confirm deletion
tegilenfs092214-step14a
b. After this box disappears the snapshot is deleted
tegilenfs092214-step14b

i. *UPDATED 10/9/14* There was a bug in version 2.1.2.4.140802 of the Zebi software that stopped the confirmation box was going away after the snapshot deletion completed. Clicking “No” would allow you to return to the snapshot list without any errors. In version 2.1.2.5.140925 this has been fixed and now the confirmation box disappears after the snapshot deletion completes.

Those are the basic functions you can perform from within the plugin. In a future release I would like to see the ability to create full snapshot schedules from the plugin. Since I am the one who is responsible for VMware and storage in our environment it’s simple for me to create the schedule on the web interface of the Tegile array, but that is not always the case. Another function I would like to see is mounting existing datastores on new hosts without having to go through the “Add Storage” process in vCenter for each host.

I’m confident the functionality will get there and I’ll continue to build my list of feature requests for the Tegile team.

Register vCenter Server on Tegile

After 7 years of NetApp administration and implementation I have started looking for a new storage vendor that can “do it all” like NetApp has been able to do. Protocol support is a big deal in each of the environments I’ve worked in, but performance (IOPs and low-latency) are 2 things my existing NetApps haven’t been able to provide. The idea of adding capacity just to add performance is an antiquated way of thinking and NetApp just hasn’t been able to keep up with the evolving storage market.

I am starting a short series on Tegile setup and administration. Tegile came to us a couple of months ago and has impressed us from the very first conversation and all throughout our sizing and implementation. The box is simple to setup and administer and its performance is crushing our current NetApp.

This guide walks you through connecting the Tegile array to your vCenter server, installing the NFS VAAI Plugin, and setting the Tegile recommended values on the ESXi hosts. Once this is completed, you’ll be able to provision new volumes, resize existing volumes, create VM-aware storage snapshots as well as view storage performance of your VMs all from within the vSphere client.

Prerequisites:
1. Admin credentials to the Tegile and vCenter server
2. Dedicated service account in vCenter (I created an account called “ZebiAdmin”)
3. Root password for the ESXi hosts (required to set recommended values)

 

Steps:
1. Connect to the web interface of the storage array and login with Admin credentials

a. Default username: admin
b. Default password: tegile

vctegile091614-step1
2. Click on “Settings” then choose “App-Aware”
vctegile091614-step2
3. Click “Add vCenter/ESXi Host” towards the bottom
vctegile091614-step3
4. Enter the following information:

a. Host Name/IP address: Host name or IP of the vCenter server
b. Username: User account with admin access to vCenter
c. Password: Password for user account
d. Enable Quiesce: This needs to be checked if quiescing will be used at all (a VMware snapshot is taken during thestorage snapshot process for OS consistency). Can be toggled per snapshot job

vctegile091614-step4d
5. Click “Test” to see if the connection is successful. If it is, the “Save” button will turn solid blue and can be clicked
vctegile091614-step5
6. Click “OK” to confirm enabling of quiesce on VMware
vctegile091614-step6
7. Once saved, click the green “Register” button to add the Tegile plugin to vCenter
vctegile091614-step7
8. Once the registration is successful, click “OK”
vctegile091614-step8
9. Login to the vSphere thick client (not the web client). Click the “Home” button then click on “Tegile Management” under “Solutions and Applications” (Click yes to proceed through any certificate warnings)
vctegile091614-step9
10. Login to the Tegile web interface (Likely the same username and password as in step 1)
vctegile091614-step1
11. In this interface you’ll see a list of Datastores on the Tegile that are mounted on your ESXi hosts as well as real-time stats of your array, datastores, and VMs.
vctegile091614-step11
vctegile091614-step11-2
12. Click on “ESX Settings”
vctegile091614-step12
13. Select all the ESXi hosts and then click the Green Arrow icon to install/upgrade the VAAI NFS plugin on these hosts
vctegile091614-step13
14. After the install completes (may take 2-3 minutes), click the “Configuration” button for each host
vctegile091614-step14
15. Login to the ESXi host (likely “root” credentials)
vctegile091614-step15

a. Click “Yes” to enable SSH on this host if it isn’t already enabled
vctegile091614-step15a

16. NFS.MaxQueueDepth should be set to “32” and the rules for iSCSI and FC can be installed in this location. Click “Save” to enable these changes

17. After the NFS VAAI plugin has been installed and settings saved, reboot the host. Repeat for each host in vCenter.

a. The settings changes are immediate, but the NFS VAAI plugin requires a host reboot

 

The process is simple and straight forward. This same process on the NetApp requires the Virtual Storage Console plugin to be installed on a separate server and configured then registered on the vCenter side with much more configuration. Also, installing the NetApp NFS VAAI plugin on the hosts is done through vCenter Update Manager and has been downloaded separately from the NetApp support site. That being said, the Tegile solution is lacking some of the polish that NetApp provides. I would like to see recommended values of the ESXi hosts set all at once, as opposed to one host at a time. In addition, I’d like the Tegile to change NFS.MaxVolumes default value from 8 to something much higher like the NetApp (256).

Align Virtual Machine Disks with NetApp’s MBR Tools

In most environments I’ve been in, disk alignment is something no one knew about or no one cared about. There are still plenty of Windows Server 2003 VMs out in the wild and I’m sure few people realize the impact these misaligned disks have on their storage arrays. There are a few different ways to fix this issue (which are listed in the link above) as well as using VMware Converter which has the option to optimize the partition layout during conversion. For this post we’ll focus on using NetApp’s MBR Tools and mbralign.

This documentation assumes you have the NetApp Virtual Storage Console (VSC) plugin connected to your vCenter instance. If you have an active NetApp account you can download it directly from the NetApp website.

1. Login to the vSphere client
2. Click the Home button in the top-left corner
step2
3. Click on “NetApp” under Solutions and Applications
step3
4. Click on the “Tools” link on the left side under “Monitoring and Host Configuration”
step4
5. Click the “Download” button for your version of ESX(i) and the location on your system to save it to
step5
6. Enable SSH on the ESXi host that currently hosts the VM to be aligned

a. Under “Configuration” tab for the host, click on “Security Profile” under the “Software” section. Click on “Properties” towards the top right corner. Scroll to “SSH” and click on “Options” then click “Start”

7. Click on “Storage” under “Configuration” for that ESXi host and locate a datastore (an NFS volume for this documentation) to upload the MBR Tools to.

a. Right click and choose browse datastore. Then Click the Upload button and upload the .TGZ
step7a

8. Open your SSH client (Putty) and connect to the ESXi host
9. Type the following commands

a. cd /vmfs/volumes
b. ls
step9b
c. Locate the datastore you uploaded the “mbrtools_esxi.tgz” to (ISO_Templates for this writing) and change directory to that datastore
d. cd ISO_Templates

10. Run the following command (Note the directory will show the ID of the volume, not the name of the volume)

a. tar xvzf mbrtools_esxi.tgz
step10a

11. Run the following command replacing the ID of the datastore with the ID of the datastore you’re using

a. cp -r /vmfs/volumes/26ee88d8-9323fd52/opt/ontap /opt/ontap

12. Change directory to the datastore hosting the VM that needs alignment

a. Cd /vmfs/volumes
b. Ls
c. Cd /vmfs/volumes/volumename
d. Ls (to find the folder name)
e. Cd VM_FolderName
step12e

13. Locate the VMName-flat.vmdk file and run the following command

a. /opt/ontap/mbrscan Test-Alignment-flat.vmdk (You should see “aligned:No” at the end)
step13a

14. Before running the alignment, verify these items FIRST!

a. THERE IS AT LEAST AS MUCH SPACE FREE IN THE DATASTORE AS THE SIZE OF THE DISK
i. If the disk is 50GB, there needs to be 50GB free
b. THERE CAN BE NO SNAPSHOTS OF THE VM IN VCENTER (Delete any snapshots that exist first)
c. THE VM MUST BE POWERED OFF PRIOR TO RUNNING THIS.

15. After all that has been verified, run the following command replacing the VM disk name

a. /opt/ontap/mbralign Test-Alignment-flat.vmdk
b. Press “y” to confirm there are no snapshots

16. Run the mbrscan command from step 13 to verify that disk is aligned
step16
17. Power on the VM and login. You will likely receive a message about installing new devices and system settings changes. Click Yes to reboot the VM.
18. Once you have verified functionality for the VM, remove the mbralign backup files created in the datastore with the following command

a. “rm *-mbralign-backup”
step18a

 

The time for alignment varies based on the size of the disk. This process is fairly straightforward and quick to run. When you have a VM with multiple disks or just a lot of VMs to complete it is often times faster to use VMware Converter.

NetApp VSC 4.2.1 Install on vCenter 5.5

I’m open to change, usually. New technology, new releases, new features, new options; I just like new. Though I do like new there is some “new” I struggle to get behind and that “new” is the vSphere Web Client. Even though it has been around awhile, with vSphere 5.5 there are things you have to use the web client for in order to administer your VMs (assuming they’re Harware Version 10). As a man not quite ready to make the leap to full-blown Web Client administration (and how can you since you still need the thick client to run Update Manager), I decided to install NetApp’s Virtual Storage Console 4.2.1 so I can continue to perform NetApp administration in the vSphere thick client.

As the primary VMware and NetApp engineer, having one place to create, mount, and resize volumes is a time saver. Plus the ability to create volume-level snapshots that quiesce the guest operating systems of your VMs provides a quick, point-in-time recovery time for your infrastructure. The steps below show how to install and configure the VSC 4.2.1 plugin in vSphere 5.5 connecting to an HA-pair of NetApps running ONTAP 8.1.2 in 7-mode.

Prerequisites:

1. A server that will run the VSC service (I usually install it on the Windows Server that hosts the vCenter Service or Update Manager as there are no port conflicts)
2. A domain account with “Administrator” rights to vCenter and local administrator on the Windows Server (this account will run the VSC service)
3. Credentials for the NetApps (I use ‘root’ for this, but a new user can be created on the NetApps with appropriate permissions)

 

Steps:

1. Run VSC-4.2.1-win64.exe as administrator
2. Click “Next”
VSC4052814-step2
3. Read through the “Shared Credentials” notes, click “I Understand” then click “Next”
VSC4052814-step3
4. Check the box for “Backup and Recovery” (if licensed) and click “Next”
VSC4052814-step4
5. Set the installation directory (I prefer to install to a non-OS drive) and click “Next”
VSC4052814-step5
6. Note the URL and click the “Install” button
VSC4052814-step6
7. When the installation completes click “Finish” and you should have a browser pop-up to the URL above
VSC4052814-step7
8. On the browser page, continue through any security warnings
VSC4052814-step8
9. Choose the IP of the local service the plugin will use to communicate with vCenter (This is the IP of the server you are installing VSC to)

  • a. Enter the IP of the vCenter server
  • b. Enter a service account username and password (this should not be your own credentials)
  • c. Click “Register”
    VSC4052814-step9c
  • d. You should see this message below if successful
    VSC4052814-step9d

10. On the server, go to “Start”, “Administrative Tools”, and click on “Services”
VSC4052814-step10
11. Locate the “Virtual Storage Console for VMware vSphere Server” service, right click and choose “Properties”
VSC4052814-step11
12. Click the “Log On” tab then click “This account”

  • a. Enter the account username and password used to connect to vCenter then click “OK”
    VSC4052814-step12a
  • b. Click “OK” for the message about granting Log On As A Service rights
  • c. Click “OK” about not taking affect until the service is restarted

13. Right-click on the service and click “Restart”
VSC4052814-step13
14. Login to the vSphere thick client to the vCenter server

  • a. You should receive a Security Warning pop up, click the box to install the certificate and click “Ignore”
    VSC4052814-step14a

15. Click “Plug-ins” then “Manage Plug-ins” at the top
VSC4052814-step15
16. Locate the “Virtual Storage Console” plugin, right click it and click “Enable” then click “Close”
VSC4052814-step16

17. Click the “Home” button towards the top left then click “NetApp” under “Solutions and Applications”
VSC4052814-step21

  • a. If you receive a Security Alert click “Yes” to proceed

18. Right-click on one of the storage controllers listed at the top and click “Modify Credentials”
VSC4052814-step22
19. Enter the management IP address, username (likely root) and password (try with SSL, but if it doesn’t work try without SSL) and click “OK”
VSC4052814-step23

  • a. Click “OK” for the controller privileges summary
  • b. Repeat for any additional Storage Controllers
  • c. Once completed this is what you should see
    VSC4052814-step23c

20. Click on “Provisioning and Cloning” towards to the bottom left
VSC4052814-step24
21. Click the link for “Storage controllers” and click the “Refresh” link towards the top right
22. Right click on one of the controllers and click “Resources”
VSC4052814-step26
23. Move the NFS/iSCSI network interface(s) to the right column, move any volumes that VMware will manage to the right column and move any aggregates to the right column to be managed as well. Click “Save” (The 10.32.22.x network is for management of the NetApp while the 192.168.10.x network is a private, non-routable network for NFS traffic. All volumes should be mounted on that network.)
VSC4052814-step27

  • a. Repeat for any other controllers

 

*The original documentation I posted showed changing the role in vCenter for the “netappvsc” user to “VSC Administrator” instead of just “Administrator”. Turns out this breaks the Virtual Storage Console. When you attempt to mount or provision datastores, you receive the following error; “HTTP ERROR 403. Problem accessing /kamino/index.html. Reason: Forbidden”
VSC4052814-error

At this point you are ready to manage your NetApp filers from the vSphere thick client. By right-clicking on your cluster in vCenter you can provision volumes to all the hosts in a cluster saving so much time of provisioning a new volume and then mounting it one host at time. Too bad you can mount a volume that’s already been created at the cluster level; it can only be done one host at a time.