vSphere SDRS Design Considerations

If you happen to have your vSphere cluster to be licensed with Enterprise Plus edition, you may be aware of some of the advanced storage management features it includes, such as Storage DRS and Profile-Driven Storage.

These two features work together to let you optimise VM distribution between multiple VMware datastores from capability, capacity and latency perspective, much like DRS does for memory and compute. But they have some interoperability limitations, which I want to discuss in this post.

Datastore Clusters

In simple terms, datastore cluster is a collection of multiple datastores, which can be seen as a single entity from VM provisioning perspective.

VMware poses certain requirements for datastore clustes, but in my opinion the most important one is this:

Datastore clusters must contain similar or interchangeable datastores.

In other words, all of the datastores within a datastore cluster should have the same performance properties. You should not mix datastores provisioned on SSD tier with datastores on SAS and SATA tier and vise versa. The reason why is simple. Datastore clusters are used by SDRS to load-balance VMs between the datastores of a datastore cluster. DRS balances VMs based on datastore capacity and I/O latency only and is not storage capability aware. If you had SSD, SAS and SATA datastores all under the same cluster, SDRS would simply move all VMs to SSD-backed datastores, because it has the lowest latency and leave SAS and SATA empty, which makes little sense.

Design Decision 1:

• If you have several datastores with the same performance characteristics, combine them all in a datastore cluster. Do not mix datastores from different arrays or array storage tiers in one datastore cluster. Datastore clusters is not a storage tiering solution.

Storage DRS

As already mentioned, SDRS is a feature, which when enabled on a datastore cluster level, lets you automatically (or manually) distribute VMs between datastores based on datastore storage utilization and I/O latency basis. VM placement recommendations and datastore maintenance mode are amongst other useful features of SDRS.

Quite often SDRS is perceived as a feature that can work with Profile-Driven Storage to enforce VM Storage Policy compliance. One of the scenarios, that is often brought up is what if there’s a VM with multiple .vmdk disks. Each disk has a certain storage capability. Mistakenly one of the disks has been storage vMotion’ed to a datastore, which does not meet the storage capability requirements. Can SDRS automatically move the disk back to a compliant datastore or notify that VM is not compliant? The answer is – no. SDRS does not take storage capabilities into account and make decisions only based on capacity and latency. This may be implemented in future versions, but is not supported in vSphere 5.

Design Decision 2:

• Use datastore clusters in conjunction with Storage DRS to get the benefit of VM load-balancing and placement recommendations. SDRS is not storage capability aware and cannot enforce VM Storage Policy compliance.

Profile-Driven Storage

So if SDRS and datastore clusters are not capable of supporting  multiple tiers of storage, then what does? Profile-Driven Storage is aimed exactly for that. You can assign user-defined or system-defined storage capabilities to a datastore and then create a VM Storage Policy and assign it to a VM. VM Storage Policy includes the list of required storage capabilities and only those datastores that mach them, will be suggested as a target for the VM that is assigned to that policy.

You can create storage capabilities manually, such as SSD, SAS, SATA. Or more abstract, such as Bronze, Silver and Gold and assign them to corresponding datastores. Or you can leverage VASA, which automatically assigns corresponding storage capabilities. Below is an example of a datastore connected from a Dell Compellent storage array.

You can then use storage capabilities from the VASA provider to create VM Storage Policies and assign them to VMs accordingly.

Design Decision 3:

• If you have more than one datastore storage type, use Profile-Driven Storage to enforce VM placement based on VM storage requirements. VASA can simplify storage capabilities management.

Conclusion

If all of your datastores have the same performance characteristics, such as a number of LUNs auto-tiered on the storage array side, then one SDRS-enabled datastore cluster is a perfect solution for you.

But if your storage design is slightly more complex and you have datastores with different performance characteristics, such as SSD, SAS and SATA, leverage Profile-Driven Storage to control VM placement and enforce compliance. Just make sure to use a separate cluster for each tier of storage and you will get the most benefit out of vSphere Storage Policy-Based Management.

Advertisement

Requirements for Unmounting a VMware Datastore

I have come across issues unmounting VMware datastores myself multiple times. In recent vSphere versions vCenter shows you a warning if some of the requirements are not fulfilled. It is not the case in the older vSphere versions, which makes it harder to identify the issue.

Interestingly, there are some pre-requisites which even vCenter does not prompt you about. I will discuss all of the requirements in this post.

General Requirements

In this category I combine all requirements which vCenter checks against, such as:

Requirement: No virtual machine resides on the datastore.

Action: You have to make sure that the host you are unmounting the datastore from has no virtual machines (running or stopped) registered on this datastore.  If you are unmounting just one datastore from just one host, you can simply vMotion all VMs residing on the datastore from this host to the remaining hosts. If you are unmounting the datastore from all hosts, you’ll have to either Storage vMotion all VMs to the remaining datastores or shutdown the VMs and unregister them from vCenter.

Requirement: The datastore is not part of a Datastore Cluster.

Requirement: The datastore is not managed by storage DRS.

Action: Drag and drop the datastore from the Datastore Cluster in vCenter to move it out of the Datastore Cluster. Second requirement is redundant, because SDRS is enabled on a datastore which is configured withing a Datastore Cluster. By removing a datastore from a Datastore Cluster you atomatically disable storage DRS on it.

Requirement: Storage I/O control is disabled for this datastore.

Action: Go to the datastore properties and uncheck Storage I/O Control option. On a SIOC-enabled datastore vSphere creates a folder named after the block device ID and keeps a file called “slotsfile” in it. Its size will change to 0.00 KB once SIOC is disabled.

Requirement: The datastore is not used for vSphere HA heartbeat.

Action: vSphere HA automatically selects two VMware datastores, creates .vSphere-HA folders and use them to keep HA heartbeats. If you have more than two datastores in your cluster, you can control which datastores are selected. Go to cluster properties > Datastore Heartbeating (under vSphere HA section) and select preferred datastores from the list. This will work if you are unmounting one datastore. If you need to unmount all datastores, you will have to disable HA on the cluster level altogether.

Additional Requirements

Requirements which fall in this category are not checked by vCenter, but are still have to be satisfied. Otherwise vCenter will not let you unmount the datastore.

Requirement: The datastore is not used for swap.

Action: When VM is powered on by default it creates a swap file in the VM directory with .vswp extension. You can change the default behavior and on a per host basis select a dedicated datastore where host will be creating swap files for virtual machines. This setting is enabled in cluster properties in Swapfile Location section. The datastore is then selected for each host in Virtual Machine Swapfile Location settings on the the host configuration tab.

What host also does when you enable this option is it creates a host local swap file, which is named something like this: sysSwap-hls-55de2f14-6c5d-4d50-5cdf-000c296fc6a7.swp

There are scenarios where you need to unmount the swap datastore, such as when you say need to reconnect all of your storage from FC to iSCSI. Even if you shutdown all of your VMs, datastore unmount will fail because the host swap files are still there and you will see an error such as this:

The resource ‘Datastore Name: iSCSI1 VMFS uuid: 55de473c-7f3ae2b5-f9f8-000c29ba113a’ is in use.

See the error stack for details on the cause of the problem.

Error Stack:

Call “HostStorageSystem.UnmountVmfsVolume” for object “storageSystem-29” on vCenter Server “VC.lab.local” failed.

Cannot unmount volume ‘Datastore Name: iSCSI1 VMFS uuid: 55de473c-7f3ae2b5-f9f8-000c29ba113a’ because file system is busy. Correct the problem to retry the operation.

The workaround is to change the setting on the cluster level to store VM swap file in VM directory and reboot all hosts. After a reboot the host .swp file will disappear.

If rebooting the hosts is not desirable, you can SSH to each host and type the following command:

# esxcli sched swap system set –hostlocalswap-enabled false

To confirm that the change has taken effect run:

# esxcli sched swap system get

Then check the datastore and the .swp files should no longer be there.

Conclusion

If you satisfy all of the above requirements you should have no problems when unmounting VMware datastores. vSphere creates a few additional system folders on each of the datastores, such as .sdd.sf and .dvsData, but I personally have never had issues with them.