Posts Tagged ‘RDM’

History of vSphere Storage Size Limitations

June 5, 2016

data-storageThis seems like a straightforward topic. If you are on vSphere 6 you can create VMFS datastores and VM disks as big as 64TB (62TB for VM disks to be precise). The reality is, not all customers are running the latest and greatest for various reasons. The most common one is concerns about reliability. vSphere 6 is still at version 6.0. Once vSphere 6.1 comes out we will see wider adoption. At this stage I see various versions of vSphere 5 in the field. And even vSphere 4 at times, which is officially not supported by VMware since May 2015. So it’s not surprising I still get this question, what are the datastore and disk size limits for various vSphere versions?

Datastore size limit

The biggest datastore size for both VMFS3 and VMFS5 is 64TB. What you need to know is, VMFS3 file system uses MBR partition style, which is limited to 2TB. The way VMFS3 overcomes this limitation is by using extents. To extend VMFS3 partition to 64TB you would need 32 x 2TB LUNs on the storage array. VMFS5 file system has GPT partition style and can be extended to 64TB by expanding one underlying LUN without using extents, which is a big plus.

VMFS3 datastores are rare these days, unless you’re still on vSphere 4. The only consideration here is whether your vSphere 5 environment was a greenfield build. If the answer is yes, then all your datastores are VMFS5 already. If environment was upgraded from vSphere 4, you need to make sure all datastores have been upgraded (or better recreated) to VMFS5 as well. If the upgrade wasn’t done properly you may still have some VMFS3 datastores in your environment.

Disk size limit

For .vmdk disks the limitation had been 2TB for a long time, until VMware increased the limit to 62TB in vSphere 5.5. So if all of your datastores are VMFS5, this means you still have 2TB  .vmdk limitation if you’re on 5.0 or 5.1.

For VMFS3 file system you also had an option to choose block size – 1MB, 2MB, 4MB or 8MB. 2TB .vmdk disks were supported only with the 8MB block size. The default was 1MB. So if you chose the default block size during datastore creation you were limited to 256GB .vmdk disks.

The above limits led to proliferation of Raw Device Mapping disks in many pre 5.5 environments. Those customers who needed VM disks bigger than 2TB had to use RDMs, as physical RDMs starting from VMFS5 supported 64TB (pRDMs on VMFS3 were still limited to 2TB).

This table summarises storage configuration maximums for vSphere version 4.0 to 6.0:

vSphere Datastore Size VMDK Size pRDM Size
4.0 64TB 2TB 2TB
4.1 64TB 2TB 2TB
5.0 64TB 2TB 64TB
5.1 64TB 2TB 64TB
5.5 64TB 62TB 64TB
6.0 64TB 64TB 64TB


The bottom line is, if you’re on vSphere 5.5 or 6.0 and all your datastores are VMFS5 you can forget about the legacy .vmdk disk size limitations. And if you’re not on vSphere 5.5 you should consider upgrading as soon as possible, as vSphere 5.0 and 5.1 are coming to an end of support on 24 of August 2016.

Connecting VMware ESXi Hosts to NetApp: MPIO Configuration

May 23, 2013


NetApp filers are active/active ALUA arrays. It means that you can access LUNs configured on one controller via the second one. But access to the partner’s LUNs is provided through the internal interconnect and is always slower. That’s why the paths to the controller through the partner are called “unoptimized”. Their primary usage is to provide backup paths in case of a failover.

Fixed path selection

VMware hosts by default use “VMW_SATP_DEFAULT_AA” Storage Array Type Policy (SATP) and “Fixed” Path Selection Policy (PSP) for active/active arrays. If ESXi host is configured with these SATP and PSP, it will access each LUN via one particular path, even if you have two FC ports on each of the controllers.

VMware host can’t automatically identify optimized path. So you can either set it manually or use functionality of NetApp Virtual Storage Console (VSC) plug-in for VMware. Just go to the Monitoring and Host Configuration -> Overview section of VSC, right click on ESXi host and click “Set Recommended Values”. If you don’t do that, ESXi hosts will run I/O traffic through a randomly identified path, which could turn out to be unoptimized. It means you will push heaps of I/O through the partner node and experience higher latencies.

You can check if you’re using non-optimized paths by looking for such warnings on NetApps:

filer_01> Mon May 6 10:30:45 EST [filer_01: ems.engine.inputSuppress:error]: Event ‘scsitarget.partnerPath.misconfigured’ suppressed 327 times since Mon May 6 09:30:48 EST 2013.
Mon May 6 10:30:45 EST [filer_01: scsitarget.partnerPath.misconfigured:error]: FCP Partner Path Misconfigured – Host I/O access through a non-primary and non-optimal path was detected.

Or run “lun stats -o” and look for huge numbers under “Partner Ops” and “Partner KBytes”.

ALUA configuration

If you want to utilize both links to the controller in a round robin fashion, you need to do some additional configuration. You should enable ALUA for your VMware ESXi hosts initiator group on NetApp:

igroup set <group> alua yes

Now you need to reboot ESXi host. After a reboot it will see that storage is ALUA-capable and change SATP to VMW_SATP_ALUA and PSP is “Most Recently Used”. To utilize load balancing between two controller paths you need to change PSP to “Round Robin”. Again, you can do that either manually or via VSC.

Note: Don’t ever use ALUA and VMW_SATP_ALUA if you have Windows Server 2003 MSCS or Windows Server 2008 Failover Cluster with shared RDM LUNs. It’s an unsupported configuration and you can run into a cluster failure situation. It’s described in many places:

In this case leave SATP as “VMW_SATP_DEFAULT_AA”,  PSP as “Fixed” and make sure that you use optimized paths.