NSX Optimistic Locking and PowerNSX

Recently, when working on some NSX-V automation, I came across an interesting issue, which I want to discuss here, since there’s almost no information on the Internet (while I’m writing this), that would help to solve it or even point you in the right direction. It has to do with PowerNSX and Optimistic Locking in NSX (which technically is not even a locking mechanism), but let’s start from the beginning.

If you ever used PowerNSX module to automate NSX via PowerShell you noticed that most of the code examples use pipelines to run PowerNSX cmdlets. So instead of using variables, like so:

$Edge = Get-NsxEdge vRA7 _ edge
$LoadBalancer = Get-NsxLoadBalancer -Edge $Edge
Set-NsxLoadBalancer -LoadBalancer $LoadBalancer -enabled
New-NsxLoadBalancerApplicationProfile -LoadBalancer $LoadBalancer -Name $WebAppProfileName -Type $VipProtocol –SslPassthrough

all commands are run this way instead:

Get-NsxEdge vRA7_edge | Get-NsxLoadBalancer | Set-NsxLoadBalancer -enabled
Get-NsxEdge vRA7_dge | Get-NsxLoadBalancer | New-NsxLoadBalancerApplicationProfile -Name $WebAppProfileName -Type $VipProtocol –SslPassthrough

What’s the difference you may ask, besides the fact the the second variant is slower, because you retrieve edge and load balancer objects multiple times, instead of once, compared to the first variant? There’s actually a strong reason for it. More specifically, it is the following error that you gonna get if you don’t use pipelines:

invoke-nsxwebrequest : Invoke-NsxWebRequest : The NSX API response received indicates a failure. 409 : Conflict : Response Body: {“errorCode”:101, “details”:”Concurrent object access error. Refresh UI or fetch the latest copy of the object and retry the operation.”, “rootCauseString”:null, “moduleName”:null, “errorData”:null}

See, NSX uses Optimistic Locking (yes, there’s Pessimistic Locking as well) to handle concurrency. Its purpose is to make sure that if you’re making a change to an object in NSX you are aware of its current state. In the above example, you saved load balancer into a variable, changed the load balancer state to enabled and then tried to create an application profile, supplying load balancer saved in a variable as a parameter to the cmdlet. But the load balancer (and edge) state has changed and you’re basically using an old (stale) version of the object. You either have to retrieve the current state of the object again or avoid this issue all together by simply using pipelines and retrieve the up-to-date version of the object with every call.

Read this article if you want to know more about Optimistic Locking:

• https://enterprisecraftsmanship.com/2017/09/18/optimistic-locking-automatic-retry/

If you found this useful, please leave a comment, smash that like button and hit notification bell to not miss new blog posts ever again.

Advertisement

Isilon Performance Considerations

In my previous post “EMC Isilon Overview” I talked about general Isilon configuration. In this post I want to describe some of the performance tuning options.

SSD Acceleration

You can choose from three types of Isilon nodes: S-series, X-series and NL-series. And within the node series you can select amount of memory, number/size/type of disk drives and the number of 1GB/s and 10GB/s network ports.

S- and X- series nodes can have SSD drives, which you can use for metadata and/or data, depending on how much flash storage you have. Isilon have four SSD strategies: “Metadata Read Acceleration”, “Metadata Read/Write Acceleration”, “Data on SSDs” and “Avoid SSDs”. All strategies are pretty much self-explanatory. When using first strategy Isilon creates a copy of metadata on SSD disks for read acceleration. With second strategy it mirrors metadata reads and writes to SSDs (requires four to six times more space). Third strategy allows you to have data on SSDs. And forth disables SSD acceleration all together. You can configure SSD strategy on a file level policy level. And if you want to have “Data on SSDs” you can redirect only particular files (say with the most recent access timestamps) to SSDs.

Isilon allows you to have SSD metadata acceleration even on node pools that don’t have SSDs in them. It’s called Global Namespace Acceleration (GNA) and requires at least 20% of nodes to have 1 or more  SSD disks and requires 1.5% or more (2% is recommended) of total storage to be SSD-based.

Data Access Patterns

Isilon can be used for different type of workloads. You can have VMware datastores on it connected via iSCSI or have CIFS file shares or maybe use it for streaming. Depending on your data access patterns you can tweak Isilon for Random, Concurrency or Streaming content, which affects how Isilon writes data on disks and how it uses its cache.

“Random” read/write type of workloads are typical for VMware environments. With this setting Isilon disables prefetching of data into read cache. And this setting is default for iSCSI LUNs.

With “Concurrency” Isilon optimizes data layout for simultaneous access of many files from the storage array. It uses moderate level of prefetching in this case. And for every 32MB of a file it tries to hit the same disk within the node. This it is default for all data except iSCSI LUNs.

And “Streaming” is for the fast access to big files like media content. It has the most aggressive level of prefetching and tries to use as many disk drives as possible when writing data on a cluster.

SmartCache

This setting affects write caching. With SmartCache turned on, data that comes in to a cluster will be cached in node’s memory. This is not NVRAM and if node fails uncommitted data is lost. This might not be that critical for NFS and CIFS data, but loosing iSCSI blocks can result in file system being unreadable. So be careful with SmartCache. You can specifically disable write cache on iSCSI LUNs if you want to.

Accelerator Nodes

Isilon provides two types of accelerator nodes: Backup Accelerator and Performance Accelerators. Both of them don’t contribute their disks to storage pool and provide additional capabilities to a cluster instead. Backup accelerator has four 4GB/s FC ports for connections to tape libraries and allows to offload backups from the storage nodes. And performance accelerators add additional CPU and memory capacity to a cluster without adding any storage.

Data Protection Overhead

The default data protection level on Isilon is +2:1. It protects the storage system from two disk failures or one node failure. It gives you sufficient level of protection and doesn’t have too much overhead. If you need a higher level of protection, you need to realize that it can introduce much overhead. Below is the table which shows amount of overhead depending on protection level and number of nodes.

As you can see, for six nodes +1 and +3:1 protection levels have the same overhead, but +3:1 gives better protection. So you need to understand the impact of changing protection level and set it according to your needs.