Load Balancing Ansible Tower Using NSX

Disclamer: this configuration is not validated by either VMware or Red Hat. Make sure it is applicable to your use case and thoroughly test before implementing in production.

Overview

If you landed on this page I trust you already know what Ansible is. It’s a great configuration management tool centred around using YAML to describe the desired state configuration of your various infrastructure components. This desired state is captured in what Ansible calls playbooks, which once written, can then be used in a repeatable way to deploy brand new components or enforce configuration on already deployed ones.

Ansible can be installed and used from CLI, which is usually a good starting point. If you have multiple people using Ansible in your organization, you can also deploy AWX. It’s a free GUI add-on to Ansible, which makes managing concurrent user access to Ansible easier, by adding projects, schedules and credentials management. On top of that there is Ansible Tower. Ansible Tower is a paid version of AWX and gives you additional enterprise features and services like clustering, product support, validated upgrade paths, etc. In this article we will be focusing on Ansible Tower version of the product.

Also worth mentioning that this configuration will be based on Ansible Tower cluster feature, which lets you run all nodes as active/active. Prior to version 3.1 it was called redundancy and worked only in active/passive mode. Redundancy feature is deprecated and is outside the scope of this blog post.

Topology

Deploying multiple Ansible Tower nodes in a cluster already gives you redundancy. If one of the nodes fails you can connect to another node, by just changing your browser URL. The benefit of having a load balancer is that you have one URL you can hit and if a node goes down, such situation is handled by load balancer automatically.

In this example we will be deploying a VMware NSX load-balancer in the following topology:

Configuration

Deploying an NSX load-balancer for HTTPS port 443 is simple, you can find numerous examples of how to create application profiles, monitors, pools and VIPs in official VMware documentation or out on the Internet. But with Ansible there’s one catch. If you try to use the default HTTPS monitor that NSX load balancer comes with, you will find HTTP 400 code in Ansible nginx logs:

10.20.30.40 - - [20/Jan/2020:04:50:19 +0000] "GET / HTTP/1.0" 400 3786 "-" "-" "-"
10.20.30.40 - - [20/Jan/2020:04:50:24 +0000] "GET / HTTP/1.0" 400 3786 "-" "-" "-"
10.20.30.40 - - [20/Jan/2020:04:50:29 +0000] "GET / HTTP/1.0" 400 3786 "-" "-" "-"

And an error in NSX load balancer health check:

As it turns out, when you make a HTTP request to Ansible Tower, specifying HTTP “Host” header is a requirement. Host header simply contains the hostname of the server you’re making a request to. Browsers add this header automatically, that’s why you’re not going to see any errors, when accessing Ansible Tower Using Firefox or Chrome. But NSX doesn’t add this header to the monitor checks by default, which makes Ansible Tower upset.

Here is the trick you need to do to make Tower happy:

Now nginx logs show success code 200:

10.20.30.40 - - [21/Jan/2020:22:54:42 +0000] "GET / HTTP/1.0" 200 11337 "-" "-" "-"
10.20.30.40 - - [21/Jan/2020:22:54:47 +0000] "GET / HTTP/1.0" 200 11337 "-" "-" "-"
10.20.30.40 - - [21/Jan/2020:22:54:52 +0000] "GET / HTTP/1.0" 200 11337 "-" "-" "-"

Load balancer health check is successful:

And pool members are up and reachable:

Note: technically the host header should contain the hostname of the Tower node we’re making a health check on. But since NSX monitor is configured per pool and not per pool member, we have to use a fake hostname “any.host.com” as a workaround. When I was testing it, Tower didn’t complain.

Reference

Even though I said that the rest of the load-balancer configuration is standard, I still think having screenshots for reference is helpful if you need to validate configuration. So find the full list of settings below.

Screenshot 1: Application Profile

Screenshot 2: Service Monitor

Screenshot 3: Pool

Screenshot 4: Virtual Server

Advertisement

Overview of NetApp Replication and HA features

NetApp has quite a bit of features related to replication and clustering:

• HA pairs (including mirrored HA pairs)
• Aggregate mirroring with SyncMirror
• MetroCluster (Fabric and Stretched)
• SnapMirror (Sync, Semi-Sync, Async)

It’s easy to get lost here. So lets try to understand what goes where.

SnapMirror

SnapMirror is a volume level replication, which normally works over IP network (SnapMirror can work over FC but only with FC-VI cards and it is not widely used).

Asynchronous version of SnapMirror replicates data according to schedule. SnapMiror Sync uses NVLOGM shipping (described briefly in my previous post) to synchronously replicate data between two storage systems. SnapMirror Semi-Sync is in between and synchronizes writes on Consistency Point (CP) level.

SnapMirror provides protection from data corruption inside a volume. But with SnapMirror you don’t have automatic failover of any sort. You need to break SnapMirror relationship and present data to clients manually. Then resynchronize volumes when problem is fixed.

SyncMirror

SyncMirror mirror aggregates and work on a RAID level. You can configure mirroring between two shelves of the same system and prevent an outage in case of a shelf failure.

SyncMirror uses a concept of plexes to describe mirrored copies of data. You have two plexes: plex0 and plex1. Each plex consists of disks from a separate pool: pool0 or pool1. Disks are assigned to pools depending on cabling. Disks in each of the pools must be in separate shelves to ensure high availability. Once shelves are cabled, you enable SyncMiror and create a mirrored aggregate using the following syntax:

> aggr create aggr_name -m -d disk-list -d disk-list

HA Pair

HA Pair is basically two controllers which both have connection to their own and partner shelves. When one of the controllers fails, the other one takes over. It’s called Cluster Failover (CFO). Controller NVRAMs are mirrored over NVRAM interconnect link. So even the data which hasn’t been committed to disks isn’t lost.

MetroCluster

MetroCluster provides failover on a storage system level. It uses the same SyncMirror feature beneath it to mirror data between two storage systems (instead of two shelves of the same system as in pure SyncMirror implementation). Now even if a storage controller fails together with all of its storage, you are safe. The other system takes over and continues to service requests.

HA Pair can’t failover when disk shelf fails, because partner doesn’t have a copy to service requests from.

Mirrored HA Pair

You can think of a Mirrored HA Pair as HA Pair with SyncMirror between the systems. You can implement almost the same configuration on HA pair with SyncMirror inside (not between) the system. Because the odds of the whole storage system (controller + shelves) going down is highly unlike. But it can give you more peace of mind if it’s mirrored between two system.

It cannot failover like MetroCluster, when one of the storage systems goes down. The whole process is manual. The reasonable question here is why it cannot failover if it has a copy of all the data? Because MetroCluster is a separate functionality, which performs all the checks and carry out a cutover to a mirror. It’s called Cluster Failover on Disaster (CFOD). SyncMirror is only a mirroring facility and doesn’t even know that cluster exists.

Further Reading

• TR-3548: Best Practices for MetroCluster Design and Implementation
• Data Protection Online Backup and Recovery Guide
• High Availability and MetroCluster Configuration Guide

NetApp Read Caching

In my previous post I gave a high level overview of how NetApp uses NVRAM for write caching. Now I want to move on to read caching in main memory and NetApp Flash Cache/Flash Pool features.

Layers of Memory

The first layer of read caching in NetApp is main memory. For example, there is 4GB of ECC main memory in FAS3140 as opposed to 512MB of NVRAM. It’s 12GB for FAS3220. You can check amount of main memory in your filer by running:

> sysconfig -a

But if you have a random read intensive environment and the size of main memory is too small for your workloads, instead of buying additional spindles you can consider using Flash Cache or Flash Pool features.

Flash Cache (formerly PAM – Performance Acceleration Module) is a PCIe card with flash memory chips on board. The most recent Flash Cache II modules have 2TB of flash. And you can fit as many as 4 such cards in high-end 6xxx NetApp series (or 8 x 1TB cards).

Flash Pool is basically a SSD drive Raid Group which is combined with HDD Raid Groups in the same aggregate to provide caching capabilities. Data is copied (not moved) from HDDs to SSDs to give faster access to more frequently used (hot) data blocks. Both Flash Cash and Flash Pool use FIFO logic to eject less frequently used (cold) data from cache.

Flash Cache

Flash Cache is a second level of read cache memory. When filer decides to evict cached data from main memory, it’s actually moved to a Flash Cash card. Similarly, when client needs to read a data block and filer doesn’t have the data in main memory it now first looks up in Flash Cache and if it’s not there, data is retrieved from disk.

Flash Cache can operate in three modes: Metadata Caching, Normal User Data Caching (default) and Low-Priority Data Caching. First mode caches only metadata, second caches metadata and data blocks and the last one lets you cache data which is not normally cached, which is writes and sequential reads.

In fact, when write request comes into the system, it’s actually cached in main memory first and then logged in NVRAM. When CP occurs, data is sent to hard drives and becomes a first target for eviction from main memory after that.  If you enable Low-Priority Data Caching this data goes to Flash Cash card instead. It’s not write caching per se, because writes have been already sent to disks. But it’s helpful in workloads, when data which has just been written may need to be accessed again in a short period of time. It’s called read-write caching.

Caching sequential reads is generally not a good idea, because they overwrite large amounts of cache. But if your environment may benefit from that, you again can use Low-Priority Data Caching option.

Flash Pool

Flash Pool has one significant difference from Flash Cache. It works at an aggregate level, not at system level as Flash Cache. If you have only one stack of shelves and one aggregate, it makes no difference. But it’s almost always not the case.

Read Caching

Flash Pool uses essentially the same mechanism for read caching. When data is first accessed it goes to main memory. When filer needs to free up some space in main memory, blocks are moved to SSD disks as part of a Consistency Point.

NetApp uses scanner to evict blocks from SSD cache (see figure above). When cache gets full, scanner kicks in and reduces each block’s temperature by one level. Blocks with the lowest temperature are evicted. Each time block is accessed by client it’s temperature is incremented.

Write caching

Flash Pool can be used for write caching of partial overwrites, in contrast to Flash Cache which is purely a read cache.

WAFL is optimized for writes, because filer can put data at any place in file system. And when new data comes in, filer makes a so called “full write”, which writes a full stripe of data and parity. But when part of stripe needs to be overwritten, all other data blocks from the stripe need to be read from disk to recalculate parity. It’s a very expensive operation. Flash Pool can be used to cache partial overwrites and even better optimize performance.

If write caching is enabled, instead of going to HDDs this data is written to SSD disks as part of a Consistency Point. And unlike read caching it exists temporarily only on SSDs.

After each scanner run, write cache block temperature is decremented by one. When write is overwritten, it’s temperature gets back to normal, but it can’t go higher than that. When block is about to be evicted, it’s read back to main memory and then written to HDDs as part of next CP.

Policies

Flash Pool read/write policies are almost the same as Flash Cache ones. Read policies are: meta, random read (default), random-read-write, none. Write policies: random-write, none (default).

Notes

Flash Pool and Flash Cache can be combined in one system and configured at per volume level. But Flash Cache can’t be used for volumes which are already being cached by Flash Pool. It’s either-or.

NetApp filers have Predictive Cache Statistics (PCS) feature built in, which allows you to analyse your workload and predict if storage system will benefit from additional cache.

Further Reading

TR-3832: Flash Cache Best Practices Guide

TR-3801: Introduction to Predictive Cache Statistics

TR-4070: Flash Pool Design and Implementation Guide

Differences between NetApp Flash Cache and Flash Pool