Configure Health Checks for NodeBalancers to Remove Non-Working Back Ends

Health checks are a key part of ensuring maximum availability for cloud-based applications. NodeBalancers offer flexible choices in both passive and active health checks, and they can automatically take unreachable servers offline while using TCP and HTTP/S parameters, guaranteeing clients always get the content they request.

This guide discusses various types of passive and active heath checks, along with how to configure, test, and verify each type of health check.

Types of Health Checks

NodeBalancers support passive and active health checks. When a back-end server doesn’t respond to a health check, the NodeBalancer won’t forward requests to it until it starts responding again.

Passive checks simply monitor traffic between the NodeBalancer and back-end servers. With active health checking, you configure the NodeBalancer to send probes involving small amounts of traffic to check back-end server status. NodeBalancers offer three types of active health checks:

• TCP Connection: Requires completion of a valid TCP three-way handshake to establish a connection.
• HTTP Status: Requires a valid HTTP 2xx or 3xx response from a back-end server.
• HTTP Body: Requires a match against a regular expression in the body of an HTTP response.

For additional details and options for configuring NodeBalancer health checks, see Configuration Options for NodeBalancers: Health Checks.

Deploy the Testbed

To set up a testbed and get started with NodeBalancer health checks, provision four Compute Instances and one NodeBalancer:

• Three back-end server instances (srv1, srv2, and srv3) running the NGINX web server.
• One client instance (clt1) to send requests via the NodeBalancer using hey, an open source web traffic generator.
• One NodeBalancer (nb1) to distribute requests among the back-end servers.

The figure below illustrates the testbed topology:

Deploy Back-End Servers

Use the following settings to provision each of the three back-end server instances:

• Choose a Distribution: This guide was written for Ubuntu 24.04 LTS, but is applicable to most other Debian-based Linux distributions.
• Region: All servers must be in the same region so that they can communicate with your NodeBalancer.
• Linode Plan: A Shared CPU Nanode 1 GB Compute Instance is adequate for the tests in this guide.
• Details : Name each instance srv1, srv2, and srv3, respectively, under Linode Label.
• Add-ons: A Private IP is mandatory for each instance. NodeBalancers only distribute client requests to back-end servers over a region’s private network.

1. Once your servers are up and running, log in to each using SSH or Lish, and install the NGINX web server:

   sudo apt install nginx

2. Use any web browser to verify server reachability. The default NGINX web pages should be accessible at http://SERVER_IP_ADDRESS:

   

3. Altering this message to be server-specific is useful later when you use HTTP Body checks, and allows you to tell the difference between each back-end server. Login to each server instance and alter the welcome message to be server-specific:

   srv1

   cd /var/www/html
   sudo perl -pi -e 's/Welcome to nginx/Welcome to srv1/' index.nginx-debian.html
   srv2

   cd /var/www/html
   sudo perl -pi -e 's/Welcome to nginx/Welcome to srv2/' index.nginx-debian.html
   srv3

   cd /var/www/html
   sudo perl -pi -e 's/Welcome to nginx/Welcome to srv3/' index.nginx-debian.html

4. Return to your web browser and refresh the welcome pages. You should see your changes:

   

Do not proceed until you can reach all three servers and see the updated homepage on each.

Deploy the Client

Create a fourth Compute Instance to serve as a client. This can be in the same region as the servers, but it’s not required. In this example, the client is called clt1.

After the client setup completes, install hey, an open-source web traffic generator:

sudo apt install hey

Deploy the NodeBalancer

Once your back-end servers and your client are in place, the next step is provisioning your NodeBalancer.

It is a best practice to configure load balancing first, and then add health checks once load balancing is functional. In this case, the NodeBalancer performs TCP load balancing on port 80 using a round robin algorithm. Because HTTP runs on top of TCP, you can use either TCP or HTTP/S protocols to load balance web traffic.

1. To provision your NodeBalancer, go to the Cloud Manager main menu, click on NodeBalancers in the left sidebar, and select Create NodeBalancer on the subsequent screen.

2. Most settings may be left as their default values, but the following settings should be edited for the purposes of this test:

   • NodeBalancer Label: In this example, name the NodeBalancer nb1.
   • Region: Choose the same region as your back-end servers.
   • Protocol: Use TCP for the initial setup, though you’ll be generating HTTP traffic later on.
   • Session Stickiness: Set to None. The default setting (Table) would send all subsequent requests from the same client to the same back-end server.
   • Passive Checks: Disable these. You’ll enable, configure, and test passive checks in the next step.
   • Backend Nodes: Add all three of your back-end server instances. The Label setting should match the server labels, in this case srv1, srv2, and srv3.

   This figure shows a NodeBalancer setup screen after basic configuration:

3. When done, click the Create NodeBalancer button to initialize the instance. Our Create a NodeBalancer guide has more details on load balancer setup.

Verify Load Balancing

This section shows how to test if your load balancing setup works.

1. Before running hey from your client instance, check the NGINX log file /var/log/nginx/access.log on your back-end servers to determine the number of times each back-end server has returned the default homepage located at /var/www/html/index.nginx-debian.html. The grep command finds all instances of that filename, and piping the output to the wc -l command counts the number of instances.

   srv1, srv2, and srv3

   cd /var/log/nginx
   sudo grep index.nginx-debian.html access.log | wc -l

   0

   0

   0

   These are your baseline counts. They may be zero on brand-new instances, but expect these values to increase as the server stays online. Your outputs may not necessarily match those shown in this guide.

2. From your client machine, run hey to generate web traffic. This command sends 9,999 HTTP requests over 99 concurrent connections. Make sure to use the hostname or IP address of the NodeBalancer, and not that of any individual back-end server:

   clt1

   hey -c 99 -n 9999 http://NODEBALANCER_IP_ADDRESS/index.nginx-debian.html

   A successful command run returns a message like this:

   Status code distribution:
     [200]	9999 responses

3. Now rerun the wc command on each back-end server:

   srv1, srv2, and srv3

   cd /var/log/nginx
   sudo grep index.nginx-debian.html access.log | wc -l

   3333

   3333

   3333

   Note the differences in line counts between this run and the previous one. The NodeBalancer distributed roughly the same number of requests to srv1, srv2, and srv3, with each server respectively logging 3,333, 3,333, and 3,333 requests. Now that you’ve verified that load balancing is working as intended, it’s time to enable health checks.

Passive Health Checks

A passive health check doesn’t generate any traffic, but instead monitors traffic between the NodeBalancer and back-end servers. This is the simplest method of monitoring server health.

1. To use passive checks, return to the NodeBalancers section of the Cloud Manager.

2. Select the nb1 NodeBalancer, switch to the Configurations tab, and open the Port 80 configuration.

3. Enable the Passive Checks switch:

   

4. Click Save to commit this change.

Verify Health Checks

1. Test passive health checking by stopping the NGINX service on one of your back-end servers (e.g. srv1):

   srv1

   sudo systemctl stop nginx

2. Return to the client and run the hey generator again:

   clt1

   hey -c 99 -n 9999 http://NODEBALANCER_IP_ADDRESS/index.nginx-debian.html

3. Once it completes, compare the access.log file lengths on the two remaining web servers:

   srv1, srv2, and srv3

   cd /var/log/nginx
   sudo grep index.nginx-debian.html access.log | wc -l

   3333

   8282

   8383

   Taking one of back-end servers offline should result in the NodeBalancer sending roughly 5,000 HTTP requests to each of the two remaining servers. Without passive checks enabled, the NodeBalancer would have continued to forward requests to all three back-end servers, with about 3,333 of them failing.

4. Before proceeding, restart NGINX on the failed server (e.g. srv1):

   srv1

   sudo systemctl start nginx

5. Optional: Repeat steps two and three to verify that the NodeBalancer again sends traffic to all three back-end servers.

TCP Connection Active Health Checks

With TCP Connection active health checking enabled, the NodeBalancer sends probes to each back-end server. These add slightly more traffic, but also provide more fine-grained control over health-checking attributes.

There are many different TCP implementations, and it’s not uncommon to tune parameters for improved scalability and/or data-transfer rates. With this type of health checking, a NodeBalancer responds to the exact TCP behavior in your cloud network.

From your NodeBalancer’s Configurations menu, select TCP Connection under Active Health Checks. NodeBalancers offer three configuration options this type of health checking:

• Interval: The number of seconds between probes.
• Timeout: The number of seconds to wait before marking a probe as unsuccessful.
• Attempts: The number of failed probes required before the NodeBalancer takes a back-end server out of rotation.

For testing purposes, use the default setting for all three options. Be sure to disable Passive Checks and to Save your NodeBalancer configuration.

To test TCP Connection active health checks, use the procedure outlined in the Verify Health Checks section above. Once again, the remaining two active back-end servers should each have increased by nearly 5,000 requests each:

3333

13332

13332

HTTP Status Active Health Checks

HTTP Status active health checks also send probes, and require a valid HTTP 2xx or 3xx response from each back-end server. This is a good choice when the top priority is ensuring successful delivery of web pages. This type of active health check requires configuration of the NodeBalancer’s Protocol option to HTTP or HTTPS. Use HTTP for this example.

NodeBalancers offer four configuration options for HTTP Status active health checks:

• Interval: The number of seconds between probes.
• Timeout: The number of seconds to wait before marking a probe as unsuccessful.
• Attempts: The number of failed probes required before the NodeBalancer takes a back-end server out of rotation.
• Check HTTP Path: The part of the URL after the hostname or IP address. For testing, set this to /index.nginx-debian.html, which is the NGINX default homepage.

Use the default settings for the first three options and set the Check HTTP Path. Change the Protocol from TCP to HTTP, and be sure to Save the NodeBalancer’s configuration changes.

To test HTTP Status active health checks, use the procedure outlined in the Verify Health Checks section above. The two active back-end server’s request counts should once again increase by approximately 5,000 requests each:

3333

18281

18382

HTTP Body Active Health Checks

HTTP Body is the most sophisticated type of health check. It peers into the content of web pages and uses regular expression (regex) pattern matching to verify if back-end servers correctly deliver content.

This type of health check is a good choice for multi-tier applications, such as those backed by database servers. The advantage of an HTTP Body check is that it actually parses text to validate delivery of the intended content. Whereas an HTTP Status check only shows that back-end servers successfully return a web page, even an empty or erroneous one.

NodeBalancers offer five configuration options for HTTP Body active health checks:

• Interval: The number of seconds between probes.

• Timeout: The number of seconds to wait before marking a probe as unsuccessful.

• Attempts: The number of failed probes required before the NodeBalancer takes a back-end server out of rotation.

• Check HTTP Path: The part of the URL after the hostname or IP address. For testing, set this to /index.nginx-debian.html, which is the default NGINX homepage.

• Expected HTTP Body: A Perl-compatible regular expression (PCRE) embedded within the body of the requested object, in this case, text within the body of a web page. Because you previously inserted unique text on each server’s homepage, you can use the following string to cover all three back-end servers:

  Welcome to srv[1-3]!

  This regex pattern matches any of srv1, srv2, or srv3.

Use the default settings for the first three options. Set the Check HTTP Path location and the Expected HTTP Body string as described above, and make sure the Protocol is set to HTTP. Remember to Save your NodeBalancer configuration before moving on.

You can use two tests to verify if HTTP Body active health checks are working:

• First, run a test that matches against content on all three servers using the regex above. To test for body content and the ability to take a server offline, use the procedure outlined in the Verify Health Checks section above. You should see roughly equal loading across all back-end servers that the NodeBalancers considers to be active:

  3333

  23331

  23331

• Second, run a test that matches against the content of just one server:

  1. Since each back-end server has a unique string in its homepage, you can change the NodeBalancer’s Expected HTTP Body setting to something like this:

     Welcome to srv1!

  2. Click Save to enable this change in the NodeBalancer configuration.

  3. With all three back-end servers active, use the hey command as described above to generate web traffic to the NodeBalancer.

  4. Since only srv1 matches the new pattern, the NodeBalancer marks the other two back-end servers offline. Verify this by comparing the length of each server’s access.log file before and after offering web traffic from the client:

     13332

     23331

     23331

In this simple example, the NodeBalancer forwards traffic to only one back-end server. You can use this same capability with server groups as well as individual servers.

One example use case is upgrading a web application that displays a version number on a particular page. You can use HTTP body active health checks to take some servers offline before upgrading them. Afterwards, change the string to match the upgraded version number, then take the remaining servers offline to complete the upgrade. This method ensures clients never encounter unreachable servers.