In this guide, we will explore ways of troubleshooting the F5 load balancer for Cloudmersive Private Cloud. As with any troubleshooting, we will take a systematic approach to identify the root cause of the issue. By checking for the most common issues, we can identify and confirm the root cause.

In this guide we will cover:

• Confirming Private Cloud server health
• Confirming network connectivity between the F5 load balancer and Cloudmersive Private Cloud, and from other machines and Cloudmersive Private Cloud
• Send a Test Request from the F5 Appliance to the Cloudmersive Private Cloud Server
• Checking for Windows Firewall rules on the Cloudmersive Private Cloud server
• Checking for hardware/virtualized firewall rules impacting traffic flow to the Cloudmersive Private Cloud server

Confirming Private Cloud server health

First, we need to confirm if the Cloudmersive Private Cloud server(s) are healthy. To do this, navigate to the following web page on the server itself:

http://ip-address-of-server/virus/status

Or if using TLS:

https://server-dns-name/virus/status

You should see an HTTP 200 response and in the response XML/JSON see the value Online under Health.

Now, try navigating to the same URL on a different machine on the same network that would have access to the server. Verify that this is also reachable and shows the same page and response code.

Finally, try navigating to this URL through your F5 load balancer:

https://F5-load-balancer-DNS-name/virus/status

Note that since you will likely have round robin enabled you will need to refresh this page multiple times until you see a response from the node in question. You will be able to identify this page because in the status page response, the IP address of the server is included.

Configure F5 Load Balancer Health Checks

If you do not see the status page from section 1 load through the F5 load balancer, or you receive a timeout, then you know that traffic is not flowing to the Cloudmersive Private Cloud server in question. Based on this, we recommend setting up F5 health checks against the target servers. This will configure the F5 load balancer to continuosly monitor the health of each target server and only direct traffic to healthy servers. To do this, follow the (F5 health check procedures)[https://cloudmersive.com/knowledge-base/Cloudmersive-Private-Cloud-F5-Load-Balancer-Configuration-Best-Practices].

Once configured, review the health check status in the F5 load balancer. Any nodes that are shown as unhealthy have an issue. If these nodes show as healthy in section 1, then it means that the network traffic cannot flow from the F5 Load Balancer to the Cloudmersive Private Cloud server in question. If the nodes show as unhealthy in section 1, then proceed to addressing health issues on the Cloudmersive Private Cloud server itself.

Confirming network connectivity between the F5 load balancer and Cloudmersive Private Cloud, and from other machines and Cloudmersive Private Cloud

In this section, we will test low-level connectivity between the F5 load balancer and Cloudmersive Private Cloud server. This test will bypass the Cloudmersive API microservices and just test network connectivity alone.

To do this, navigate to the following web page on the server itself:

http://ip-address-of-server/index.html

Or if using TLS:

https://server-dns-name/index.html

This page is statically served by the Cloudmersive Private Cloud server through IIS at a very low level. It does not require the Cloudmersive API microservices on the machine to be available and so can be used to test network connectivity.

You should see an HTTP 200 response and in the response HTML content.

Now, try navigating to the same URL on a different machine on the same network that would have access to the server. Verify that this is also reachable and shows the same page and response code.

Finally, try navigating to this URL through your F5 load balancer:

https://F5-load-balancer-DNS-name/index.html

Note that since you will likely have round robin enabled you will need to refresh this page multiple times until you see a response from the node in question. You will be able to identify this page because in the status page response, the IP address of the server is included.

You can further check the Cloudmersive Private Cloud server IIS logs. All incoming requests will appear here. If there are requests that do NOT appear here, e.g. from the F5 load balancer IP, then it means that the traffic is not reaching the Cloudmersive Private Cloud server at a network level. In this case, the traffic is being blocked by network routing rules, a hardware firewall, or a software firewall.

Send a Test Request from the F5 Appliance to the Cloudmersive Private Cloud Server

Connect to the F5 appliance BIG-IP server. This can be done typically through SSH. Note that this is different than the management server.

# Step 1: SSH to your BIG-IP
ssh admin@<F5_Management_IP>

# Step 2: Go into bash
tmsh run /util bash

# Step 3: Send an HTTP request from a specific self IP 
curl -v --interface 10.10.10.5 http://<cloudmersive_server_IP>:80

You can also try these additional commands:

# ping
tmsh run /util ping <cloudmersive_server_IP>

# telnet
tmsh run /util telnet <cloudmersive_server_IP> 443

# traceroute
tmsh run /util traceroute <cloudmersive_server_IP>

Checking for Windows Firewall rules on the Cloudmersive Private Cloud server

In this section we will check if there are any Windows Firewall rules on the Cloudmersive Private Cloud server restricting inbound HTTP and HTTPS traffic. Run the following command from Administrator Powershell:

$fwPolicy2 = New-Object -ComObject HNetCfg.FwPolicy2

$fwPolicy2.Rules |
    Where-Object {
        $_.Protocol -eq 6 -and
        $_.LocalPorts -match "\b(80|443)\b" -and
        $_.Enabled -eq $true
    } |
    ForEach-Object {
        $rule = $_
        # Split the LocalPorts by comma
        $ports = $rule.LocalPorts -split ','
        
        foreach ($port in $ports) {
            # Trim spaces, just in case
            $portTrim = $port.Trim()
            # Only output if it's '80' or '443'
            if ($portTrim -eq '80' -or $portTrim -eq '443') {
                [PSCustomObject]@{
                    Name            = $rule.Name
                    Description     = $rule.Description
                    Enabled         = $rule.Enabled
                    Action          = if ($rule.Action -eq 0) {"Block"} else {"Allow"}
                    Direction       = if ($rule.Direction -eq 1) {"Inbound"} else {"Outbound"}
                    LocalPort       = $portTrim
                    Protocol        = "TCP"
                    LocalAddresses  = $rule.LocalAddresses
                    RemoteAddresses = $rule.RemoteAddresses
                    Profiles        = $rule.Profiles
                }
            }
        }
    } |
    Format-Table -AutoSize

Review and modify as appropriate any rules shown.

Checking for hardware/virtualized firewall rules impacting traffic flow to the Cloudmersive Private Cloud server

Work with your networking team to identify all routing rules and hardware firewall rules that could impact traffic flow from the F5 load balancer to the Cloudmersive Private Cloud servers. Note that specific rules may apply to only a subset of the Private Cloud servers and so may cause selective issues that require rule adjustment.