When you run HAProxy in a two-node Pacemaker cluster, monitoring the HAProxy service itself is not always enough.
A node can be reachable while the cluster resource is stopped. One cluster node can also fail without causing an immediate service outage because the second node takes over. The situation that really needs attention is when both HAProxy nodes are down.
In this setup, Zabbix can use
crm_mon
to check the Pacemaker cluster and alert when the cluster loses one or both nodes.
This article shows one way to build that monitoring.
The overall Cluster setup
For this example, we have two HAProxy servers:
haproxy01
haproxy02
Both servers are members of the same Pacemaker/Corosync cluster.
Zabbix is running separately and is responsible for monitoring the environment.
The goal is to have Zabbix report three basic states:
0 = both nodes are online
1 = one node is offline
2 = both nodes are offline
The first two states are useful for operational monitoring, but the important one is
2
. If both nodes are down, the HAProxy service is no longer highly available.
Why check ?
crm_mon
A common approach is to check the HAProxy service directly:
# systemctl is-active haproxy
That is useful, but it only tells us about HAProxy on the machine where the command is executed.
Pacemaker has its own view of the cluster.
crm_mon
shows the state of the cluster nodes and resources.
For example:
# crm_mon -1
On a healthy cluster, you might see something like:
Node List:
Online: [ haproxy01 haproxy02 ]
Full List of Resources:
haproxy (systemd:haproxy): Started haproxy01
If
haproxy02
goes down, the output could look like:
Node List:
Online: [ haproxy01 ]
OFFLINE: [ haproxy02 ]
If both nodes are unavailable:
Node List:
OFFLINE: [ haproxy01 haproxy02 ]
The exact output depends on the Pacemaker version, so it is worth checking the output on your own servers before writing the monitoring script.
1. Creating a simple check script (approach 1)
Rather than putting a long shell command directly into Zabbix, I prefer using a small script and letting the Zabbix agent return a single value.
For example:
/usr/local/bin/check_haproxy_cluster.sh
A simple version could be:
#!/bin/bash CRM_MON="/usr/sbin/crm_mon" if [ ! -x "$CRM_MON" ]; then echo "-1" exit 1 fi OUTPUT=$($CRM_MON -1 2>/dev/null) if [ $? -ne 0 ]; then echo "-1" exit 1 fi ONLINE=$(echo "$OUTPUT" | grep -A1 "Online:" | grep -o "haproxy[0-9]*" | wc -l) case "$ONLINE" in 2) echo "0" ;; 1) echo "1" ;; 0) echo "2" ;; *) echo "-1" ;; esac
Then make it executable:
# chmod +x /usr/local/bin/check_haproxy_cluster.sh
Run it manually:
/usr/local/bin/check_haproxy_cluster.sh
On a healthy cluster, the result should be:
0
If one node is down:
1
And if both nodes are down:
2
The
-1
value is used for an error, for example if
# crm_mon
cannot be executed.
One thing to watch out for
The example above assumes the node names follow a pattern such as
haproxy01
and
haproxy02
That is fine for a simple setup, but I would not use the script unchanged in production without checking the actual
crm_mon
output.
You should also make sure that finding a node name in the output really means that the node is online. Depending on the Pacemaker version, a more precise parser may be necessary.
2. Add the check to the Zabbix agent
The next step is to expose the script through a Zabbix
UserParameter
For example:
UserParameter=haproxy.cluster.status,/usr/local/bin/check_haproxy_cluster.sh
This can be added to the Zabbix agent configuration or to a separate file under the agent's configuration directory.
After changing the configuration, restart the agent:
# systemctl restart zabbix-agent
You can test the item before doing anything in the Zabbix frontend:
zabbix_agentd -t haproxy.cluster.status
A healthy result should look similar to:
haproxy.cluster.status [t|0]
3. Create the Zabbix item
In Zabbix, create an item using the following settings:
Name
HAProxy Cluster Status
Type
Zabbix agent
Key
haproxy.cluster.status
Type of information
Numeric (unsigned)
For the update interval, something like 30 seconds is usually reasonable:
30s
There is nothing special about 30 seconds. The interval should depend on how quickly you need to know about a failure.
4. Create the triggers
Now we can turn the returned values into actual alerts.
For both nodes being down:
last(/HAProxy Cluster/haproxy.cluster.status)=2
The trigger name could simply be:
Both HAProxy cluster nodes are DOWN
I would normally give this a high severity because there is no longer a working HAProxy node available.
For a single node failure:
last(/HAProxy Cluster/haproxy.cluster.status)=1
The trigger could be:
One HAProxy cluster node is DOWN
This can have a lower severity.
That distinction is useful. A single node failure means the cluster is degraded, but the service may still be working normally. Two failed nodes are a different situation.
The resulting states
The Zabbix configuration is then roughly:
| Cluster state | Value | Alert |
|---|---|---|
| Both nodes online |
0 |
OK |
| One node offline | 1 | Warning |
| Both nodes offline |
2 |
High/Critical |
| Check failed |
-1 |
Monitoring problem |
This is much easier to work with than having Zabbix process the complete output of
# crm_mon
There is a problem with checking from the HAProxy node
There is an important point here.
If you run the script only on
haproxy01
, what happens when
haproxy01
itself completely disappears?
The Zabbix agent on that server is gone as well. Zabbix cannot ask it what happened to the cluster.
This can leave a blind spot exactly when you need monitoring the most.
For that reason, I would run the cluster check from an independent monitoring location whenever possible.
For example:
Zabbix Server
|
|
Zabbix Proxy
|
+————+——–+
| |
haproxy01 haproxy02
The monitoring system should not rely on one of the two HAProxy servers to tell you that both HAProxy servers have failed.
5. A more reliable approach
For a production installation, I would also consider using the machine-readable output from
crm_mon
For example:
# crm_mon -1 -X
The
-X
option produces XML output. That makes it possible to parse the cluster state without depending on the formatting of the normal human-readable output.
This matters because a script based on simplicity: e.g. – grep, awk, sed
can easily break after a Pacemaker update if the output format changes.
The general idea remains the same:
Pacemaker
|
crm_mon
|
status parser
|
Zabbix item
|
Zabbix trigger
The only difference is that the parser is working with structured data instead of text intended for a person to read.
6. Don't stop at the cluster state
Checking whether the nodes are online is a good start, but it is not the same as checking whether the application is actually working.
For example, you could have:
haproxy01 Online
haproxy02 Online
HAProxy Running
VIP Available
and still have a problem with the application behind HAProxy.
For that reason, I normally split the monitoring into several checks:
+ Is the server reachable?
+ Is the Pacemaker node online?
+ Is the HAProxy resource running?
+ Is the HAProxy service running?
+ Is the virtual IP available?
+ Can a client actually connect through HAProxy?
+ Is the application behind HAProxy responding?
The last check is often the most useful one because it tests the service from the user's point of view.
7. Testing the setup
Before relying on the Zabbix alert, test the different failure scenarios.
Start with both nodes online:
/usr/local/bin/check_haproxy_cluster.sh
Expected:
0
Next, take one node offline using your normal cluster maintenance procedure.
The check should return:
1
Zabbix should report that one HAProxy node is down, but it should not report a complete HAProxy outage.
Finally, test the situation where neither node is available.
The check should return:
2
At that point Zabbix should generate:
Both HAProxy cluster nodes are DOWN
After testing, verify that the problem clears automatically when the cluster returns to its normal state.
Monitoring a 2-Node HAProxy Cluster with Zabbix with more simplistic crm_mon Userparameter (Approach 2)
When you run a high-availability (HA) cluster with Pacemaker, Corosync, and HAProxy, monitoring can be tricky. If you set up standard alerts, a single node rebooting for maintenance can trigger a false "service down" alarm, even though the other node handled the failover perfectly.
To prevent alert fatigue, you need Zabbix to look at the cluster as a single unit and only fire a critical page when both HAProxy nodes go down at the same time.
Here is a practical guide to setting this up using
crm_mon
1. Collect Cluster Data via Zabbix Agent
Instead of parsing messy text outputs, we can make
crm_mon
output clean XML data, then coun
Run these steps on both cluster nodes:
a. Create a new Zabbix agent configuration file:
vim /etc/zabbix/zabbix_agentd.d/haproxy_cluster.conf
b. Paste the following
UserParameter
line.
This runs
crm_mon
grabs the XML structure, and returns the total count of active HAProxy resources:
UserParameter=crm.haproxy.active,sudo crm_mon -1 --as-xml | grep -c 'resource id="haproxy".*active="true"'
(Note: Change
"haproxy"
to match the exact resource ID used in your Pacemaker setup).
c. Give the
zabbix
user permission to run
crm_mon
without a password prompt.
Open the sudoers file:
# visudo
Add this line at the bottom:
zabbix ALL=(ALL) NOPASSWD: /usr/sbin/crm_mon
d. Restart your Zabbix agent:
# systemctl restart zabbix-agent
2. Set Up the Zabbix Item
Open your Zabbix Web UI. Create this item inside a template and apply it to your cluster nodes:
Name: HAProxy Active Nodes in Cluster
Type: Zabbix agent
Key:
crm.haproxy.active
Type of information: Numeric (unsigned)
Update interval: 1m
What these values mean:
- 2 = Healthy. Both nodes are up and active.
- 1 = Degraded. One node died, but traffic is still flowing through the survivor.
- 0 = Outage. Both nodes are down; the service is dead.
3. Create the Smart Triggers
Now we can create two different alerts so your team knows the difference between a minor issue and an emergency.
The "All Hands on Deck" Alert (Both Nodes Down)
This fires only when the active count drops to zero.
Name: HAProxy Cluster Outage: Both nodes are DOWN!
Severity: Disaster
Expression:
last(/Template HAProxy Cluster/crm.haproxy.active)=0
The Warning Alert (One Node Down)
This lets you know you lost redundancy, but the site is still up. [1]
Name: HAProxy Cluster Degraded: One node is DOWN
Severity: Average
Expression:
last(/Template HAProxy Cluster/crm.haproxy.active)=1
Alternative: The Quick and Dirty "AND" Trigger
If you do not want to deal with crm_mon or sudoers permissions, you can achieve the same result using Zabbix's built-in network checks and a logical and condition.
If you monitor the HAProxy port (e.g., 8080 or 443) on node1
and node2 individually, you can chain them together in a single trigger expression:
last(/node1/net.tcp.service[http,,443])=0 and last(/node2/net.tcp.service[http,,8080])=0
With this logic, Zabbix stays quiet during routine single-node reboots and only alerts your team if both ports become unresponsive simultaneously.
Final thoughts
For a two-node HAProxy cluster, checking only
# systemctl status haproxy
does not give you the whole picture.
crm_mon
gives Zabbix access to the Pacemaker view of the cluster, and a small monitoring script can turn that information into a simple status value.
The basic idea is straightforward:
0 = both nodes OK
1 = one node down
2 = both nodes down
From there, Zabbix can handle the alerting.
The most important part is not the Zabbix trigger itself. It is making sure that the monitoring check is independent of the servers it is supposed to monitor. If both HAProxy nodes can fail, the check should ideally run from a Zabbix Server, Zabbix Proxy, or another independent monitoring host.
For a small environment, the simple
crm_mon parser is usually enough.
For a production environment, I would use the XML output from crm_mon
and explicitly monitor the two expected cluster nodes and the HAProxy resource as separate checks.
