Fault-tolerant cluster with haproxy and keepalived. Also backend sync with glusterfs

Case: build fault tolerant souliton using linux servers and sync data between cluster members with glusterfs.

solution: keepalived + HAProxy for VRRP and glusterfs to sync data.

Install keepalived + haproxy

apt-get update
apt-get upgrade
apt install haproxy keepalived

keepalived.conf configuration example

vrrp_script chk_haproxy {
    script "/etc/keepalived/check_haproxy.sh"
    interval 2
    timeout 3
    fall 2
    rise 2
}

vrrp_instance VI_1 {
    interface ens192
    state MASTER
    virtual_router_id 51          # A unique number [1-255] for this VRRP instance
    priority 100                  # 100 for master, 50 for backup
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass mysecretpass   # A password for authentication, should be the same on all servers
    }
    virtual_ipaddress {
    10.10.10.10                 # The virtual IP address shared between master and backup
    10.10.10.11
    10.10.10.12
    }
    track_script {
        chk_haproxy
    }
}

vrrp_instance VI_2 {
    interface ens224
    state MASTER
    virtual_router_id 52          # A unique number [1-255] for this VRRP instance
    priority 100                  # 100 for master, 50 for backup
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass mysecretpass   # A password for authentication, should be the same on all servers
    }
    virtual_ipaddress {
    10.10.20.10                 # The virtual IP address shared between master and backup
    10.10.20.11
    10.10.20.12
    }
    track_script {
        chk_haproxy
    }

Script to check if haproxy is alive – /etc/keepalived/check_haproxy.sh:

#!/bin/bash
# Check if HAProxy is running

if systemctl is-active --quiet haproxy; then
    exit 0
else
    exit 1

Sample of haproxy.cfg:

global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
    log global
    mode http
    option httplog
    option dontlognull
    retries 3
    option redispatch
    timeout connect 5000ms
    timeout client  50000ms
    timeout server  50000ms
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

# Stats server fof Haproxy
listen stats_and_disclaimer
    bind 0.0.0.0:80
    stats enable
    stats uri /haproxy?stats
    stats realm "Strictly Private"
    stats auth A_Username:YourPassword
    stats auth Another_User:passwd
    mode tcp
    option tcplog
    server lamp1 172.24.1.106:80 check
    server lamp2 172.24.1.107:80 check backup

listen some_server
    bind 0.0.0.0:123
    mode tcp
    option tcplog
    server lamp1 10.11.11.11:123 check
    server lamp2 10.11.11.12:123 check backup

Done.

Now let`s configure glusterfs to sync data between servers.

install Glusterfs

apt install glusterfs-server -y
systemctl enable glusterd --now
systemctl start glusterd

Prepare bricks on both servers. Ensure the directories are empty, as GlusterFS bricks must be initialized with empty directories

mkdir -p /data/brick1/etc_haproxy
chown -R gluster:gluster /data/brick1

On server1, add server2 as a peer and vice-versa

gluster peer probe server2.your_domain.ca

Verify the peer connection

gluster peer status

Create a replicated volume for /etc/haproxy

gluster volume create haproxy_vol replica 2 server1.your_domain.ca:/data/brick1/etc_haproxy server2.ypur_domain.ca:/data/brick1/etc_haproxy

Start the volume

gluster volume start haproxy_vol

Mount the /etc/haproxy volume (same on both servers) & add to fstab for persistence

mount -t glusterfs server1.your_domain.ca:/haproxy_vol /etc/haproxy
echo "server1.your_domain.ca,server2.your_domain.ca:/haproxy_vol /etc/haproxy glusterfs defaults,_netdev 0 0" | sudo tee -a /etc/fstab

Verify sync

Create a test file in /opt on server1

echo "Hello from server1" > /etc/haproxy/testfile.txt

Check if file appears on server2

Add firewall rules

firewall-cmd --add-service=glusterfs --permanent
firewall-cmd --reload

Allow specific IPs or subnets when creating the volume

gluster volume set haproxy_vol auth.allow 11.11.11.0/24
gluster volume stop opt_vol
gluster volume start opt_vol

Check volume current state.

gluster volume info opt_vol

enjoy.