ER-X Load balancing with failover not working [ Edited ]

[heychirag]

Hi,

I was trying to set up a failover load balancing config on my ER-X (with EdgeOS v1.10.8). I have two PPPoE internet connections and so I used the wizard to set up my ER-X in failover load balancing mode. I wanted to test out what would actually happen if my primary connection goes off. So I tried to simulate this scenario by unplugging the cable. Surprisingly, the internet wasn't working anymore. The router failed to switch to my failover internet connection.

Here are successive outputs of the "show load-balance watchdog" command taken right after disconnecting my primary internet connection:

ubnt@ubnt:~$ show load-balance watchdog
Group G
  pppoe3
  status: Running
  failover-only mode
  pings: 7
  fails: 1
  run fails: 1/3
  route drops: 2
  ping gateway: ping.ubnt.com - REACHABLE
  last route drop   : Sun Feb 10 21:15:39 2019
  last route recover: Sun Feb 10 21:23:16 2019

  pppoe4
  status: Running
  pings: 7
  fails: 1
  run fails: 1/3
  route drops: 2
  ping gateway: ping.ubnt.com - REACHABLE
  last route drop   : Sun Feb 10 21:15:32 2019
  last route recover: Sun Feb 10 21:23:13 2019

ubnt@ubnt:~$ show load-balance watchdog
Group G
  pppoe3
  status: Running
  failover-only mode
  pings: 7
  fails: 1
  run fails: 1/3
  route drops: 2
  ping gateway: ping.ubnt.com - REACHABLE
  last route drop   : Sun Feb 10 21:15:39 2019
  last route recover: Sun Feb 10 21:23:16 2019

  pppoe4
  status: Failed
  pings: 8
  fails: 2
  run fails: 4/3
  route drops: 3
  ping gateway: ping.ubnt.com - DOWN
  last route drop   : Sun Feb 10 21:15:32 2019
  last route recover: Sun Feb 10 21:23:13 2019

ubnt@ubnt:~$ show load-balance watchdog
Group G
  pppoe3
  status: Failed
  failover-only mode
  pings: 9
  fails: 3
  run fails: 3/3
  route drops: 3
  ping gateway: ping.ubnt.com - DOWN
  last route drop   : Sun Feb 10 21:27:12 2019
  last route recover: Sun Feb 10 21:23:16 2019

  pppoe4
  status: Waiting on recovery (0/3)
  pings: 8
  fails: 2
  run fails: 4/3
  route drops: 3
  ping gateway: ping.ubnt.com - DOWN
  last route drop   : Sun Feb 10 21:27:01 2019
  last route recover: Sun Feb 10 21:23:13 2019

ubnt@ubnt:~$ show load-balance watchdog
Group G
  pppoe3
  status: Waiting on recovery (0/3)
  failover-only mode
  pings: 9
  fails: 3
  run fails: 3/3
  route drops: 3
  ping gateway: ping.ubnt.com - DOWN
  last route drop   : Sun Feb 10 21:27:12 2019
  last route recover: Sun Feb 10 21:23:16 2019

  pppoe4
  status: Waiting on recovery (0/3)
  pings: 8
  fails: 2
  run fails: 4/3
  route drops: 3
  ping gateway: ping.ubnt.com - DOWN
  last route drop   : Sun Feb 10 21:27:01 2019
  last route recover: Sun Feb 10 21:23:13 2019

I have removed redundant outputs as it took a while for the watchdog to realize that my primary connection is down. A delay in realizing this is also something I am not happy with. But what's more even infuriating is the fact that WHY IS watchdog saying that my secondary internet is Down when IT IS ACTUALLY WORKING FINE? As soon as I plug my primary cable in, both of them start working. This doesn't make sense to me at all! Here again, watchdog takes years to realize that my connection is back up while the dashboard is really quick and shows my connection to be plugged in or not almost instantaneously. Why can't watchdog be as fast as the dashboard? 

Sorry for getting sidetracked from the real problem. I will really appreciate if someone could tell me WHY IS FAILOVER LOAD BALANCING NOT WORKING on my ER-X?!

Kind regards,

Chirag

[BranoB · Accepted solution]

Regarding 1) above

set load-balance group G interface pppoe3 route-test type ping target 8.8.8.8
set load-balance group G interface pppoe4 route-test type ping target 8.8.8.8

Regarding 2) above

set interfaces ethernet eth3 pppoe 3 default-route force
set interfaces ethernet eth4 pppoe 4 default-route force

[redfive]

Are you using custom DNS servers (eg. 8.8.8.8, 8.8.4.4), on the ER-X. as system name-server ?

Cheers,

jonatha

[heychirag]

I am actually using a Pi Hole that handles my DNS and DHCP.

[redfive]

I mean, even the edgerouter itself, is using the Pi Hole, as DNS server ? Otherwise, something like

Spoiler

configure
set interfaces ethernet ethX pppoe 3 name-server none
set interfaces ethernet ethY pppoe 4 name-server none
set system name-server x.x.y.y
set system name-server x.x.z.z
commit

[heychirag]

I tried the above config but no luck.

[redfive]

Can you post the config ?

[heychirag]

ubnt@ubnt:~$ show configuration
firewall {
    all-ping enable
    broadcast-ping disable
    group {
        network-group PRIVATE_NETS {
            network 192.168.0.0/16
            network 172.16.0.0/12
            network 10.0.0.0/8
        }
    }
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians disable
    modify balance {
        rule 10 {
            action modify
            description "do NOT load balance lan to lan"
            destination {
                group {
                    network-group PRIVATE_NETS
                }
            }
            modify {
                table main
            }
        }
        rule 50 {
            action modify
            description "do NOT load balance destination public address"
            destination {
                group {
                    address-group ADDRv4_pppoe3
ubnt@ubnt:~$ show configuration
firewall {
    all-ping enable
    broadcast-ping disable
    group {
        network-group PRIVATE_NETS {
            network 192.168.0.0/16
            network 172.16.0.0/12
            network 10.0.0.0/8
        }
    }
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians disable
    modify balance {
        rule 10 {
            action modify
            description "do NOT load balance lan to lan"
            destination {
                group {
                    network-group PRIVATE_NETS
                }
            }
            modify {
                table main
            }
        }
        rule 50 {
            action modify
            description "do NOT load balance destination public address"
            destination {
                group {
                    address-group ADDRv4_pppoe3
                }
            }
            modify {
                table main
            }
        }
        rule 60 {
            action modify
            description "do NOT load balance destination public address"
            destination {
                group {
                    address-group ADDRv4_pppoe4
                }
            }
            modify {
                table main
            }
        }
        rule 70 {
            action modify
            modify {
                lb-group G
            }
        }
    }
    name WAN_IN {
        default-action drop
        description "WAN to internal"
        rule 10 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
    }
    name WAN_LOCAL {
        default-action drop
        description "WAN to router"
        rule 10 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
    }
    options {
        mss-clamp {
            mss 1412
        }
    }
    receive-redirects disable
    send-redirects enable
    source-validation disable
    syn-cookies enable
}
interfaces {
    ethernet eth0 {
        duplex auto
        speed auto
    }
    ethernet eth1 {
        address dhcp
        description eth1-pppoe
        duplex auto
        speed auto
    }
    ethernet eth2 {
        duplex auto
        speed auto
    }
    ethernet eth3 {
        description Excitel
        duplex auto
        pppoe 3 {
            default-route auto
            firewall {
                in {
                    name WAN_IN
                }
                local {
                    name WAN_LOCAL
                }
            }
            mtu 1492
            name-server auto
            password ****************
            user-id chiragarora182
        }
        speed auto
    }
    ethernet eth4 {
        description ACT
        duplex auto
        poe {
            output off
        }
        pppoe 4 {
            default-route auto
            firewall {
                in {
                    name WAN_IN
                }
                local {
                    name WAN_LOCAL
                }
            }
            mtu 1492
            name-server auto
            password ****************
            user-id 110288511241
        }
        speed auto
    }
    loopback lo {
    }
    switch switch0 {
        address 10.0.0.1/24
        description Local
        firewall {
            in {
                modify balance
            }
        }
        mtu 1500
        switch-port {
            interface eth0 {
            }
            interface eth2 {
            }
            vlan-aware disable
        }
    }
}
load-balance {
    group G {
        interface pppoe3 {
            failover-only
        }
        interface pppoe4 {
        }
        lb-local disable
        lb-local-metric-change disable
    }
}
service {
    dhcp-server {
        disabled false
        hostfile-update disable
        shared-network-name LAN {
            authoritative enable
            disable
            subnet 10.0.0.0/24 {
                default-router 10.0.0.1
                dns-server 10.0.0.1
                lease 86400
                start 10.0.0.38 {
                    stop 10.0.0.243
                }
            }
        }
        static-arp disable
        use-dnsmasq disable
    }
    dns {
        forwarding {
            cache-size 150
            listen-on switch0
        }
    }
    gui {
        http-port 80
        https-port 443
        older-ciphers enable
    }
    nat {
        rule 5006 {
            description "masquerade for WAN 2"
            outbound-interface pppoe3
            type masquerade
        }
        rule 5008 {
            description "masquerade for WAN"
            outbound-interface pppoe4
            type masquerade
        }
    }
    pppoe-server {
        authentication {
            local-users {
                username paranoid {
                    password ****************
                    static-ip 10.0.1.2
                }
            }
            mode local
        }
        client-ip-pool {
            start 10.0.1.2
            stop 10.0.1.254
        }
        dns-servers {
            server-1 10.0.0.53
            server-2 10.0.0.53
        }
        interface eth1
    }
    ssh {
        port 22
        protocol-version v2
    }
    unms {
        disable
    }
}
system {
    conntrack {
        expect-table-size 4096
        hash-size 4096
        table-size 32768
        tcp {
            half-open-connections 512
            loose enable
            max-retrans 3
        }
    }
    host-name ubnt
    login {
        user ubnt {
            authentication {
                encrypted-password ****************
            }
            level admin
        }
    }
    name-server 8.8.8.8
    name-server 8.8.4.4
    ntp {
        server 0.ubnt.pool.ntp.org {
        }
        server 1.ubnt.pool.ntp.org {
        }
        server 2.ubnt.pool.ntp.org {
        }
        server 3.ubnt.pool.ntp.org {
        }
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone UTC
}

[BranoB]

Few things

1) Use IP as health-check target instead of FQDN. Better yet, use multiple IPs for health-check by use of custom pinger.

2) Check your default gateway for each PPPoE. With PPPoE interfaces set the default-route to force instead of default default-route auto.

3) Post your full config for verification. When you post it it'll be easier to give you specific directions how to perform the two above mentioned steps.

[heychirag]

Please tell me how to post the "Full" config?

[BranoB · Accepted solution]

Regarding 1) above

set load-balance group G interface pppoe3 route-test type ping target 8.8.8.8
set load-balance group G interface pppoe4 route-test type ping target 8.8.8.8

Regarding 2) above

set interfaces ethernet eth3 pppoe 3 default-route force
set interfaces ethernet eth4 pppoe 4 default-route force

[BranoB]

---

@heychirag wrote:

Please tell me how to post the "Full" config?

---

show configuration | no-more

[redfive]

Can you try

Spoiler

configure
set interfaces ethernet eth3 pppoe 3 default-route none
set interfaces ethernet eth4 pppoe 4 default-route none
set interfaces ethernet eth3 pppoe 3 name-server none
set interfaces ethernet eth4 pppoe 4 name-server none
set protocols static interface-route 0.0.0.0/0 next-hop-interface pppoe3
set protocols static interface-route 0.0.0.0/0 next-hop-interface pppoe4
set load-balance group G interface pppoe3 route-test type ping target 8.8.8.8
set load-balance group G interface pppoe4 route-test type ping target 8.8.8.8
commit;save;exit
reboot now

[heychirag]

@BranoB It worked! Should I still post the full config?

@redfive Are these configurations better to what @BranoB suggested? I would love to improve the watchdog response time in making a switch.

Thanks a lot for your help!!