Veteran Member
Posts: 4,128
Registered: ‎05-15-2014
Kudos: 1546
Solutions: 281

Default route lost

[ Edited ]

We have two ER-X at clients sites, both running 1.9.7+hotfix3. Both routers are on PPPoE WAN and neither has anything advanced configured (no Load-balancing or any other funky stuff).

Both routers are reporting back to UNMS. The uptime on both routers is since hotfix 3 was released.

 

Couple days ago I noticed both routers were reporting disconnected in UNMS. However I was able to log in and found out that default route 0.0.0.0/0 was missing ...was simply not there, it disappeared. Local traffic was failing, pass through traffic was routed properly.

PPPoE default-route is set to auto. Reboot solved the issue on both routers.

 

Not sure what the issue was, what caused it but I'm concerned. I'm not able to replicate it but as I said it happened at two routers at two clients independently. Both on PPPoE. The only thing that I'm aware of was UNMS upgrade to 0.10.0 day before ...but I'm not saying UNMS caused it.

 

@UBNT-afomins @UBNT-Fenng @UBNT-sandisn @UBNT-Jindrich

 

 

Ubiquiti Employee
Posts: 2,910
Registered: ‎05-08-2017
Kudos: 520
Solutions: 416

Re: Default route lost

I have not (yet) been able to replicate this issue. My test setup:

 

ER-X / 1.9.7 hf3

UNMS 0.9.0 

 

 

Before upgrade to 0.10.0:

2.png

 

After upgrade:

 

1.png

 

show ip route
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       > - selected route, * - FIB route, p - stale info
IP Route Table for VRF "default"
K    *> 0.0.0.0/0 [0/0] via pppoe0
C    *> 10.255.253.0/32 is directly connected, pppoe0
C    *> 127.0.0.0/8 is directly connected, lo
C    *> 172.16.1.0/24 is directly connected, eth1
C    *> 172.30.1.0/24 is directly connected, eth4
C    *> 203.0.113.1/32 is directly connected, pppoe0

I find it strange that the kernel route would just disappear. The only way I found to reproduce that is to change the default-route statement to none instead of auto. I have also tried to remove the device from UNMS and readd it to try and break it, but it seems to keep the kernel route in the table in my setup.

 

Ben

 


 

Ben Pin | Ubiquiti Support

Veteran Member
Posts: 4,128
Registered: ‎05-15-2014
Kudos: 1546
Solutions: 281

Re: Default route lost

I'm not able to replicate it either, l'm not sure what caused it either. I'm not saying it is UNMS related. 

 

But it happened on two unrelated routers. 

 

Thanks for looking into this, I am keeping my eyes on this as well. 

Ubiquiti Employee
Posts: 1,256
Registered: ‎07-20-2015
Kudos: 1548
Solutions: 82

Re: Default route lost

Please check /var/core directory, do you see any coredumps there?

Veteran Member
Posts: 4,128
Registered: ‎05-15-2014
Kudos: 1546
Solutions: 281

Re: Default route lost

Veteran Member
Posts: 7,968
Registered: ‎03-24-2016
Kudos: 2077
Solutions: 912

Re: Default route lost

Does /var/core survive your reboot?

Ubiquiti Employee
Posts: 1,256
Registered: ‎07-20-2015
Kudos: 1548
Solutions: 82

Re: Default route lost


@16again wrote:

Does /var/core survive your reboot?


Yes, it is unionfs filesystem in flash storage:

root@skyler# df /var/core
Filesystem           1K-blocks      Used Available Use% Mounted on
unionfs                3508480    330536   2999720  10% /
Veteran Member
Posts: 4,128
Registered: ‎05-15-2014
Kudos: 1546
Solutions: 281

Re: Default route lost

[ Edited ]

@UBNT-afomins@UBNT-benpin

It happened again, this time on another router with dual wan. Primary wan on pppoe. Looks like pppoe disconnected and re-connected. However system kept my fail-over link default route even tough I have pppoe default-route auto.

 

Here is the pppoe log ...too bad it desn't have time stamps. I've highlighted the suspicious entry.

root@router-brano:/var/log/vyatta# cat ppp_pppoe0.log 
Sun Oct  1 21:41:48 EDT 2017: PPP interface pppoe0 created
Sun Oct  1 21:41:50 EDT 2017: Stopping PPP daemon for pppoe0
Sun Oct  1 21:41:50 EDT 2017: Starting PPP daemon for pppoe0
Connected to 00:25:ba:7f:31:71 via interface eth0
using channel 1
Using interface ppp0
Connect: ppp0 <--> eth0
sent [LCP ConfReq id=0x1 <mru 1492> <magic 0xcacec045>]
rcvd [LCP ConfAck id=0x1 <mru 1492> <magic 0xcacec045>]
sent [LCP ConfReq id=0x1 <mru 1492> <magic 0xcacec045>]
rcvd [LCP ConfAck id=0x1 <mru 1492> <magic 0xcacec045>]
rcvd [LCP ConfReq id=0x82 <mru 1492> <auth pap> <magic 0x4fb6fd63>]
lcp_reqci: returning CONFACK.
sent [LCP ConfAck id=0x82 <mru 1492> <auth pap> <magic 0x4fb6fd63>]
sent [LCP EchoReq id=0x0 magic=0xcacec045]
sent [PAP AuthReq id=0x1 user="brano@teksavvy.com" password=<hidden>]
rcvd [LCP EchoRep id=0x0 magic=0x4fb6fd63]
sent [PAP AuthReq id=0x2 user="brano@teksavvy.com" password=<hidden>]
sent [LCP EchoReq id=0x1 magic=0xcacec045]
rcvd [LCP EchoRep id=0x1 magic=0x4fb6fd63]
sent [PAP AuthReq id=0x3 user="brano@teksavvy.com" password=<hidden>]
sent [PAP AuthReq id=0x4 user="brano@teksavvy.com" password=<hidden>]
sent [LCP EchoReq id=0x2 magic=0xcacec045]
rcvd [LCP EchoRep id=0x2 magic=0x4fb6fd63]
sent [PAP AuthReq id=0x5 user="brano@teksavvy.com" password=<hidden>]
sent [LCP EchoReq id=0x3 magic=0xcacec045]
sent [PAP AuthReq id=0x6 user="brano@teksavvy.com" password=<hidden>]
rcvd [LCP EchoRep id=0x3 magic=0x4fb6fd63]
sent [PAP AuthReq id=0x7 user="brano@teksavvy.com" password=<hidden>]
sent [LCP EchoReq id=0x4 magic=0xcacec045]
sent [PAP AuthReq id=0x8 user="brano@teksavvy.com" password=<hidden>]
rcvd [LCP EchoRep id=0x4 magic=0x4fb6fd63]
sent [PAP AuthReq id=0x9 user="brano@teksavvy.com" password=<hidden>]
sent [LCP EchoReq id=0x5 magic=0xcacec045]
rcvd [LCP EchoRep id=0x5 magic=0x4fb6fd63]
sent [PAP AuthReq id=0xa user="brano@teksavvy.com" password=<hidden>]
No response to PAP authenticate-requests
sent [LCP TermReq id=0x2 "Failed to authenticate ourselves to peer"]
rcvd [LCP TermAck id=0x2]
Connection terminated: no multilink.
Connected to 00:25:ba:7f:31:71 via interface eth0
using channel 2
Using interface ppp0
Connect: ppp0 <--> eth0
sent [LCP ConfReq id=0x3 <mru 1492> <magic 0x7857f789>]
rcvd [LCP ConfAck id=0x3 <mru 1492> <magic 0x7857f789>]
sent [LCP ConfReq id=0x3 <mru 1492> <magic 0x7857f789>]
rcvd [LCP ConfAck id=0x3 <mru 1492> <magic 0x7857f789>]
rcvd [LCP ConfReq id=0xb2 <mru 1492> <auth pap> <magic 0x444baf13>]
lcp_reqci: returning CONFACK.
sent [LCP ConfAck id=0xb2 <mru 1492> <auth pap> <magic 0x444baf13>]
sent [LCP EchoReq id=0x0 magic=0x7857f789]
sent [PAP AuthReq id=0xb user="brano@teksavvy.com" password=<hidden>]
rcvd [LCP EchoRep id=0x0 magic=0x444baf13]
rcvd [PAP AuthAck id=0xb ""]
PAP authentication succeeded
peer from calling number 00:25:BA:7F:31:71 authorized
sent [IPCP ConfReq id=0x1 <addr 0.0.0.0>]
rcvd [IPCP ConfNak id=0x1 <addr 24.140.235.21>]
sent [IPCP ConfReq id=0x2 <addr 24.140.235.21>]
rcvd [IPCP ConfAck id=0x2 <addr 24.140.235.21>]
rcvd [IPCP ConfReq id=0x58 <addr 206.248.155.244>]
ipcp: returning Configure-ACK
sent [IPCP ConfAck id=0x58 <addr 206.248.155.244>]
ipcp: up
Script /etc/ppp/ip-pre-up started (pid 3014)
Script /etc/ppp/ip-pre-up finished (pid 3014), status = 0x0
not replacing existing default route via 172.16.16.3
local IP address 24.140.235.21 remote IP address 206.248.155.244 Script /etc/ppp/ip-up started (pid 3107) Script /etc/ppp/ip-up finished (pid 3107), status = 0x0

PPPoE shows connected and has IP but load-balance watchdog shows failed status on pppoe.

 

I'm suspecting a bug in the load-balance lb-local-metric-change engine when interface is pppoe.

 

Ubiquiti Employee
Posts: 1,256
Registered: ‎07-20-2015
Kudos: 1548
Solutions: 82

Re: Default route lost


@BranoB wrote:

@UBNT-afomins@UBNT-benpin

It happened again, this time on another router with dual wan.

 

I'm suspecting a bug in the load-balance lb-local-metric-change engine when interface is pppoe. 


I don't think it is LB related because previously you mentioned that same issue happened without LB.

 

Maybe you can try removing "default-route auto" on pppoe interface and setting static deafult route via pppoe0 interface instead:

set interfaces ethernet eth0 pppoe 0 default-route none
set protocols static interface-route 0.0.0.0/0 next-hop-interface pppoe0

 

Established Member
Posts: 787
Registered: ‎01-27-2015
Kudos: 303
Solutions: 6

Re: Default route lost

[ Edited ]

Any update on this issue?

 

I've had this occur to me a number of occasions, most recently just last night. 

 

The router has a very simple config and uses OSPF for all routing information including its default route. The router that hiccupped last evening is at the tower-end of a PTP link to the data center.

 

Since this happened in the middle of the evening Netflix peak I didn't take time to really investigate and just rebooted the router.

 

Unfortunately, the last few times this happened it was during the same evening period when everyone is streaming. 

 

The router is running v1.9.1.1, with no PPPoE or UNMS.

 

 

interfaces {
    ethernet eth0 {
        address 10.1.1.4/24
        description "SW Dom A: TW01-PTP-SW-01:eth7"
        duplex auto
        ip {
            ospf {
                dead-interval 40
                hello-interval 10
                priority 1
                retransmit-interval 5
                transmit-delay 1
            }
        }
        speed auto
    }
    ethernet eth1 {
        address 10.1.2.4/24
        description "SW Dom B: TW01-PTP-SW-02:eth7"
        duplex auto
        ip {
            ospf {
                dead-interval 40
                hello-interval 10
                priority 1
                retransmit-interval 5
                transmit-delay 1
            }
        }
        speed auto
    }
    ethernet eth2 {
        duplex auto
        speed auto
    }
    ethernet eth3 {
        address 10.0.10.20/24
        description PHBH01-TW01-end-pt
        duplex auto
        ip {
            ospf {
                dead-interval 40
                hello-interval 10
                priority 1
                retransmit-interval 5
                transmit-delay 1
            }
        }
        speed auto
    }
    ethernet eth4 {
        duplex auto
        speed auto
    }
    loopback lo {
    }
    switch switch0 {
        mtu 1500
    }
}
protocols {
    ospf {
        area 0.0.0.0 {
            area-type {
                normal
            }
            network 10.1.1.0/24
            network 10.1.2.0/24
            network 10.0.10.0/24
        }
        log-adjacency-changes {
            detail
        }
        parameters {
            abr-type cisco
            router-id 10.1.1.4
        }
        redistribute {
            connected {
                metric-type 2
            }
            static {
                metric-type 2
            }
        }
    }
}
service {
    gui {
        http-port 80
        https-port 443
    }
    snmp {
        community public {
            authorization ro
        }
        location PWTW01
    }
    ssh {
        port 22
        protocol-version v2
    }
}
system {
    host-name TW01-PTP-RTR-01
    login {
[snip]
    }
    name-server 8.8.8.8
    name-server 8.8.4.4
    ntp {
        server 0.ubnt.pool.ntp.org {
        }
        server 1.ubnt.pool.ntp.org {
        }
        server 2.ubnt.pool.ntp.org {
        }
        server 3.ubnt.pool.ntp.org {
        }
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone UTC
}

 

 

 

Veteran Member
Posts: 4,128
Registered: ‎05-15-2014
Kudos: 1546
Solutions: 281

Re: Default route lost

I've seen this only on pppoe WAN devices with default-route auto. Since switching to default-route force I've not seen this again.

Established Member
Posts: 787
Registered: ‎01-27-2015
Kudos: 303
Solutions: 6

Re: Default route lost

Thanks @BranoB. Was afraid your case was PPPoE related.

 

Since this has happened a few times, I'm fairly confident something bogus is going on, just no clue what.

 

 

 

 

Veteran Member
Posts: 4,128
Registered: ‎05-15-2014
Kudos: 1546
Solutions: 281

Re: Default route lost

Why don't you open a new thread and post your config. As well as output from show interfaces and show ip route.

Highlighted
Ubiquiti Employee
Posts: 1,256
Registered: ‎07-20-2015
Kudos: 1548
Solutions: 82

Re: Default route lost

@Solideco:

> The router has a very simple config and uses OSPF for all routing information including its default route. 

> The router is running v1.9.1.1, with no PPPoE or UNMS.

Looks like your issue is different. Please create new topic and show output of following commands:

ps ax|grep ospf
cat /var/log/messages|grep ospf
show interfaces
ip route