Highlighted
New Member
Posts: 10
Registered: 4 weeks ago
Kudos: 1

EdgeRouter 4 looses link on SFP interface and ARP stopped working

[ Edited ]

Hi,

 

I have a strange issue after short uptime with Edgerouter 4 (Running latest and v1.10.8 / v2.0.0).

 

When I boot up the ER-4 device it's starting without any errors... BGP is establishing, OSPF runs well and overall device is working so far. After waiting for about 20 minutes ER-4 looses link on eth3 / SFP port, other ports are remaining "up" and I'am also able to receive packets (tcpdump) on eth0-eth2 and ARP stopped working on all interfaces. A soft reboot is _NOT_ resolving this issue. The current state also exists after soft-reboot and also if I reboot into a different

EdgeOS version i.e. v1.10.8. Removing the SFP from the interface does not clear the issue. Also tried to swap SFP modules without success. Only a hard reboot by removing the power solves the issue. In the system logs of the device I'am not able to see any error that happened between. Before this behaviour started the device was running fine for about 2 weeks. Now the issue is happening every ~ 20 minutes after a cold boot.

 

Running configuration: https://pastebin.com/fSehKWFg

 

I also have a different device (EdgeRouter 6P) running which is the neighbour of ER-4 with nearly exact configuration which is not affected from this problem.

 

Does anyone have any suggestion how to debug this?

 

Regards,

Denis

Member
Posts: 275
Registered: ‎06-13-2018
Kudos: 58
Solutions: 7

Re: EdgeRouter 4 looses link on SFP interface and ARP stopped working

Hi Denis - you tried another SFP of the same make/model or a different one? What make/model are you using? Have you tried 1.10.9? If you move the eth3 configuration to another port do you still have the ARP problem?
New Member
Posts: 10
Registered: 4 weeks ago
Kudos: 1

Re: EdgeRouter 4 looses link on SFP interface and ARP stopped working

Hi,

 

Did all the suggested work but did not solve the problem.... The problem also exists running latest 1.10.9.

I will try a factory reset in the upcoming days. Currently I configured all IP addresses on a bridge and changed configuration a little bit. With the different configuration it's working now for hours. I will continue monitoring it.

New Member
Posts: 10
Registered: 4 weeks ago
Kudos: 1

Re: EdgeRouter 4 looses link on SFP interface and ARP stopped working

[ Edited ]

Short Update: Problem also exists if addresses are configured on bridges and the physical interfaces are bridged together.

Now the time is getting shorter when SFP port stops working. It's now about after 2-4 minutes after boot up and we are loosing link (Link down). ARP remains operating on eth0 - eth2. After another about 10minutes ARP stopped working on eth0 and eth2 (connected to ER-6). Eth0 is connected to a Client PC and Eth1 is also connected to a Client PC but on this port ARP remains operating. This is the current configuration on ER-4. If someone has any ideas how to debug this...

ubnt@edgerouter-4:~$ show configuration
firewall {
    all-ping enable
    broadcast-ping disable
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians enable
    receive-redirects disable
    send-redirects enable
    source-validation disable
    syn-cookies enable
}
interfaces {
    bridge br0 {
        address X.X.119.82/29
        aging 300
        bridged-conntrack disable
        hello-time 2
        max-age 20
        priority 32768
        promiscuous disable
        stp true
        vrrp {
            vrrp-group 1 {
                advertise-interval 1
                preempt true
                virtual-address X.X.119.81/29
            }
        }
    }
    ethernet eth0 {
        bridge-group {
            bridge br0
        }
        description "GK: Ubiquiti GmbH [1Gbit] {} (N123456) ::"
        duplex auto
        speed auto
    }
    ethernet eth1 {
        address X.X.101.81/29
        description "GK: Ubiquiti GmbH [1Gbit] {} (N123456) ::"
        duplex auto
        ip {
            source-validation strict
        }
        speed auto
    }
    ethernet eth2 {
        description "B: EDGEROUTER-6P [1Gbit] {direct} () ::"
        duplex auto
        speed auto
        vif 2 {
            address 10.0.0.1/30
        }
        vif 3 {
            bridge-group {
                bridge br0
                cost 65535
            }
        }
    }
    ethernet eth3 {
        description "B: GK-SWITCH [1Gbit] {IRGE1x/x} () ::"
        duplex auto
        speed auto
        vif 1009 {
            address X.X.3.2/24
        }
        vif 3456 {
            address X.X.106.13/31
            ip {
                source-validation disable
            }
        }
    }
}
policy {
}
protocols {
    bfd {
    }
    bgp 4200000001 {
        neighbor 10.0.0.2 {
            nexthop-self
            remote-as 4200000001
        }
        neighbor X.X.104.133 {
            ebgp-multihop 255
            fall-over {
                bfd {
                    multihop
                }
            }
            password ****************
            remote-as 15943
            soft-reconfiguration {
                inbound
            }
        }
        network X.X.73.144/28 {
        }
        network X.X.101.80/29 {
        }
        network X.X.119.80/29 {
        }
        parameters {
            router-id X.X.106.13
        }
        timers {
            holdtime 30
            keepalive 10
        }
    }
    ospf {
        area 0.0.0.0 {
            network 10.0.0.0/30
        }
        parameters {
            abr-type cisco
            router-id X.X.106.13
        }
    }
    static {
        route 10.0.0.0/8 {
            next-hop X.X.3.1 {
            }
        }
        route X.X.73.144/28 {
            next-hop X.X.119.84 {
            }
        }
        route X.X.104.122/32 {
            next-hop X.X.106.12 {
            }
        }
        route X.X.104.133/32 {
            next-hop X.X.106.12 {
            }
        }
        route 172.16.0.0/12 {
            next-hop X.X.3.1 {
            }
        }
    }
}
service {
    gui {
        http-port 80
        https-port 443
        listen-address X.X.3.2
        older-ciphers enable
    }
    lldp {
        interface all {
        }
    }
    snmp {
        community HIDDEN {
            authorization ro
        }
        contact "IP"
        listen-address X.X.3.2 {
            port 161
        }
        location POP1
    }
    ssh {
        disable-host-validation
        listen-address X.X.3.2
        listen-address 10.0.0.1
        port 22
        protocol-version v2
    }
}
system {
    host-name edgerouter-4
    login {
        banner {
            pre-login ""
        }
        user ubnt {
            authentication {
                encrypted-password ****************
            }
            level admin
        }
    }
    name-server X.X.104.220
    name-server X.X.104.250
    name-server X:X:fd00::cafe
    name-server X:X:fd00::affe
    ntp {
        server 0.ubnt.pool.ntp.org {
        }
        server 1.ubnt.pool.ntp.org {
        }
        server 2.ubnt.pool.ntp.org {
        }
        server 3.ubnt.pool.ntp.org {
        }
    }
    offload {
        hwnat disable
        ipv4 {
            forwarding disable
            vlan disable
        }
        ipv6 {
            forwarding disable
            vlan disable
        }
    }
    options {
        reboot-on-panic true
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
        host X.X.60.20 {
            facility all {
                level notice
            }
            facility auth {
                level info
            }
            facility daemon {
                level notice
            }
            facility protocols {
                level info
            }
        }
    }
    time-zone Europe/Berlin
}
traffic-control {
    smart-queue 500M-250M {
        download {
            ecn disable
            flows 1024
            fq-quantum 1514
            limit 10240
            rate 275mbit
        }
        upload {
            ecn disable
            flows 1024
            fq-quantum 1514
            limit 10240
            rate 550mbit
        }
        wan-interface eth1
    }
}
zone-policy {
    zone VR_DEFAULT {
        default-action drop
        from VR_INT_MGMT {
        }
        interface eth3.3456
        interface eth1
        interface eth0
        interface eth2.2
    }
    zone VR_INT_MGMT {
        default-action drop
        from VR_DEFAULT {
        }
        interface eth3.1009
    }
}
Member
Posts: 275
Registered: ‎06-13-2018
Kudos: 58
Solutions: 7

Re: EdgeRouter 4 looses link on SFP interface and ARP stopped working

You have a lot going on in your config. I understand that is the desired end-state and since that is not working, slim down the config until things work. Then build it back up one 'thing' at a time. Can you drop the vrrp, zone-policy, bridging etc and get a base config working? With so much going on as it is now it becomes difficult to find any 'smoking gun'.
New Member
Posts: 10
Registered: 4 weeks ago
Kudos: 1

Re: EdgeRouter 4 looses link on SFP interface and ARP stopped working

Also this suggestions did not help. Removed step-by-step everything from the configuration until only user login data left.... in the end I did a hard factory reset. Problem still exists.

 

eth3 / SFP looses link after seconds, maximum after 1-2 minutes after booting up. First our Extreme switch detects the link loss and after 1-2 minutes ER-4 changes also status to link down.

Member
Posts: 275
Registered: ‎06-13-2018
Kudos: 58
Solutions: 7

Re: EdgeRouter 4 looses link on SFP interface and ARP stopped working

Did you still have the ARP problem on other interfaces with the stripped down configuration?

 

When you swapped SFP's, did you use the same cable?  Are we talking copper or fiber?

New Member
Posts: 10
Registered: 4 weeks ago
Kudos: 1

Re: EdgeRouter 4 looses link on SFP interface and ARP stopped working

I'am currently focusing on the SFP issue.

 

Swapped the SFP transceivers, also different cables. Today I tested against other SFP vendors (Acal, Flexoptix and Ericsson) and different types of them i.e. SR, LR and BiDi modules. All modules are not working if I try to connect to a different device i.e. Extreme Switch and Ubiquiti ER-6.


But... If I loop RX with TX from the SFP the port comes up and stays up. It looks like that the negotiation of link state with different devices is broken. This is working on all SFP vendors.