Reply
New Member
Posts: 6
Registered: Monday

edgerouter lite high load issue

need some help with 2 issues i have since i bought edgerouter lite.  i set up the edgerouter with 2 cable modems(same isp) in a 2 wan configuration with 50/50 load balancing. not too familiar with the CLI, everything was done by the config tree.

 

issue #1 high priority: seems every day or two  the upload (Tx) will hover at 160Mbps~ on 1 WAN with 100% cpu load which slows or kills most established connections. if i disable the WAN affected, it simply switches to WAN2. tried to wait it out but after 10mins it's still stuck like that, i'm forced to reboot the edgerouter to fix this. been happening for weeks now every day or two. it's not actually transferring any data through the modems as the modems both have a max of 40mpbs upload each. 160mbps load on one random eth port makes no sense to me, and rebooting the router every 2 days is irritating. any help or explanation would be appreciated.

 

issue #2 low priority: can't seem to make fail-over work properly though i probably don't need it. when i set fail-over, at some point it detects a wan down and sets it as unreachable after the set amount of pings(3) to 8.8.8.8. it then routes all traffic to the other wan. that's fine, the problem is that the fallen wan is never restored unless i manually disable/enable the corresponding eth port in the edgeOS dashboard. solved this by simply disabling the fail-over and leaving 50/50 load balance on. these modems almost never go down so it's strange that it becomes unreachable with fail-over after a day or so. since fail-over was disabled i've had no problems. aside from having no fail-over.

Highlighted
SuperUser
Posts: 19,593
Registered: ‎09-17-2013
Kudos: 4925
Solutions: 1388

Re: edgerouter lite high load issue

post your config.

Senior Member
Posts: 4,136
Registered: ‎01-04-2017
Kudos: 567
Solutions: 196

Re: edgerouter lite high load issue

also "show version"
New Member
Posts: 6
Registered: Monday

Re: edgerouter lite high load issue

config:

Spoiler
 firewall {                                                                      
    all-ping enable                                                             
    broadcast-ping disable                                                      
    group {                                                                     
        network-group PRIVATE_NETS {                                            
            network 192.168.0.0/16                                              
            network 172.16.0.0/12                                               
            network 10.0.0.0/8                                                  
        }                                                                       
    }                                                                           
    ipv6-receive-redirects disable                                              
    ipv6-src-route disable                                                      
    ip-src-route disable                                                        
    log-martians disable                                                        
    modify balance {                                                            
        rule 10 {                                                               
            action modify                                                       
            description "do NOT load balance lan to lan"                        
            destination {                                                       
                group {                                                         
                    network-group PRIVATE_NETS                                  
                }                                                               
            }                                                                   
            modify {
                table main
            }
        }
        rule 30 {
            action modify
            description "do NOT load balance destination public address"
            destination {
                group {                                                         
                    address-group ADDRv4_eth1                                   
                }                                                               
            }                                                                   
            modify {                                                            
                table main                                                      
            }                                                                   
        }                                                                       
        rule 40 {                                                               
            action modify                                                       
            description "do NOT load balance destination public address"        
            destination {
                group {                                                         
                    address-group ADDRv4_eth2                                   
                }                                                               
            }                                                                   
            modify {                                                            
                table main                                                      
            }                                                                   
        }                                                                       
        rule 110 {                                                              
            action modify                                                       
            modify {                                                            
                lb-group G
            }                                                                   
        }                                                                       
    }                                                                           
    receive-redirects disable                                                   
    send-redirects enable                                                       
    source-validation disable                                                   
    syn-cookies enable                                                          
}                                                                               
interfaces {                                                                    
    ethernet eth0 {                                                             
        address 192.168.1.1/24                                                  
        description Local
        duplex auto                                                             
        firewall {                                                              
            in {                                                                
                modify balance                                                  
            }                                                                   
        }                                                                       
        speed auto                                                              
    }                                                                           
    ethernet eth1 {                                                             
        address dhcp                                                            
        description "WAN 1"                                                     
        duplex auto
        mac 00:B2:45:23:F2:BC                                                   
        speed auto                                                              
    }                                                                           
    ethernet eth2 {                                                             
        address dhcp                                                            
        description "WAN 2"                                                     
        duplex auto                                                             
        mac 00:A6:34:25:F1:66                                                   
        speed auto                                                              
    }                                                                           
    loopback lo {                                                               
    }
}                                                                               
load-balance {                                                                  
    group G {                                                                   
        interface eth1 {                                                        
            route-test {                                                        
                count {                                                         
                    failure 12                                                  
                    success 2                                                   
                }                                                               
                initial-delay 3                                                 
                interval 3                                                      
                type {
                    ping {                                                      
                    }                                                           
                }                                                               
            }                                                                   
            weight 50                                                           
        }                                                                       
        interface eth2 {                                                        
            route-test {                                                        
                count {                                                         
                    failure 12                                                  
                    success 2                                                   
                }
                initial-delay 3                                                 
                interval 3                                                      
                type {                                                          
                    ping {                                                      
                    }                                                           
                }                                                               
            }                                                                   
            weight 50                                                           
        }                                                                       
        lb-local enable                                                         
        lb-local-metric-change disable                                          
    }
}                                                                               
service {                                                                       
    dhcp-server {                                                               
        disabled false                                                          
        hostfile-update disable                                                 
        shared-network-name LAN {                                               
            authoritative enable                                                
            subnet 192.168.1.0/24 {                                             
                default-router 192.168.1.1                                      
                dns-server 192.168.1.1                                          
                lease 86400                                                     
                start 192.168.1.38 {
                    stop 192.168.1.243                                          
                }                                                               
            }                                                                   
        }                                                                       
        static-arp disable                                                      
        use-dnsmasq disable                                                     
    }                                                                           
    dns {                                                                       
        forwarding {                                                            
            cache-size 150                                                      
            listen-on eth0                                                      
        }
    }                                                                           
    gui {                                                                       
        http-port 80                                                            
        https-port 443                                                          
        older-ciphers enable                                                    
    }                                                                           
    nat {                                                                       
        rule 5002 {                                                             
            description "masquerade for WAN"                                    
            outbound-interface eth1                                             
            type masquerade                                                     
        }
        rule 5004 {                                                             
            description "masquerade for WAN 2"                                  
            outbound-interface eth2                                             
            type masquerade                                                     
        }                                                                       
    }                                                                           
    ssh {                                                                       
        port 22                                                                 
        protocol-version v2                                                     
    }                                                                           
}                                                                               
system {
    conntrack {                                                                 
        expect-table-size 4096                                                  
        hash-size 4096                                                          
        table-size 32768                                                        
        tcp {                                                                   
            half-open-connections 512                                           
            loose enable                                                        
            max-retrans 3                                                       
        }                                                                       
    }                                                                           
    host-name ubnt                                                              
    login {
        user ubnt {                                                             
            authentication {                                                    
                encrypted-password ****************                             
            }                                                                   
            level admin                                                         
        }                                                                       
    }                                                                           
    ntp {                                                                       
        server 0.ubnt.pool.ntp.org {                                            
        }                                                                       
        server 1.ubnt.pool.ntp.org {                                            
        }
        server 2.ubnt.pool.ntp.org {                                            
        }                                                                       
        server 3.ubnt.pool.ntp.org {                                            
        }                                                                       
    }                                                                           
    offload {                                                                   
        ipsec enable                                                            
    }                                                                           
    syslog {                                                                    
        global {                                                                
            facility all {                                                      
                level notice
            }                                                                   
            facility protocols {                                                
                level debug                                                     
            }                                                                   
        }                                                                       
    }                                                                           
    time-zone UTC                                                               
    traffic-analysis {                                                          
        dpi enable                                                              
        export enable                                                           
    }                                                                           
}

version:

Spoiler
Version:      v1.10.0                                                           
Build ID:     5056246                                                           
Build on:     01/25/18 10:07                                                    
Copyright:    2012-2018 Ubiquiti Networks, Inc.                                 
HW model:     EdgeRouter Lite 3-Port                                            
HW S/N:       F09FC2CFBBF1                                                      
Uptime:       17:54:37 up 10:20,  2 users,  load average: 0.50, 0.38, 0.32    
Senior Member
Posts: 4,136
Registered: ‎01-04-2017
Kudos: 567
Solutions: 196

Re: edgerouter lite high load issue

Enable offload

 

You do not have any ip's in for the lb test.  I would suggest switching to this custom script:

ubnt@ER-Pro:~$ cat /config/scripts/pinger
#!/bin/bash

# add your ping targets here 
targets=(
        '172.16.3.242'
        '8.8.8.8'
        '8.8.4.4'
)

if [ $# != 3 ]
then
   echo "Usages: $0 <group> <intf> <status>"
   exit 1
fi

group=$1
intf=$2
status=$3
        
for host in "${targets[@]}"
do
    /bin/ping -n -c 1 -W 1 -w1 -I $intf $host
    if [ $? == 0 ] 
    then
       exit 0
    fi    
done

# fail
exit 1
configure
edit load-balance group <name> interface <name> route-test
set type script /config/scripts/pinger
top
commit
save
exit

Source

New Member
Posts: 6
Registered: Monday

Re: edgerouter lite high load issue

[ Edited ]

that was issue#2. i manually removed the ping target via config tree to prevent the watchdog from permanently setting a WAN as down and never restoring it. since i did that both WANs have remained up and functional.

 

bigger problem is issue#1 which just happened now. both WAN eth loads spiked to 200Mbps each (cable modems max upload is 40mpbs) and cpu load got stuck at 100% killing all connections to the router. had to do a hard reboot to fix.

 

no idea why this happens every day or two nor how to diagnose/fix. anyone have any input or theories?

 

took pics of the dashboard and console during this episode before rebooting:

edgeOS1.jpgedgeOS2.jpg

 

edit: trimmed the pics

Established Member
Posts: 2,158
Registered: ‎08-06-2015
Kudos: 895
Solutions: 127

Re: edgerouter lite high load issue

Did you enable offloading as suggested?

 

It is difficult to tell from your screenshots.  Did you obscure any part of the 'top' display?  I don't see process names listed for many and it looks like those are user 'ubnt'.

 

I noted it appears you are logged in as the default 'ubnt' user which makes it a little harder to identify, but there should not be so many processes running as 'ubnt' or any other non-root user.  Not having those process names visible in the 'top' screen is very unusual.

 

Put all together (including the very high CPU) I suspect your ER may have been compromised and is running malicious executables.

 

 

New Member
Posts: 6
Registered: Monday

Re: edgerouter lite high load issue

i did my best not to obscure any of the top displays, when this happens i usually see a lot of those greyed-out "ubnt" entries with no description. i've been using the default ubnt login.

 

this is what i get on offload status:

 

IP offload module : loaded
IPv4
forwarding: enabled
vlan : disabled
pppoe : disabled
gre : disabled
IPv6
forwarding: disabled
vlan : disabled
pppoe : disabled
 
IPSec offload module: loaded
 
Traffic Analysis :
export : enabled
dpi : enabled
version : 1.354

 

any recommendations? should i reset the router and configure it all again? or maybe install the newer firmware version 1.10.1? i'm currently running 1.10.0 since i bought this 2 months ago

 

New Member
Posts: 6
Registered: Monday

Re: edgerouter lite high load issue

since it was mentioned my edgerouter could be compromised i decided to make a new user and deleted ubnt user account.

 

after this change my logs have been filled with nonstop auth failures (around 10 entries every minute for the past few hours). any reason for this? should i be worried or can it be ignored?

 

here is the sys log pic:

ubnt_log01.jpg

SuperUser
Posts: 19,593
Registered: ‎09-17-2013
Kudos: 4925
Solutions: 1388

Re: edgerouter lite high load issue

1. backup config.boot

2. restore to factory defaults

3. (re-)flash it with the latest firmware

4. restore config.boot

5. triple-check that the "ubnt" user is gone.

New Member
Posts: 6
Registered: Monday

Re: edgerouter lite high load issue

[ Edited ]

will do. any explantion for what could be behind those log entries? just curious

 

edit: logs fixed, just had to reconfigure everything while logged as the new user account

Established Member
Posts: 2,158
Registered: ‎08-06-2015
Kudos: 895
Solutions: 127

Re: edgerouter lite high load issue

My recommendation would be to do a hard-reset (EdgeRouter - Reset to Factory Defaults) with the router disconnected from any network, then change the default username and password before connecting to any WAN (public-facing networks).

 

You may want to apply new firmware, twice, to ensure neither of the firmware images on your ER currently remain.

 

The wizards tab in the GUI will give you an option to create a new user and remove 'ubnt' or you can do this via CLI.  At minimum the default credential (password) must be changed before connecting to public networks even if you keep the 'ubnt' user.

 

Then after reconfiguring your router see if the same starts to occur.

 

Those logs would also tend to suggest your router may have been compromised.  Those are SSH attempts from external hosts trying to login using the user 'ubnt'.  That itself is not necesarily indicative of any issues (you may still see these in the future) but if you had not seen those prior to removing the ubnt user would suggest the logins had been successfull.  You may find records of successful logins, but those are logged at a lower priority than failures so may not be seen.

 

 

 

Reply