Reply
Established Member
Posts: 1,466
Registered: ‎07-17-2011
Kudos: 443
Solutions: 63

ER8 Pro High CPU moderate throughput very basic config

Hi,

 

See below for a CPU graph from an ER 8 PRO we just put into production.

 

er8-pro-high-cpu.png

 

Very straight forward setup, eth0 goes to a newly installed 500/500 fibre. eth1 to a switch with 3 x AF5X (back hauls) and 1 x Rocket5 PTMP (ptmp sector). eth2 through to eth7 are disconnected. No VPN's, no CBQ, very basic firewall, just 1 to 1 source and destination NAT rules.

 

Tonight we started to get reports of laggy performace, looking at the ER8 PRO, the CPU was constantly close to 100%, see graph above prior to 21:14.

 

We did have Traffic Analysis enabled, so we disabled it and rebooted. This did seem to help and you can see the reduced CPU usage after the reboot (21:19) onwards.

 

The CPU usage still seems very high for around 100Mbps of traffic, would this be expected?

 

We are planning to push more traffic through this new fibre circuit over the coming weeks and I am concerned with the high CPU load. What will happen if we quadruple the traffic to 400Mbps? Do we need a higher specification ER and if so which model? Looking at the range only the ER8-XG seems to be significantly more powerful, the ER4 and ER6 less so.

 

Here is top from the router shortly after the reboot:

 

er8-pro-high-cpu-top.png

 

The unusual thing here is the sum of the individual process CPU usages seems to add up to much less than the average CPU load, perhaps this is simply my lack of understanding of top?

 

Finally, I know the GUI is not the best place to vew CPU load etc, however for the purposes of explanation I took a screen shot, again after the reboot:

 

er8-pro-high-cpu-dashboard.png

 

If you look at the traffic displayed for eth0 and eth1 it seems to be out of sync, again perhaps this is normal or down to my lack of understanding of what is being displayed?

 

In particular the Rx of eth1 at 9.53Mbps is a lot lower than the TX of eth0 at 22.8Mbps. Observing this over a period of a few minutes showed this difference to be pretty constant. Also note the CPU at 76% with 107.83Mbps Rx and 22.82Mbps Tx.

 

Thanks in advance for any help and or advice.

>>If you find replies helpful please press the Thumbs Up button to give kudos (thanks). If you feel a reply solves your query please click the Mark as Solution button.
Senior Member
Posts: 4,581
Registered: ‎01-04-2017
Kudos: 623
Solutions: 218

Re: ER8 Pro High CPU moderate throughput very basic config

show configuration | cat

is offload enabled?
Established Member
Posts: 1,466
Registered: ‎07-17-2011
Kudos: 443
Solutions: 63

Re: ER8 Pro High CPU moderate throughput very basic config

[ Edited ]

 

Ipv4 forwarding is enabled, nothing else, but they're not in use.

 

:~$ show ubnt offload

IP offload module   : loaded
IPv4
  forwarding: enabled
  vlan      : disabled
  pppoe     : disabled
  gre       : disabled
IPv6
  forwarding: disabled
  vlan      : disabled
  pppoe     : disabled

IPSec offload module: loaded

Traffic Analysis    :
  export    : disabled
  dpi       : disabled
    version       : 1.354
:~$

Config is very long due to NAT rules, I will remove them and other sensitive info and post shortly...

 

>>If you find replies helpful please press the Thumbs Up button to give kudos (thanks). If you feel a reply solves your query please click the Mark as Solution button.
Established Member
Posts: 1,466
Registered: ‎07-17-2011
Kudos: 443
Solutions: 63

Re: ER8 Pro High CPU moderate throughput very basic config

 

 Removed NAT rules and edited IP's etc:

 

 

firewall {
    all-ping enable
    broadcast-ping disable
    group {
        address-group IPs-to-Masq {
            address 10.10.0.0/24
            address 10.10.1.0/24
            address 10.10.2.0/24
            address 10.10.4.0/24
address 192.168.2.0/24 description "IP Addresses to masquerade" } address-group Public-IPs { address 1.2.7.0/24 address 1.2.9.0/24 description "Public IP Addresses" } } ipv6-receive-redirects disable ipv6-src-route disable ip-src-route disable log-martians enable name WAN_IN { default-action drop description "WAN to internal" rule 10 { action accept description Public-IPs destination { group { address-group Public-IPs } } log disable protocol all } rule 20 { action accept description "Allow established/related" state { established enable related enable } } rule 30 { action drop description "Drop invalid state" state { invalid enable } } } name WAN_LOCAL { default-action drop description "WAN to router" rule 20 { action accept description "Allow established/related" state { established enable related enable } } rule 30 { action drop description "Drop invalid state" state { invalid enable } } } receive-redirects disable send-redirects enable source-validation disable syn-cookies enable } interfaces { ethernet eth0 { address 1.2.5.251/31 description Internet duplex auto firewall { in { name WAN_IN } local { name WAN_LOCAL } } speed auto } ethernet eth1 { address 10.10.0.5/24 description eth1 duplex auto speed auto } ethernet eth2 { address 192.168.2.1/24 description eth2 duplex auto speed auto } ethernet eth3 { duplex auto speed auto } ethernet eth4 { duplex auto speed auto } ethernet eth5 { description eth5 disable duplex auto speed auto } ethernet eth6 { description eth6 disable duplex auto speed auto } ethernet eth7 { description eth7 disable duplex auto speed auto } loopback lo { } } port-forward { auto-firewall enable hairpin-nat enable lan-interface eth1 wan-interface eth0 } protocols { static { route 10.10.1.0/24 { next-hop 10.10.0.29 { description /* removed */ distance 1 } } route 10.10.2.0/24 { next-hop 10.10.0.29 { description /* removed */ distance 2 } } route 10.10.4.0/24 { next-hop 10.10.0.49 { description /* removed */ distance 3 } } } } service { dhcp-server { disabled false hostfile-update disable shared-network-name LAN1 { authoritative enable subnet 10.10.0.0/24 { default-router 10.10.0.5 dns-server 8.8.8.8 dns-server 208.67.222.222 domain-name /* removed */ lease 86400 start 10.10.0.191 { stop 10.10.0.199 } } } shared-network-name LAN2 { authoritative enable subnet 192.168.2.0/24 { default-router 192.168.2.1 dns-server 8.8.8.8
dns-server 208.67.222.222 lease 86400 start 192.168.2.131 { stop 192.168.2.139 } } } static-arp disable use-dnsmasq disable } dns { forwarding { cache-size 150 listen-on eth1 } } gui { http-port 80 https-port 443 older-ciphers enable } nat { /* */ /* removed source and destination nat rules */ /* to reduce length of config to aid readability */ /* */ } ssh { port 22 protocol-version v2 } ubnt-discover { disable } unms { connection wss:// /* removed */ } } system { domain-name /* removed */ gateway-address 1.2.5.250 host-name /*removed*/ login { user /* removed */ { authentication { encrypted-password /* removed */ } level admin } } name-server 8.8.8.8 name-server 208.67.222.222 name-server 9.9.9.9 name-server 64.6.64.6 ntp { server 0.ubnt.pool.ntp.org { } server 1.ubnt.pool.ntp.org { } server 2.ubnt.pool.ntp.org { } server 3.ubnt.pool.ntp.org { } } syslog { global { facility all { level notice } facility protocols { level debug } } } time-zone Europe traffic-analysis { dpi disable export disable } } /* Warning: Do not remove the following line. */

 

 

>>If you find replies helpful please press the Thumbs Up button to give kudos (thanks). If you feel a reply solves your query please click the Mark as Solution button.
Senior Member
Posts: 4,581
Registered: ‎01-04-2017
Kudos: 623
Solutions: 218

Re: ER8 Pro High CPU moderate throughput very basic config

[ Edited ]

You need to enable offload

 

I don't see offload in your config

 

Also after you enable offload the router needs to be restarted


https://help.ubnt.com/hc/en-us/articles/115006567467-EdgeRouter-Hardware-Offloading-Explained

Established Member
Posts: 1,466
Registered: ‎07-17-2011
Kudos: 443
Solutions: 63

Re: ER8 Pro High CPU moderate throughput very basic config

[ Edited ]

 

Is it possible IPv4 offloading is enabled by default on the ER8 PRO?

 

I posted the output from "show ubnt offload" above and it reports "IPv4 forwarding : enabled"

 

Should I try enabling it again ?

>>If you find replies helpful please press the Thumbs Up button to give kudos (thanks). If you feel a reply solves your query please click the Mark as Solution button.
Senior Member
Posts: 4,581
Registered: ‎01-04-2017
Kudos: 623
Solutions: 218

Re: ER8 Pro High CPU moderate throughput very basic config

It is not enabled by default. I would enable it as per the article
Highlighted
Established Member
Posts: 1,466
Registered: ‎07-17-2011
Kudos: 443
Solutions: 63

Re: ER8 Pro High CPU moderate throughput very basic config

[ Edited ]

I applied IPv4 forwarding as described in the article:

 

 

configure

set system offload ipv4 forwarding enable

commit ; save

and then restarted the router.

 

I didn't apply any of the other offloading as the specific functionality is not in use. I assume there is some disadvantage to having all the offloading enabled if it is not being used, otherwise I would expect all of the offloading to be enabled by default?

 

I now have a new offload key in the system section of my config as follows:

 

system {
    domain-name /*removed*/
    gateway-address 1.2.5.250
    host-name /*removed*/
    login {
        user /*removed*/ {
            authentication {
                encrypted-password /*removed*/
            }
            level admin
        }
    }
    name-server 8.8.8.8
    name-server 208.67.222.222
    name-server 9.9.9.9
    name-server 64.6.64.6
    ntp {
        server 0.ubnt.pool.ntp.org {
        }
        server 1.ubnt.pool.ntp.org {
        }
        server 2.ubnt.pool.ntp.org {
        }
        server 3.ubnt.pool.ntp.org {
        }
    }
    offload {
        hwnat disable
        ipv4 {
            forwarding enable
        }
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone Europe
    traffic-analysis {
        dpi disable
        export disable
    }
}

/* Warning: Do not remove the following line. */

 

The output of show ubnt offload did not change and also the CPU load did not decrease noticeably until the traffic level reduced.

 

 

:~$ show ubnt offload

IP offload module   : loaded
IPv4
  forwarding: enabled
  vlan      : disabled
  pppoe     : disabled
  gre       : disabled
IPv6
  forwarding: disabled
  vlan      : disabled
  pppoe     : disabled

IPSec offload module: loaded

Traffic Analysis    :
  export    : disabled
  dpi       : disabled
    version       : 1.354
:~$

 

I know I am like a dog with a bone but this leads me to believe IPv4 forwarding offloading was enabled all along! Looking at the article there are differences between the Cavium and MediaTek based routers. For example hwnat does not apply to Cavium based units. For the avoidance of doubt could UBNT comment on this please? @UBNT-afomins @UBNT-sandisn @UBNT-benpin

 

Anyway, CPU load is fine this morning but there is only 30 to 40Mbps of total traffic.I will keep en eye on the CPU later tonight and see how it behaves once the traffic levels increase.

>>If you find replies helpful please press the Thumbs Up button to give kudos (thanks). If you feel a reply solves your query please click the Mark as Solution button.
Senior Member
Posts: 4,581
Registered: ‎01-04-2017
Kudos: 623
Solutions: 218

Re: ER8 Pro High CPU moderate throughput very basic config

[ Edited ]

Relooking over your TOP output SI is what is killing you, which means its a system interupt.  did a quick search and found this threads that might help you:

 

Looks like its a bug

 

https://community.ubnt.com/t5/UniFi-Routing-Switching-Beta/extreme-cpu-usage-ksoftirqd/m-p/1885369#M...

 

 

 

 

 NEVERMIND your not using flow accounting.

 

Established Member
Posts: 1,466
Registered: ‎07-17-2011
Kudos: 443
Solutions: 63

Re: ER8 Pro High CPU moderate throughput very basic config

Thank you for your help @smyers119 appreciate your time.
>>If you find replies helpful please press the Thumbs Up button to give kudos (thanks). If you feel a reply solves your query please click the Mark as Solution button.
Ubiquiti Employee
Posts: 1,067
Registered: ‎07-20-2015
Kudos: 1087
Solutions: 73

Re: ER8 Pro High CPU moderate throughput very basic config

[ Edited ]

@Ripv:
> The CPU usage still seems very high for around 100Mbps of traffic, would this be expected?
Nope, that's not normal.

 

In screenshot I see that you have 158 NAT rule:

  1. Why do you have so many NAT rules?
  2. Can you substitute multiple SNAT rules with single MASQUARADE rule?
Established Member
Posts: 1,466
Registered: ‎07-17-2011
Kudos: 443
Solutions: 63

Re: ER8 Pro High CPU moderate throughput very basic config


@UBNT-afomins wrote:

@Ripv:
> The CPU usage still seems very high for around 100Mbps of traffic, would this be expected?
Nope, that's not normal.

 

In screenshot I see that you have 158 NAT rule:

  1. Why do you have so many NAT rules?
  2. Can you substitute multiple SNAT rules with single MASQUARADE rule?

At the moment the NAT rules are assigning Public IP addresses. We are still deciding how best to finish out, but in broad terms the eventual plan is to move to PPPoE or IPoE or similar.

>>If you find replies helpful please press the Thumbs Up button to give kudos (thanks). If you feel a reply solves your query please click the Mark as Solution button.
Senior Member
Posts: 4,581
Registered: ‎01-04-2017
Kudos: 623
Solutions: 218

Re: ER8 Pro High CPU moderate throughput very basic config

[ Edited ]

Can you give us an example of a SNAT and DNAT rule. Looking through your config, it appears your using NAT in a weird way.

IT appears that

- you have a carrier link at 1.2.5.251/31

- you have two public ip blocks routed to you 1.2.7.0/24 and 1.2.9.0/24

- you have 2 rfc1918 lan's attached to the router.

- You do not have your two public blocks connected directly to the router in anyway other then a firewall rule.

So I am not really understanding where/why the NAT comes in.

If the NAT is causing the issue We should take a step back and look at the big picture, there definately is a better way to do what you are trying to achieve.

Are you acting as a ISP? Are you able to skip the NAT for now and just let the router route? that's going to be the easiest solution. until you come up with some type of authentication plan such as pppoe.

Are you doing CGNAT as well? if so you should be using RFC6598 space. RFC1918 is meant for your internal network, it should not extend to your customers.


Established Member
Posts: 1,466
Registered: ‎07-17-2011
Kudos: 443
Solutions: 63

Re: ER8 Pro High CPU moderate throughput very basic config

[ Edited ]

@smyers119 wrote:

Can you give us an example of a SNAT and DNAT rule. Looking through your config, it appears your using NAT in a weird way.

IT appears that

- you have a carrier link at 1.2.5.251/31

- you have two public ip blocks routed to you 1.2.7.0/24 and 1.2.9.0/24

- you have 2 rfc1918 lan's attached to the router.

- You do not have your two public blocks connected directly to the router in anyway other then a firewall rule.

So I am not really understanding where/why the NAT comes in.

If the NAT is causing the issue We should take a step back and look at the big picture, there definately is a better way to do what you are trying to achieve.

Are you acting as a ISP? Are you able to skip the NAT for now and just let the router route? that's going to be the easiest solution. until you come up with some type of authentication plan such as pppoe.

Are you doing CGNAT as well? if so you should be using RFC6598 space. RFC1918 is meant for your internal network, it should not extend to your customers.



 

snat-eg.png

 

 

dnat-eg.png

 

 

- you have a carrier link at 1.2.5.251/31

Yes

- you have two public ip blocks routed to you 1.2.7.0/24 and 1.2.9.0/24
Yes


- you have 2 rfc1918 lan's attached to the router.

Yes, 192.168.2.1/24 is on eth2 just for local access.

 

- You do not have your two public blocks connected directly to the router in anyway other then a firewall rule.

Not sure on this one, they are routed to eth0 (1.2.5.251/31) by our upstream.

 

- So I am not really understanding where/why the NAT comes in.

If we take the 10.10.2.0/24 network, this represents a POP,  an AF5X backhaul connects to a local ERX-SFP at that POP. This POP also has sectors connected to CPE's. CPE's are in router mode. They get a 10.10.2.x/24 address assigned by DHCP from the ERX-SFP on their WAN and provide 192.168.168.0/24 to the LAN. Traffic is routed to the ER8-PRO and the NAT rules map the 10.10.2.x/24 addresses to Public IP's.

 

-Are you acting as a ISP? Are you able to skip the NAT for now and just let the router route? that's going to be the easiest solution. until you come up with some type of authentication plan such as pppoe.

Yes. No, I don't think so, we are looking for each CPE to use it's own public IP.

 

- Are you doing CGNAT as well? if so you should be using RFC6598 space. RFC1918 is meant for your internal network, it should not extend to your customers.
No.

>>If you find replies helpful please press the Thumbs Up button to give kudos (thanks). If you feel a reply solves your query please click the Mark as Solution button.
Senior Member
Posts: 4,581
Registered: ‎01-04-2017
Kudos: 623
Solutions: 218

Re: ER8 Pro High CPU moderate throughput very basic config

It sounds like your better off straight routing then. and just have 3 NAT rules 1 masq and 2 that exclude your 2 public blocks from NAT.
Reply