Upcoming Maintenance Alert:

The UBNT Community will be upgraded at 5pm MDT on April 25th. During this time the community forums will be set to read-only status.

Learn more

×
Reply
Emerging Member
Posts: 83
Registered: ‎07-07-2014
Kudos: 15
Solutions: 1

High CPU with QoS - Optimisation Feedback & Hardware Offload

Hi All,

 

Very happy with the ER series - our biggest issue right now is that we have mutliple (10+) ERP's hitting over 90% CPU during peak load (drops back to 5% overnight), we believe this is caused by the QoS (which I believe is not hardware offloaded).

 

We've found the problem most prevelant on devices that have to perform a lot of packet shaping (i.e. sustained periods of maxxing out the throttled speed).

 

We currently use a config like this:

 

interfaces {
    ethernet eth0 {
        vif 10 {
            address 10.0.0.1/30
            redirect ifb10
            traffic-policy {
                out '20mb'
            }
        }
    }
    input ifb10 {
        traffic-policy {
            out '20mb'
        }
    }
}

traffic-policy {
    shaper 20mb {
        bandwidth 20mbit
        default {
            bandwidth 100%
            burst 1k
            ceiling 100%
            queue-type fair-queue
        }
    }
}
  • Could this policy be improved for better performance?
  • Is QoS hardware offloading of any kind possible on the current chipset?
Previous Employee
Posts: 13,551
Registered: ‎06-10-2011
Kudos: 5401
Solutions: 1656
Contributions: 2

Re: High CPU with QoS - Optimisation Feedback & Hardware Offload

Just to confirm, is the "20mbit" in the above config an example or is it the typical bandwidth level for those setups that are showing the CPU usage? Also, you are right that traffic going through QoS cannot be "offloaded" by the current "offload" mechanism, and it wouldn't be feasible to implement full QoS functionality in there, but it might be possible for some more limited form and we may look into such possibilities.

Emerging Member
Posts: 83
Registered: ‎07-07-2014
Kudos: 15
Solutions: 1

Re: High CPU with QoS - Optimisation Feedback & Hardware Offload

Thanks for the reply @UBNT-ancheng - we have observed this on devices running multiple policies - for example one ER has 1 interface limited to 100mb, 1 limited to 50Mb, and 5 limited to 30mb, it sit's at 80-95% CPU at peak times during the day. There is another interface on that device with high usage but no limit, if I add 100Mb limit to this, CPU spikes to 90-99% for most of the day (back to <5% overnight).

 

I think this kind of very basic policing is very common (particularly with service providers) and so any kind of "offload" for the most basic traffic policies would be a big win Man Happy

 

Equally, if there was an option to have double CPU, we'd buy them all day long!

 

 

Member
Posts: 120
Registered: ‎01-14-2015
Kudos: 45
Solutions: 11

Re: High CPU with QoS - Optimisation Feedback & Hardware Offload

The ERL can only do ~60-70Mbps total QoS throughput (but this will vary depending on the number and complexity of shaper policies and some of the advanced settings).

 

If you attempt to "limit" an interface to, say, 100Mbps - the limiting itself cannot be offloaded, and since the ERL can only do 60-70Mbps, it won't even be able to reach the speed of the limit you set.

 

Best to not use QoS if you need much higher throughput (>100Mbps).  If you only need 50-60Mbps, the ERL may be adequate.  You might also look into an ER-X, which can do ~150Mbps QoS throughput from my experience.

Emerging Member
Posts: 83
Registered: ‎07-07-2014
Kudos: 15
Solutions: 1

Re: High CPU with QoS - Optimisation Feedback & Hardware Offload

Interesting...to be clear, the specific behaviour I was referencing above is being seen on ER and ERP's (we do use ERL but not for this role)

Previous Employee
Posts: 13,551
Registered: ‎06-10-2011
Kudos: 5401
Solutions: 1656
Contributions: 2

Re: High CPU with QoS - Optimisation Feedback & Hardware Offload

Yeah the main factor would be the "total bandwidth" being shaped, so in your example above that would be close to 300 Mbps (100 + 50 + 30 + 100) already, so the observed CPU usage is likely normal.

Established Member
Posts: 1,704
Registered: ‎02-17-2015
Kudos: 442
Solutions: 45

Re: High CPU with QoS - Optimisation Feedback & Hardware Offload

@UBNT-ancheng => it's an old post here, but as the Cavium SoC are "sold with a QOS offload", won't it be possible to implement a simple best efffort queing based on DSCP, and running are almost wirespeed ?

Note : all the current implement a fancy, and doing a lot, but at the end, it all about classifying the application adn giving latency priority, the bandwitdh limitation are nice, ut are they really applicable, when for most of the implementation, the idea is simply to pass latency sensitive traffic first (VoVIP), then traffic sensitive (like DNS, NTP, SSH), basic browsing, email, and let the remaning capacity available to streaming and others bittorrent or whatever P2P is there in the network ;-P And DSCP are quite nice for this.

Previous Employee
Posts: 13,551
Registered: ‎06-10-2011
Kudos: 5401
Solutions: 1656
Contributions: 2

Re: High CPU with QoS - Optimisation Feedback & Hardware Offload

di3: On some of the existing platforms there are hardware features that might possibly be used to implement certain basic QoS functions, but note that it would not have nearly the same level of flexibility as the currently available QoS features and we'll need to see when/if it might be feasible to look into adding such feature depending on various factors (e.g., different platforms would require different implementations, resources constraint, etc.).

SuperUser
Posts: 7,133
Registered: ‎01-10-2012
Kudos: 3768
Solutions: 348

Re: High CPU with QoS - Optimisation Feedback & Hardware Offload

[ Edited ]

An EdgeRouter line that had more CPU, even if the hardware cost double the current hardware, would be most welcome by many, I would think.  It's a shame that such powerful software is hobbled by truely anemic CPUs.  Atom with QuickAssit support would be awesome - Intel announcing the abandonment of Atom probably dooms that Man Sad  

 

Then again Atom isn't going anywhere for a while from an availability standpoint and stuff like this is not going to be possible with MIPS cores. 

When you receive a solution to your question/issue, don't forget to mark your thread as solved and to give kudo's to the people who have helped you out!

Having wifi problems? Take a look here first: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
Established Member
Posts: 1,704
Registered: ‎02-17-2015
Kudos: 442
Solutions: 45

Re: High CPU with QoS - Optimisation Feedback & Hardware Offload

[ Edited ]

@UBNT-ancheng => the current implemented QOS, using a lot of features, assumptions are mostly unusable, in a sense that even it work today, a simple change in traffic/usage pattern, and it's another settings.

Clearly, what users wants is their critical voice and video conf applications to be perfect, the messenging part to be delivered, and the browsing experience to be fine, no matter if an idiot is using bittorrent on the network, and simple DSCP management works perfectly.

 

I have user using a 1 Gbps connection with massive router complaining about the same issue as another user having a 2 Mbps with 100 users !

Reality is latency and priority, not bandwitdh, and delaying packets ... as anyway the TCP stack especially is doing it for us (you still need an ACK and the windows size on the opposite side to make an assumption when to send hte next packet !), but in the means time, time critical applicaiton, mostly over UDP, need low latency to be efficient.

 

 

Here, a simple desig, I used over and over, and working perfectly, and I can give you access to the controller, VPN access to check the traffic and see the performance, and really of usage.

 

I did a test in multiple charities in India, Phillippines, Indonesia and others place : 

  • between 2 to 4 Mbps on ADSL available for download
  • between 256K to 512 Kbps on ADSL available for upload

3 VLAN/mapped with SSID

  • internal => laptop, used for work and conf call
  • staff => phones and others
  • guest => external users, public

3 rules to modify the DSCP on the internal interface :

  • internal is mapped to EF => expedited forwarding
  • staff is mapped to C3 => medium priority
  • others mapped to C7 => lowest priority

 

and simple fast-prio on the external interface and internal interface

 

result is perfect, and the conneciton is sometime handling over 50 conccurent users, without issue, as as soon someone on the laptop need to do time critical application, like conf call with Europe, US, it got no bandwitdh, but the lowest latency possible in routing (not transit obviously, as they mostly don't care about the DSCP set to EF), and access to the WAN medium faster, and it works !

 

And you know what : I got the same problem with user using 1 Gbps WAN fiber connection and not because the router cannot do it : it's the USG-Pro-4P, simpy because the QOS implemented in software cannot be use at high speed, and the really of usage is clearly something else than complicate algorithms, but simply what Internet is, and have been built on : best effort, and this need simple algorithm with latency management to reduce buffer bloat and others 100 ms traffic routing and we need "intelligent edge" to match protocol or at least interface with DSCP to organize the routing traffic, nothing more complicated.

Established Member
Posts: 1,704
Registered: ‎02-17-2015
Kudos: 442
Solutions: 45

Re: High CPU with QoS - Optimisation Feedback & Hardware Offload

@EricE => the first part is not my call, and I'm personnaly not the one who say ok not ok based on the price (it's my users/customers)

 

 

but one things I'm sure, is that the Intel Atom, even well used is having the same problem and with a thermal envellop slightly higher than the Cavium CPU : I used the Sokeris net6501 with openBSD and freeBSD ( I prefer OpenBSD), and they are simply AWESOME with the Packet Filter (PF) with ALQ providing firewall and priority solution in a simple file ,and performing thousand time better than linux and all the undocument Traffic Control everywhere having hook all around the place ...

 

Note : some may say : linux is more open, this is no the point, this is mostly usability here

 

And the Atom, with QuickAssist, sadly, choke too at 100 Mbps ... because the QuickAssist do not work when you start to calculate : the quickAssist is the equivalent of the offload  on the Cavium, and after, when you have complicate solution to implement, then the core, x86 or MIPS come into the picture !

 

Please, check on the pfsense.org, the hardware proposed, and you'll see the thermal envelop proposed and price too, and you'll see if everybody wants to buy it ;-( 

I personally wants Kiss things (Keep It Simple and Stupid), and make it as fast as possible, so a simple offloaded hardware, providing capacity and low latency on a prio based algorithm is perfect, on a 2 Mbps ADSL as a 1Gbps FFT ;-)

 

Senior Member
Posts: 4,481
Registered: ‎03-24-2016
Kudos: 1232
Solutions: 515

Re: High CPU with QoS - Optimisation Feedback & Hardware Offload

Instead of throwing lots of CPU at this QoS problem, why not solve it in hardware?

My low end HP1920 switch does QoS in hardware, has 8 queues per port, and can limit bandwidth.

 

Why not use ethernet interfaces like that?

Established Member
Posts: 1,704
Registered: ‎02-17-2015
Kudos: 442
Solutions: 45

Re: High CPU with QoS - Optimisation Feedback & Hardware Offload

@16again => ideally yes, and having the same on every ports will allow to get the perfect QOS, as switching and routing as slightly different, but clearly, it would be the best.

But as per QOS on switch right now, most of them take care of the DSCP, and the port and eventualyl the VLAN, and not the application traffic itself, so, it's for now, not perfect, but could be the best starting point !

 

From the last EdgeSwitch documentation (beta forum), the next OS for switch has DSCP marking available already.

Reply