Member
Posts: 130
Registered: ‎05-26-2015
Kudos: 20
Accepted Solution

Switch Bandwith control from Router to CPE

[ Edited ]

So I've noticed UCRM creates 2 codel queues per customer to limit bandwith on ER routers.... which effectively turns Hw offloading off because QoS is used. 

 

This has a very nasty impact on the ER performance.. which becomes crippled over time, as UCRM adds more and more codel queues per service.

 

I'm starting to notice some performance issues. The ER turns slow and I'm constantly getting reports of low speed, while seeing only 60% of my channel being used... meaning a possible bottleneck.

 

Meanwhile... bandwith is set to allow 8Mbit per customer on the ER, but users are roughly getting 4 or 5 with more than enough idle bandwith on the main link... I managed to see improvements by turning off netflow and traffic accounting...

 

This leads me to think that it might be just about time to either upgrade my hardware or offload the bandwith control to each CPE in order to relief the ER from hadling 190 codel queues.

 

I believe that having offloading active without codel could relief the router.. but I'm afraid if I turn codel off, thr CPEs will just fight over bandwith and stuff like VoIP will start working like crap... 

 

If I upgrade to a powerful ER-6... it has PoE but no hardware switch, which would push me into splitting my network... or into using my old ER-X as a PoE switch and the ER6 as thr main router.

 

I'm trying to find the best gear to manage my bandwith fairly from UCRM.

 

My questions are:

 

1) Should I migrate to a ER-6 router for this amount of customers? What are the hardware limits of an ER-X as an internet gateway controlled by UCRM?

 

2) What happens if I switch UCRM to do bandwith control on the CPEs ?.. will it delete the QoS config from the router and proceed to configure each Litebeam?. 

 

3) What happens if a CPE is off during the migration to BW control on the CPEs? Does UCRM uses a queue broker to control pending jobs over each CPE?

 

any other comments? or similar experiences?

 

Best

Pedro

 

 


Accepted Solutions
Ubiquiti Employee
Posts: 3,921
Registered: ‎12-10-2015
Kudos: 1379
Solutions: 299

Re: Switch Bandwith control from Router to CPE

@pguillem any change related to BW triggers the rules update on the device (e.g. altering the plan A to plan B, modifying the plan BW, modifying service device IP,...)
Usually, the rules update on CPE is done without a need to restart the device, but this can't be ensured for all devices / configurations / firmwares.

View solution in original post


All Replies
Ubiquiti Employee
Posts: 3,921
Registered: ‎12-10-2015
Kudos: 1379
Solutions: 299

Re: Switch Bandwith control from Router to CPE

Hi @pguillem
QoS, NetFlow, etc. always disables HW offloading, that's natural for any router. We know about the impact on the throughput and the ER team will continue the development of new firmwares which will handle this better. You can take a look into the ER forum section for more info: https://community.ubnt.com/t5/EdgeRouter/bd-p/EdgeMAX

2) Yes, in this case, QoS settings will be deleted from ER

3) Unfortunately, no pending jobs are implemented for this situation. Soon, automatic QoS settings will be handled by UNMS, first on ERs, later on CPEs. UNMS will handle this in a more advanced way than UCRM currently does.
Member
Posts: 130
Registered: ‎05-26-2015
Kudos: 20

Re: Switch Bandwith control from Router to CPE

I love Ubiquiti and will not switch to Microtik over this.. but it is a very important issue to tackle, which I would think is worth to invest some time in, as it dangerously breaks scalability with your own gear.

 

As for the answers:

 

2) Thanks!!! this is exactly the expected behavior!.

 

3) As of now, what actions should be manually performed in UCRM to trigger a network-wise replication of the bandwith limits?. To be exact... will altering the plan 1Mb up/down and saving the plan trigger such an event?

 

---- also... these are more than a hundred litebeams M5 here... will an BW update from UCRM trigger a 1 minute downtime on all CPEs?

Ubiquiti Employee
Posts: 3,921
Registered: ‎12-10-2015
Kudos: 1379
Solutions: 299

Re: Switch Bandwith control from Router to CPE

@pguillem any change related to BW triggers the rules update on the device (e.g. altering the plan A to plan B, modifying the plan BW, modifying service device IP,...)
Usually, the rules update on CPE is done without a need to restart the device, but this can't be ensured for all devices / configurations / firmwares.
Member
Posts: 130
Registered: ‎05-26-2015
Kudos: 20

Re: Switch Bandwith control from Router to CPE

[ Edited ]

@UBNT-Petr

 

Just a quick note to report what happened.

Migrating the bandwith traffic control to each CPE worked like a charm for our scenario and removed all speed issues we had been having.

 

1) I deactivated netflow, traffic accounting, deep paclet inspection... or any other fancy features that seemed resource intensive.

 

2) I turned off QoS. We have plenty of spare bandwith.

 

3) I turned on HWNAT to offload the routing, GREATLY improving throughput.

Conclussions:

 

- The ER-X-SFP is now 90% idle at 100Mbps and handles 20.000 connections like a breeze. Very well for a $70 USD router.

 

- The setup works as long as the CPEs remain turned ON whenever the rules update is triggered from UCRM.

 

- The UCRM benchmark for bandwith control on the gateway (ER-X-SFP) is 140 CODEL queues (70 customers) before stuff  getting PAINFULLY slow.

 

- Activating netflow from UCRM takes a HUGE toll on this router and is incompatible with more than 70 customers.

You might as well get away with 80.... but have in mind HWNAT is not possible with traffic accounting or QoS turned ON.

 

 

Perhaps it would be reasonable to think of a soft-limit (or at least a warning) on the option to do traffic control on the gateway, whenever people use this ER model and UCRM. I imagine models with more juice like the ER4 or ER6 have more power to handle a hundreds of CODEL queues in the way UCRM does.

 

It would definetly be a great idea to handle update triggers as pending tasks rather than instantaneous async calls. This way the CPEs that were OFF during an update would get their update upon turining on. This would make UCRM traffic control flawless and synched at all times.

 

I hope it serves someone else.

 

Happy new year.

Pedro

Highlighted
Ubiquiti Employee
Posts: 3,921
Registered: ‎12-10-2015
Kudos: 1379
Solutions: 299

Re: Switch Bandwith control from Router to CPE

Thanks for the info.

We are going to look into the performance impact of netflow and QoS. Hopefully, the numbers will be improved. Talking about UCRM and its network management, note that all of this stuff will be soon taken by UNMS (which will be integrated with UCRM), so it will handle the networking in a much better manner.