Regular Member
Posts: 525
Registered: ‎07-21-2010
Kudos: 93
Solutions: 6

Re: Offloading-Flow randomly "jumps" from Offload-Engine to Linux / ER-Pro8 / v.1.10.0

Had to change as_path.length <= 1 to get 100% performance all over the time.

 

As for now, we can not make IP-Upstream Routing descissions because 99% is routed via default-route.

That is not what a semi-professional ISP want.

 

I hope, ubiquiti will investigate into that issue, so we can restore our Full-BGP-Stuff.

 

performacnce.JPG

Member
Posts: 200
Registered: ‎04-28-2015
Kudos: 107
Solutions: 1

Re: Offloading-Flow randomly "jumps" from Offload-Engine to Linux / ER-Pro8 / v.1.10.0

I would be very curious to know whether running Bird in lieu of the Noction routing stack might be more judicious in its programming of the kernel fib, which I suspect is the trigger for the offload ejects.

 

Certainly getting into unsupported turf there, but...  It would be interesting to know.

 

I'm not a fan of the Noction routing solution, but I understand why UBNT went there.  In theory.

Ubiquiti Employee
Posts: 1,246
Registered: ‎07-20-2015
Kudos: 1500
Solutions: 82

Re: Offloading-Flow randomly "jumps" from Offload-Engine to Linux / ER-Pro8 / v.1.10.0

@rps
> This offload information seems very useful. Can we get a "show ubnt offload statistics" option added to EdgeOS baseline?
This info adds small overhead then forwarding each packet that's why it is disabled by default. I think we can add CLI commands to enable/show/disable offloading stats

 

@MattHardeman
> I do, however, wonder if the FIB update routine doesn't send a "route has changed" message of some form upon FIB updates. Something along the lines of path to "a.b.c.d/xx has changed, handle appropriately" doesn't get passed as it occurs to the offload engine, which might then scrub current offloaded flows for flows which have that destination, and eject those flows from offload?
No, there's not such functionality in offload engine

 

@MattHardeman
> Can you say what the number-of-flows (for offload) entries are in the Cavium chipsets as implemented in the ER-Pro8 and on the ER-Infinity?
I was investigating offloading source code and here are my findings:

  1. Size of flow hashtable is constant for ALL Cavium-based ER - 8K buckets
  2. There are 3 flows in each bucket
  3. Total amount of flows is 8K*3=24K and it consumes ~1.3Mb of RAM
  4. New flow pushes oldest flow out of the bucket even if it is not expired yet <---- BINGO !!! @RcRaCk2k

I believe that root cause of the issue is that 24K simultaneous flows for all ER platforms is wrong. Size of offloading table should be customizible and default value should be platform dependent, something like this:

  • ER-Lite,ER-PoE (512Mb RAM) - 24K flows
  • ER-4/ER-6 - (1Gb RAM) - 48K flows
  • ER/ER-Pro - (2Gb RAM) - 96K flows
  • ER-Infinity - (16Gb RAM) - 768K flows


@RcRaCk2k - I can create new "cavium_ip_offload.ko" module with bigger flow hashtable (96K instead of 24K) and we can check if it fixes your issue. Do you agree to deploy new "cavium_ip_offload.ko" on your system?

 

Member
Posts: 200
Registered: ‎04-28-2015
Kudos: 107
Solutions: 1

Re: Offloading-Flow randomly "jumps" from Offload-Engine to Linux / ER-Pro8 / v.1.10.0

@UBNT-afomins,

 

Thanks for answering the questions.

 

It's interesting that there's not a flow eject on route change.  If I understand correctly, an already existing flow would continue to flow via the old route until such time as the flow is expired out?

 

I definitely think increasing the flow count on the larger platforms would be a major benefit.  For ISPs and WISPs using these prodcuts as provider-edge routers, this will make the offload functionality far more useful.

Regular Member
Posts: 525
Registered: ‎07-21-2010
Kudos: 93
Solutions: 6

Re: Offloading-Flow randomly "jumps" from Offload-Engine to Linux / ER-Pro8 / v.1.10.0

@UBNT-afomins

>> I do, however, wonder if the FIB update routine doesn't send a "route has changed"

> No, there's not such functionality in offload engine

 

Sorry, that makes no sense to me. That will screw up the complete routing logic.

Also my tests prove that ribd issues a flush() command to the flow-table.

 

Current-Statistic on real network load on production system:

100percent.JPG

unix-stats.JPG

 

100% of Hits on both systems after 138.265.140.965 / 89.503.998.154 bytes of traffic.

Solution: Stopp BGP-Updates by only read as_path.length <= 1

 

Our /proc/dev/net statistics prove that result. After commit this changes including BGP-Clear for IN-Table you can not see counters increasing.

 

@UBNT-afomins

> I can create new "cavium_ip_offload.ko" module with bigger flow hashtable (96K instead of 24K) and we can check if it fixes your issue. Do you agree to deploy new "cavium_ip_offload.ko" on your system?

Yes of course. I think for a first alpha version that is okay :-) But you should add a option to that module, so you can pass it on modprobe hashtable_size=96000.

 

@matthardeman

> Prefix A: 10.20.0.0/16, Prefix B: 10.20.1.0/24, Prefix C: 10.20.1.4/24

Wow, you are right. That point i have missed. But for the first step, it will be good to just take the updated prefix into account. That is not so much pain as flush to whole table.

 

 

 

Veteran Member
Posts: 7,802
Registered: ‎03-24-2016
Kudos: 2032
Solutions: 894

Re: Offloading-Flow randomly "jumps" from Offload-Engine to Linux / ER-Pro8 / v.1.10.0

What I don't understand:

 

was getting errors about 256k nf_table limit, so I figure his average number of flows should be above 200k  region.

 

But after BGP changes , the 24k size offload works pretty well,  but this offload table can roughly contain only 10...15% of his flows

Member
Posts: 200
Registered: ‎04-28-2015
Kudos: 107
Solutions: 1

Re: Offloading-Flow randomly "jumps" from Offload-Engine to Linux / ER-Pro8 / v.1.10.0


@16againwrote:

was getting errors about 256k nf_table limit, so I figure his average number of flows should be above 200k  region.

 

But after BGP changes , the 24k size offload works pretty well,  but this offload table can roughly contain only 10...15% of his flows


You have to be careful there:

 

You would seem to assume that each flow should, on average, represent an equal proportion of packets over a period or an equal proportion of total bandwidth.  In actually, a small minority of flows tend to be the heavy hitters, while many are essentially quiet background noise.

 

It wouldn't shock me that a router aggregating flows for typical users might see the vast majority of its utilization on 10% of flows.

Member
Posts: 124
Registered: ‎08-24-2015
Kudos: 20

Re: Offloading-Flow randomly "jumps" from Offload-Engine to Linux / ER-Pro8 / v.1.10.0

I think @broken has solved many issues that the ERP was having by using BIRD instead

 

https://community.ubnt.com/t5/EdgeMAX-Stories/EdgeRouter-as-BGP-router/cns-p/1716321

 

perhaps @

 

Regular Member
Posts: 615
Registered: ‎04-12-2008
Kudos: 112
Solutions: 2

Re: Offloading-Flow randomly "jumps" from Offload-Engine to Linux / ER-Pro8 / v.1.10.0

Yes. BIRD works pretty well on EdgeRouter PRO with minor changes in configuration it pushes ERPro to its limits.
We have tested in our network with more than 2k users and it seems to be stable enough passing 1Gbit easily from port to port without any lags and drops. Since you will not cross above 1Gbit per port it works as it should Man Happy We have tested IPTV services with ER-PRO in configuration where ER-Pro pass TV traffic around 960Mbit/s and it worked fine for more than a year without errors. When we have crossed 1Gbit limit we started to use XG switch for routing purposes Man Wink Just only for TV traffic - it works pretty fine too.

For us next step is to switch between ER-Pro to ER-XG and use current configuration of the BIRD when we cross 1Gbit/s of internet traffic from one upstream. Due to the lack of availability of ER-XG we have done internal BGP network with bond few of ERPro between upstreams that towards traffic between another few ER-Pro that are being used as BRAS. And... it works fine without any major issues. There is a some lack of CPU power then we need to use HTB and make a queues for particular customers - regular HTB scripts consume a lot of CPU usage so enabling queues drops ERPro performance dramatically to around ~300-400Mbit/s depends how many queues we have.
devices.pl
Veteran Member
Posts: 7,802
Registered: ‎03-24-2016
Kudos: 2032
Solutions: 894

Re: Offloading-Flow randomly "jumps" from Offload-Engine to Linux / ER-Pro8 / v.1.10.0

@matthardemanwrote:

 In actually, a small minority of flows tend to be the heavy hitters, while many are essentially 
quiet background noise.

Indeed top 10% of flows might generate >90% of the traffic

 

But is there an algoritm in place, that makes sure these "heavy hitters" get their place in offload tables instead of the almost idle flows ?

Member
Posts: 200
Registered: ‎04-28-2015
Kudos: 107
Solutions: 1

Re: Offloading-Flow randomly "jumps" from Offload-Engine to Linux / ER-Pro8 / v.1.10.0


@16againwrote:

Indeed top 10% of flows might generate >90% of the traffic

 

But is there an algoritm in place, that makes sure these "heavy hitters" get their place in offload tables instead of the almost idle flows ?


I can't say, as I've not actually reviewed the offload engine source.  I believe that's part of the proprietary bundle.

 

What I can suggest is via logical process:

 

The performance of an offload engine is all about identifying flows with high unique packet count per unit time.  It would not shock me if the logic which ejects an older flow to make room for a new one considers both age of the candidate old flow as well as recent pps of the old flow.


Even given equivalent total bandwidth transferred, you want the high packets-per-second flow to be the one in the offload engine if there's only room for one or the other.

 

Again, my theory on that matter in response to your question is based upon suppositions of behaviors that both logically fit to the overall goals of a router offload performance enginge and may simultaneously help explain the cases of the observed behavior being reported.

 

Veteran Member
Posts: 7,802
Registered: ‎03-24-2016
Kudos: 2032
Solutions: 894

Re: Offloading-Flow randomly "jumps" from Offload-Engine to Linux / ER-Pro8 / v.1.10.0

Indeed optimal to have heaviest flows offloaded, but I wouldn't be surprised if that logic isn't build in or used yet.

As I read link below, oldest flows are pushed out to fit new.

https://community.ubnt.com/t5/EdgeMAX/Offloading-Flow-randomly-quot-jumps-quot-from-Offload-Engine-t...

Regular Member
Posts: 525
Registered: ‎07-21-2010
Kudos: 93
Solutions: 6

Re: Offloading-Flow randomly "jumps" from Offload-Engine to Linux / ER-Pro8 / v.1.10.0

Hi guys,

 

watch that video, i do not have to tell you more.

This video speeks for him self.

 

 

Over night we processed 1,7 TB on ER-BGP1 and 1 TB on ER-BGP2. Offloading-Hits 100%.

While sleeping i got a brilliant idea. Insert a route manually and remove it after that. Do that on a /32 IP you never use and lets see what Offload-Engine will do. BINGO! It will flush the whole flow-table, so offline-engine have to lern from point zero.

 

@UBNT-afomins who have developed that source code? I was a programmer for 20 years and i was always knowing what logic i code and witch side-effects my programming-locig will have, but that type of locig i will never ever implement. That was a shoot in your own feets. I hope your team will correct that logic as far as possible so EdgeMAX can perform as you suggest in your videos and your datasheets.

Veteran Member
Posts: 7,802
Registered: ‎03-24-2016
Kudos: 2032
Solutions: 894

Re: Offloading-Flow randomly "jumps" from Offload-Engine to Linux / ER-Pro8 / v.1.10.0

@RcRaCk2k

Last video clearly shows problem...

but how many flows do you have, when offload efficiency is at 100% ?

Regular Member
Posts: 525
Registered: ‎07-21-2010
Kudos: 93
Solutions: 6

Re: Offloading-Flow randomly "jumps" from Offload-Engine to Linux / ER-Pro8 / v.1.10.0

[ Edited ]

There is no statistic available from cavium-stats, so i can't tell you how many flows are installed.

I hope we get more stats to see more on what is going on inside the offloading-engine.

 

We have around 2.366 Packet/s bypassed to kernel.

 

I do not know if that all are TCP-SYN and new UDP-Flows. I have to do some research while watching packets with tcpdump and analyze them.

 

@UBNT-afomins Please add a /proc/cavium/flowtable where we can see stats of packets and bytes per flow. So we are able to identifiy DDoS Attacks. We use exaBGP to insert Blackhole-Routes when we are able to detect a high bandwidth attack / high packets-attack / random-udp-port and tcp-syn attacks. With offloading enabled, we are not able to see what is going on on the network stuff.

 

Also statistics like /proc/dev/net are welcome. If you enable offloading, your traffic-chart in webui is useless.

 

create counters for 5 interfaces can not hurt the system performance so much.

 

EDIT: i did some packet inspection on not offloaded packets and i found something interesting in it:

 

  1. TCP-SYN, TCP-RST and TCP-FIN are handled by Linux (bypassed by offload-engine)
  2. Protocol 50 (ESP) so IPSEC is not offloaded, i got many ESP-Packets
  3. Also Protocol 47 (GRE ESP) so GRE-Tunnel is not offloaded, but Source and Destination was not my router! So Offload-Engine can easily forward this packet without any problem. It have only to know Egress-Interface and Destination-MAC-Address of next router.
  4. ICMP between two hosts is not offloaded
  5. GRE is not offloaded if it is in VLAN.
    GRE-Offloading is enabled, but if your GRE is in a VLAN, only VLAN is handled by Offload-Engine and GRE is handled by Linux. Offloading-Engine should do 2-cycles to process both protocols in hardware.
  6. I can see some Google-QUIC Packets that are not offloaded but normal UDP-Packets. But that may not be related to a offloading-issue because it is Protocol 17 (UDP)
Veteran Member
Posts: 7,802
Registered: ‎03-24-2016
Kudos: 2032
Solutions: 894

Re: Offloading-Flow randomly "jumps" from Offload-Engine to Linux / ER-Pro8 / v.1.10.0

1 I'd expect Linux to handle TCP-SYN, TCP-RST and TCP-FIN, as these packets initiate housekeeping.

Like create new & Clean up old connections, add/remove from offload table.

 

2  I was aware of that.

but as GRE is offloadable,  ESP should be too.

3 / 5:  GRE = 47 

New to me that VLAN + GRE is a no-go. 

Why the 2 cycles approach?  That would be also necessary for VLAN + TCP combo

If untagged traffic gets "empty" VLAN tag, single hash cycle could work

6 Lost track of topics....but I know ER has/had lots of QUIC UDP problems.

 

 

btw: How do you see which traffic is and isn't offloaded?

Regular Member
Posts: 525
Registered: ‎07-21-2010
Kudos: 93
Solutions: 6

Re: Offloading-Flow randomly "jumps" from Offload-Engine to Linux / ER-Pro8 / v.1.10.0

[ Edited ]

@16again

> btw: How do you see which traffic is and isn't offloaded?

Because only not offloaded traffic shows up on tcpdump.

New-Feature-Request: get a new interface "monitoring0" that is connected to offload-engine, so you can enable port-mirroring to dump packets that are processed by Offloading-Engine, and you can do pmacct on that interface for accounting and other stuff.

 

> GRE = 47 

I have corrected that, thank you.

 

> New to me that VLAN + GRE is a no-go.

Sorry but that is what i can see. I have SNMP-Traffic Request/Response in my dump. GRE-Tunnel arrives on eth0.100 and is destinated to my BGP-Router, inner IPv4 SRC/DST is Monitoring-Server <> Switch. If this packet will be offloaded, then i could not see it in tcpdump.

 

> Why the 2 cycles approach?  That would be also necessary for VLAN + TCP combo

> If untagged traffic gets "empty" VLAN tag, single hash cycle could work

Thats right, so the offload-engine should take a deeper look into the packet transported in VLAN. So there should no need to do unpacking in two cycles. It can do unpacking on the fly.

 

> but I know ER has/had lots of QUIC UDP problems.

wow, why. It is a normal UDP-Packet. Okay, strange things going on here.

 

............ I have a new slogan for Ubiquiti :-) :-P

< Where future begins

> Where strange things going on

Veteran Member
Posts: 7,802
Registered: ‎03-24-2016
Kudos: 2032
Solutions: 894

Re: Offloading-Flow randomly "jumps" from Offload-Engine to Linux / ER-Pro8 / v.1.10.0

from memory:

QUIC issue had to do with packet re-ordering 

If not all packets are offloaded, it's obvious packets handled by CPU travel slower

Also, previous OS versions used multiple cores for routing UDP, also introducing out of order troubles.

 

Any brand will have its troubles (remember link below?) , it's how issues are handled by their support departments.

https://www.theregister.co.uk/2017/02/03/cisco_clock_component_may_fail/

Ubiquiti Employee
Posts: 1,246
Registered: ‎07-20-2015
Kudos: 1500
Solutions: 82

Re: Offloading-Flow randomly "jumps" from Offload-Engine to Linux / ER-Pro8 / v.1.10.0

@RcRaCk2k
> But you should add a option to that module, so you can pass it on modprobe hashtable_size=96000.
Yes, that will be added in final release

 

> Do that on a /32 IP you never use and lets see what Offload-Engine will do. BINGO!
> It will flush the whole flow-table, so offline-engine have to lern from point zero.
I was able to reproduce this - after adding new route offload flow table is flushed. I was not aware of this functionality until you pointed this out, thanks. We will think of more elegant solution to this issue rather than flushing all flow table.

 

 

> I hope your team will correct that logic as far as possible so EdgeMAX can perform as you suggest in your videos and your datasheets.
We are investigating this right now.

 

I will provide new "cavium_ip_offload.ko" for testing once we figure out how to avoid flushing flow-table when FIB changes. 

Regular Member
Posts: 525
Registered: ‎07-21-2010
Kudos: 93
Solutions: 6

Re: Offloading-Flow randomly "jumps" from Offload-Engine to Linux / ER-Pro8 / v.1.10.0

@UBNT-afomins

 

Our BGP-Router only forwards traffic without manipulation (natting, firewalling, and so on). Is there a way, that the offload-engine checks for firewall / filter-rules / nat-rules on first packet and if there is no filter that will ever interact with src-ip-address and dst-ip-address to only insert a ip-forwarding-flow?

 

So lets say your hash-table looks like that structure:

 

IPv4 without TCP/UDP Information (forward packet directly as a switch would do)

{ ipproto: 0x0800, from-if: 0x00000001, to-if: 0x000000004, from-mac: 0x00163e021104, to-mac: 0x3c8ab0d5f481, from-ip: 0xb93a1f0c, to-ip:0x50ff0f89 }

 

Requirements:

  • There is no filter / raw / mangle / nat rule that was filtering on IP-Address ANY or explicit that IP-Address or Network-Range of that stream

So your offload-engine do not have to send first packet of each flow to linux user-space / kernel.

 

In case of a DDoS-Attack to a customer behind 5 EdgeMax Routers, there will be no load on the ERs-CPU after first packet.

Also a UDP-Flood or TCP-Syn-Flood will passed directly. Also your flow-table is at a minimum.

 

If your firewall rules are complete flushed, and you have 500 Customers you have only 500 entries in flow-table.

 

That is exactly our business needs.

 

Filtering and Natting is only done on customers PE.

 

I think that logic should be possible to implement to your offload-engine logic.