Upcoming Maintenance Alert:

The UBNT Community will be upgraded at 5pm MDT on April 25th. During this time the community forums will be set to read-only status.

Learn more

×
Reply
Member
Posts: 138
Registered: ‎06-10-2014
Kudos: 33

Re: UDP packet loss with EdgeRouter Lite

So no idea if the edgepoint is affected anyone?
Ubiquiti Certified - UCWA
New Member
Posts: 12
Registered: ‎08-24-2015
Kudos: 2

Re: UDP packet loss with EdgeRouter Lite


smtharrison wrote:
So no idea if the edgepoint is affected anyone?

edgepoint is based upon erx hardware ... so no i dont believe so.

New Member
Posts: 12
Registered: ‎08-27-2016
Kudos: 4

Re: UDP packet loss with EdgeRouter Lite

This post from Ubiquiti seems to provide some more details than what we have before. Man Happy

 

https://community.ubnt.com/t5/EdgeMAX/EdgeMAX-EdgeRouter-software-version-v1-9-7-has-been-released/m...

New Member
Posts: 12
Registered: ‎07-17-2017

Re: UDP packet loss with EdgeRouter Lite

What is the default value of packet-rx-core-num ?

New Member
Posts: 9
Registered: ‎02-03-2016
Kudos: 4
Solutions: 2

Re: UDP packet loss with EdgeRouter Lite

[ Edited ]

-1 on the two routers I've looked at.

 

edit - meaning that's the value in /sys/module/octeon_ethernet/parameters/max_rx_cpus

New Member
Posts: 33
Registered: ‎03-10-2014
Kudos: 3

Re: UDP packet loss with EdgeRouter Lite

So does the 1.9.7 workaround fix the problem for people? Currently running 1.7 and thinking of upgrading to get access to the workaround.
Member
Posts: 294
Registered: ‎12-17-2015
Kudos: 57
Solutions: 8

Re: UDP packet loss with EdgeRouter Lite

Unsure. I have mine set to 1 core... but I never overly saw the issue (at least consciously).
USG, UniFi AP AC Pro, UniFi US-8-150W, Unifi US-8
New Member
Posts: 18
Registered: ‎09-26-2014
Solutions: 1

Re: UDP packet loss with EdgeRouter Lite


the_slain_man wrote:
So does the 1.9.7 workaround fix the problem for people? Currently running 1.7 and thinking of upgrading to get access to the workaround.

It does, up to about 100Mbit for me at least. Anything above that, it still generates PL. I think I got something like 0.27% PL on 100Mbit.

1000Mbit though (what I have) it generates 88% for me, which is 11% higher than with both cores enabled. So it actually becomes worse.

 

I've switched it out for a cheap celeron primergy server with Sophos UTM for the time being. No drops at all anymore. But I really hope they'll fix this, because I'd prefer a low wattage product to a tower server....

Established Member
Posts: 1,200
Registered: ‎06-14-2012
Kudos: 987
Solutions: 80
Contributions: 9

Re: UDP packet loss with EdgeRouter Lite

[ Edited ]

ER-XG, running 1.9.7, core-num = 1

 

# set system packet-rx-core-num 1
[edit]
# commit
[ system packet-rx-core-num 1 ]
Setting 'max_rx_cpus' value to 1

Still getting OOO

iperf3 -c iperf.he.net -u -b 2G -6 -R -O 3
[snip]
iperf3: OUT OF ORDER - incoming packet = 2275873 and received packet = 2275874 AND SP = 4
iperf3: OUT OF ORDER - incoming packet = 2275875 and received packet = 2275876 AND SP = 4
iperf3: OUT OF ORDER - incoming packet = 2275957 and received packet = 2275958 AND SP = 4
[ 4] 9.00-10.00 sec 242 MBytes 2.03 Gbits/sec 0.007 ms 6107/177571 (3.4%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 4] 0.00-10.00 sec 2.34 GBytes 2.01 Gbits/sec 0.006 ms 65123/1760307 (3.7%)
[ 4] Sent 1760307 datagrams
[SUM] 0.0-10.0 sec 61481 datagrams received out-of-order

iperf Done.

Is there something else that should be done here?

Member
Posts: 294
Registered: ‎12-17-2015
Kudos: 57
Solutions: 8

Re: UDP packet loss with EdgeRouter Lite

My understanding is that a single core cannot handle that amount of bandwidth. If you clamped it down to 100-200mbit, it would probably do better.

At this point, I believe this is the only potential fix. We just have to wait for driver fix. :/
USG, UniFi AP AC Pro, UniFi US-8-150W, Unifi US-8
Established Member
Posts: 1,312
Registered: ‎08-06-2015
Kudos: 536
Solutions: 65

Re: UDP packet loss with EdgeRouter Lite


mrjester wrote:


[ 4] Sent 1760307 datagrams
[SUM] 0.0-10.0 sec 61481 datagrams received out-of-order


That's only 3.5% delivered OOO.  That very well could be occurring at any point between you and the iperf3 server (iperf.he.net).

 

You've also set a target bandwidth of 2Gbit/s, which is pretty high for an internet-based test.  Using he.net servers with a local gigabit ISP link, when setting target bandwidth to 1Gbit/s I often see a small percentage in both packet loss and packet reordering on UDP tests regardless of local equipment vendor.  This has been repeated at various locations with various local ISPs and is not considered a problem.  Note that this is my observered behavior and in no way means others would see the same, however.

 

The purpose of using such a test with iperf3 is indeed to actually cause such results.

 

UDP is not guaranteed and is delivered best-effort.  Expectation that 100% of packets are delivered 100% of the time in the absolute same order as they were sent is not reasonable.  If that is an expectation by a given application then that solution should use TCP rather than UDP.

 

 

Member
Posts: 294
Registered: ‎12-17-2015
Kudos: 57
Solutions: 8

Re: UDP packet loss with EdgeRouter Lite

So, there's also a bug in iperf previous to 3.1.

Not sure if you're hitting that or not.

https://github.com/esnet/iperf/issues/390
USG, UniFi AP AC Pro, UniFi US-8-150W, Unifi US-8
New Member
Posts: 12
Registered: ‎08-27-2016
Kudos: 4

Re: UDP packet loss with EdgeRouter Lite


waterside wrote:

UDP is not guaranteed and is delivered best-effort.  Expectation that 100% of packets are delivered 100% of the time in the absolute same order as they were sent is not reasonable.  If that is an expectation by a given application then that solution should use TCP rather than UDP.


That's true. In this case it is a known bug happening in these devices however. The statement I linked to earlier said the SoC vendor has promised to make use of the Octeon CPU's hardware support for fixing packet reordering in the next release of the offloading module. Let's hope that doesn't take to long. Man Happy

Established Member
Posts: 1,312
Registered: ‎08-06-2015
Kudos: 536
Solutions: 65

Re: UDP packet loss with EdgeRouter Lite

That's my point - just because there is any packet reording at all does not mean it is the noted issue, which I have been following since this long thread was first started.

 

There are too many that are running such internet-based UDP tests and are:

  1. Assuming that any OOO packets reported are the result of the noted issue
  2. Assuming that they should never have any OOO packets
  3. Assuming that any such OOO packets are in fact a problem

In this case, the documented workaround was noted to have been applied already to an ER-8-XG.  With that I don't know if we can really assume that the known issue is actually the cause.

 

A LAN-LAN test through the router with the iperf3 server on one direct-attached segment and the iperf3 client on the other direct-attached segment (IE: no other intervening routers) might be a more representative test.

 

New Member
Posts: 12
Registered: ‎08-27-2016
Kudos: 4

Re: UDP packet loss with EdgeRouter Lite


waterside wrote:

That's my point - just because there is any packet reording at all does not mean it is the noted issue, which I have been following since this long thread was first started.

 

There are too many that are running such internet-based UDP tests and are:

  1. Assuming that any OOO packets reported are the result of the noted issue
  2. Assuming that they should never have any OOO packets
  3. Assuming that any such OOO packets are in fact a problem

In this case, the documented workaround was noted to have been applied already to an ER-8-XG.  With that I don't know if we can really assume that the known issue is actually the cause.

 

A LAN-LAN test through the router with the iperf3 server on one direct-attached segment and the iperf3 client on the other direct-attached segment (IE: no other intervening routers) might be a more representative test.

 


I assume you are replying to me. I agree with you that there is likely to be some out of order packets on UDP, especially for internet traffic.

 

I was not talking about the workaround, I was talking about the fix that the SoC vendor is going to apply to their proprietary offload module as mentioned by a Ubiquiti employee in a response in the 1.9.7 release thread.

Established Member
Posts: 1,312
Registered: ‎08-06-2015
Kudos: 536
Solutions: 65

Re: UDP packet loss with EdgeRouter Lite


blunden wrote:

 

I was not talking about the workaround, I was talking about the fix that the SoC vendor is going to apply to their proprietary offload module as mentioned by a Ubiquiti employee in a response in the 1.9.7 release thread.


Yes I am well aware of the expected fix from Cavium in their offload module since it was mentioned in the Beta forums some time ago.  That isn't particularly relevant in this context as the documented workaround was already applied, which would also tend to suggest that the OOO packets may not be a result of the known issue (you stated that it in fact was)

 

That post also is regarding an ER-8-XG, which hasn't actually been confirmed to be affected.  The SoC in the Infinity is an entirely different series than the other ERs with different offload support.

 

I think we've killed this now so we can move on.

New Member
Posts: 9
Registered: ‎12-04-2014
Kudos: 1

Re: UDP packet loss with EdgeRouter Lite

[ Edited ]

Been waiting for almost 3 years now.

 

My EdgeRouter Lite (ERLite-3) in a box, on the shelf, collecting dust...

 

https://community.ubnt.com/t5/EdgeMAX/UDP-connection-timeout/m-p/1121563#M49539

 

Very suprised, that a class-action lawsuit has not resulted from this flaw.

 

If you look back through all of the postings and reports, you will see that Ubiquity staff
had little interest in the issue.

 

Several of the early postings clearly show that the Ubiquity staff
had no idea how to even re-create the issue in a lab.

 

Several people from the community gave step-by-step instructions, yet nothing was done.

 

My suggestion would be for Ubiquity post instructions on how we can return the affected

devices to receive a refund.

 

Yes - I did buy mine about 3 years ago - but only used it for about a week

until I figured out that it was causing major issues with my VoIP Traffic.


So keep telling the public that it MAYBE will be fixed in 2.0

Perhaps by then, someone will secure a class-action attorney and have a case started...

Member
Posts: 138
Registered: ‎06-10-2014
Kudos: 33

Re: UDP packet loss with EdgeRouter Lite

Yep.  Same here.  Mine are on the shelf too and after I got bit by this until it can get fixed.  I guess enough people are just living with it an not seeing it's effects?

Ubiquiti Certified - UCWA
Member
Posts: 294
Registered: ‎12-17-2015
Kudos: 57
Solutions: 8

Re: UDP packet loss with EdgeRouter Lite

Far as I can tell, on my home 200/10mbit connection, there really are no dicernable issues. I did enable the workaround in 1.9.7, but *shrugs* not sure if it actually made any difference. 

 

I don't doubt faster connections could have issues. Especially since the workaround would effect total bandwidth the router could handle. 

 

I don't overly blame Ubiquiti for this issue. It's the maker of the ER's CPU driver. They just never implemented the UDP reorder hardware in the driver. Yes, Ubiquiti could have taken it up sooner and gotten the vendor involved, but they didn't. They have now, and we hope a fix comes soon. Who knows, maybe after the fix happens I'll realize I was seeing this issue constantly and didn't realize it. Or, nothing will change <-- More likely.

USG, UniFi AP AC Pro, UniFi US-8-150W, Unifi US-8
Highlighted
Member
Posts: 138
Registered: ‎06-10-2014
Kudos: 33

Re: UDP packet loss with EdgeRouter Lite

Yeah, doubt a lawsuit would have much traction or help matters.  If that were the case they have been sued back when they got hacked a while back.  There are always bugs being found a fixed, though this is a pretty big one that renders these useless for those of us who were pushing any decent UDP traffic (which I wouldn't even call that much traffic... I was seeing it at 20 mbps).  I didn't even buy small ER, but an ER-8 and a Pro.  saw the issue on both.  expected a LOT more on those versions.  I think that's the beef most of us have with this issue not being attended to quickly.  It is what it is at this point.  I have plenty ubnt stuff out there that works great, and I'm kinda happy with the prices lol : )  Just have to really keep an eye on their stuff initially.

Ubiquiti Certified - UCWA
Reply