Upcoming Maintenance Alert:

The UBNT Community will be upgraded at 5pm MDT on April 25th. During this time the community forums will be set to read-only status.

Learn more

×
Reply
New Member
Posts: 18
Registered: ‎09-05-2015
Kudos: 8

UDP packet loss with EdgeRouter Lite

[ Edited ]

Hi,

 

I've been using ERL for a few months now, and I'm pretty happy with what I'm getting out of it.

 

However, I play some FPS games sometimes, and I've noticed that I run into packet loss, so I finally decided to look into it today. My normal setup looks like this:

 

Gaming PC > Dumb Switch > ERL > WAN

 

My WAN is symmetric 400 Mbit/s, all equipment is 1 Gbit/s. Here's what I get with iperf: 

 

> iperf3.exe -c ping.online.net -u -b 10000000
Connecting to host ping.online.net, port 5201 [ 4] local 192.168.1.3 port 50423 connected to 62.210.18.40 port 5201 [ ID] Interval Transfer Bandwidth Total Datagrams [ 4] 0.00-1.00 sec 1.08 MBytes 9.04 Mbits/sec 138 [ 4] 1.00-2.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 2.00-3.00 sec 1.19 MBytes 9.96 Mbits/sec 152 [ 4] 3.00-4.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 4.00-5.00 sec 1.19 MBytes 9.95 Mbits/sec 152 [ 4] 5.00-6.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 6.00-7.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 7.00-8.00 sec 1.19 MBytes 9.95 Mbits/sec 152 [ 4] 8.00-9.00 sec 1.20 MBytes 10.0 Mbits/sec 153 [ 4] 9.00-10.00 sec 1.19 MBytes 9.96 Mbits/sec 152 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 0.00-10.00 sec 11.8 MBytes 9.90 Mbits/sec 2.203 ms 389/1511 (26%) [ 4] Sent 1511 datagrams iperf Done.

 

This is with 1.7.0 firmware. With 1.6.0, it was closer to 10%. The results don't change, if I plug my PC directly into ERL, without a switch. However, if plug it into WAN, I get 0 lost datagrams: 

 

> iperf3.exe -c ping.online.net -u -b 10000000
Connecting to host ping.online.net, port 5201
[  4] local 46.109.20.181 port 53472 connected to 62.210.18.40 port 5201
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec  1.08 MBytes  9.04 Mbits/sec  138
[  4]   1.00-2.00   sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]   2.00-3.00   sec  1.19 MBytes  9.96 Mbits/sec  152
[  4]   3.00-4.00   sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]   4.00-5.00   sec  1.19 MBytes  9.96 Mbits/sec  152
[  4]   5.00-6.00   sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]   6.00-7.00   sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]   7.00-8.00   sec  1.19 MBytes  9.96 Mbits/sec  152
[  4]   8.00-9.00   sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]   9.00-10.00  sec  1.19 MBytes  9.96 Mbits/sec  152
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.00  sec  11.8 MBytes  9.90 Mbits/sec  1.897 ms  0/1511 (0%)
[  4] Sent 1511 datagrams

iperf Done.

 

With direct WAN connection, the loss only appears after 100+ Mbit/s.

 

So this is a confirmation of ERL being the culprit. However, I have no idea how to address this. My WAN is nowhere near being congested, especially during these tests. The loss gets lower the lower bandwidth I use with iperf, but it doesn't disappear (it even manages to somehow lose 1 packet out of 16, when sending 100 Kbit/s...), which wasn't the case with 1.6.0, by the way, but that's secondary (it got proportionally worse with an upgrade to 1.7.0).

 

My config is really simple: 

 

firewall {
    all-ping enable
    broadcast-ping disable
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians enable
    name WAN-IN {
        default-action drop
        description ""
        rule 1 {
            action accept
            description "established / related"
            log disable
            protocol all
            state {
                established enable
                invalid disable
                new disable
                related enable
            }
        }
    }
    name WAN-LOCAL {
        default-action drop
        description ""
        rule 1 {
            action accept
            description "established / related"
            log disable
            protocol all
            state {
                established enable
                invalid disable
                new disable
                related enable
            }
        }
    }
    receive-redirects disable
    send-redirects enable
    source-validation disable
    syn-cookies enable
}
interfaces {
    ethernet eth0 {
        address dhcp
        description WAN0
        duplex auto
        firewall {
            in {
                name WAN-IN
            }
            local {
                name WAN-LOCAL
            }
        }
        speed auto
    }
    ethernet eth2 {
        address 192.168.1.1/24
        description LAN
        duplex auto
        speed auto
    }
}
system {
conntrack {
expect-table-size 4096
hash-size 4096
table-size 32768
tcp {
half-open-connections 512
loose enable
max-retrans 3
}
}
}

 

With some basic DHCP mapping, port forwarding, etc. omitted. No traffic policies.

 

I know that people suggest implementing some packet shaper, when in need of UDP QoS, but isn't it helpful only in cases when there's not enough bandwidth? I'm the only user of my network, I don't have any network-intensive stuff running in the background when I'm playing / testing (and it shouldn't matter anyway, with 400 Mbit/s network). Increased latency would be tolerable, but packet loss?

 

Any help would be greatly appreciated.

 

UPDATE 1: Installed 1.8.0alpha1, the loss is almost gone, so that's a relief (was on the verge of buying myself a R7000 and forgetting about all this). The worst I've seen so far is 14 lost out of 1511 at 10 Mbit/s, but usually it's only 1-2 packets per 1511. You did something right, so thanks for that, but there's still room for improvement: direct connection consistently produces 0% loss.

 

UPDATE 2: Installed 1.8.0beta3, back to the old 20-30% packet loss...

 

UPDATE 3: Installed 1.9.0alpha1, no changes...

New Member
Posts: 18
Registered: ‎09-05-2015
Kudos: 8

Re: UDP packet loss with EdgeRouter Lite

Sorry, but... bump?

Member
Posts: 220
Registered: ‎08-15-2015
Kudos: 36
Solutions: 2

Re: UDP packet loss with EdgeRouter Lite

I'd be interested in knowing what might be going on with this, too.

 

iperf run from home. ERLite3, Comcast Business HSI, 17/3, SMC8014 cmodem:

 

$ iperf -c ping.online.net -u -b 10000000
------------------------------------------------------------
Client connecting to ping.online.net, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  3] local 172.16.104.35 port 34798 connected with 62.210.18.40 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  11.9 MBytes  10.0 Mbits/sec
[  3] Sent 8505 datagrams
[  3] Server Report:
[  3]  0.0-10.7 sec  4.45 MBytes  3.50 Mbits/sec   3.836 ms 5328/ 8504 (63%)
[  3]  0.0-10.7 sec  1 datagrams received out-of-order

Docs are almost non-existent for this app, so I don't know if that's 63% loss or 37%, but it's not good, either way.

 

iperf run from my Digital Ocean droplet:

 

$ iperf -c ping.online.net -u -b 10000000
------------------------------------------------------------
Client connecting to ping.online.net, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  3] local 104.236.115.46 port 38345 connected with 62.210.18.40 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  11.9 MBytes  10.0 Mbits/sec
[  3] Sent 8505 datagrams
[  3] Server Report:
[  3]  0.0-10.0 sec  11.9 MBytes  10.0 Mbits/sec   0.036 ms    0/ 8505 (0%)

Hmmm...

 

Jim

New Member
Posts: 18
Registered: ‎09-05-2015
Kudos: 8

Re: UDP packet loss with EdgeRouter Lite

Have you tried running this without ERLite between you and WAN, Jim? It just seems like you're exceeding your bandwidth (3.5 Mbit/s). Try this:

 

iperf -c ping.online.net -u -b 3000000

 

Member
Posts: 220
Registered: ‎08-15-2015
Kudos: 36
Solutions: 2

Re: UDP packet loss with EdgeRouter Lite

Oh, for the love of Pete! Man Sad

 

$ iperf -c ping.online.net -u -b 3000000
------------------------------------------------------------
Client connecting to ping.online.net, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  3] local 172.16.104.35 port 39952 connected with 62.210.18.40 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  3.58 MBytes  3.00 Mbits/sec
[  3] Sent 2552 datagrams
[  3] Server Report:
[  3]  0.0-10.0 sec  3.58 MBytes  3.00 Mbits/sec   3.520 ms    0/ 2551 (0%)
[  3]  0.0-10.0 sec  1 datagrams received out-of-order

Even...

 

$ iperf -c ping.online.net -u -b 3500000
------------------------------------------------------------
Client connecting to ping.online.net, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  3] local 172.16.104.35 port 55499 connected with 62.210.18.40 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  4.17 MBytes  3.50 Mbits/sec
[  3] Sent 2978 datagrams
[  3] Server Report:
[  3]  0.0-10.0 sec  4.17 MBytes  3.50 Mbits/sec   3.629 ms    0/ 2977 (0%)
[  3]  0.0-10.0 sec  1 datagrams received out-of-order

Like I said: Docs are essentially non-existent, near as I could tell.  I tried to find out what that "10000000", finally gave up and just used it.  I suppose I should have tried harder Man Tongue

 

Jim

New Member
Posts: 18
Registered: ‎09-05-2015
Kudos: 8

Re: UDP packet loss with EdgeRouter Lite

Now let's hope somebody else will come in and point out my mistake Lol

Member
Posts: 220
Registered: ‎08-15-2015
Kudos: 36
Solutions: 2

Re: UDP packet loss with EdgeRouter Lite


jeremejevs wrote:

Now let's hope somebody else will come in and point out my mistake Lol


Hey, if nothin' else: Ya got a few free bumps for my ignorance & failure to RTFM (wherever TFM is) Man Very Happy

 

Jim

 

SuperUser
Posts: 11,956
Registered: ‎12-08-2008
Kudos: 8701
Solutions: 544
Contributions: 1

Re: UDP packet loss with EdgeRouter Lite

[ Edited ]

Couple of things - for documentation try    man iperf    Biggrin

 

The UDP tests work very differently from TCP tests, and in fact packet loss is expected in the test.   The whole idea is to spit out packets at a faster rate until the packet loss hits some level.   That is how you determine the rate at which you get packet loss, which is the whole point of the test.   With TCP the tcp backoff algorithm handles this, but since udp is an unreliable protocol and doesn't have any ack-like or re-transmission correction mechanism (by design...) at some point packets get dropped.    Might want to look this all up in Stephens or Comer or some other Networking textbooks to refresh your memory.

 

Anyway, the -b parameter sets the target speed for the test to go to.   If you set it lower than where the network starts to loose packets, you get zero packet loss.   Set it higher, and you get loss.   Pretty simple.   And nothing to do per-se with the ER.

Jim

" How can anyone trust Scientists? If new evidence comes along, they change their minds! " Politician's joke (sort of...)

"Humans are allergic to change..They love to say, ‘We’ve always done it this way.’ I try to fight that. "Admiral Grace Hopper, USN, Computer Scientist
Member
Posts: 220
Registered: ‎08-15-2015
Kudos: 36
Solutions: 2

Re: UDP packet loss with EdgeRouter Lite


eejimm wrote:

Couple of things - for documentation try    man iperf    Biggrin

Jim


 

Yeah, I see that, now Man Tongue  I know about "man." I've written man pages.  Many times.  I was distracted, saw the " -B, --bind <host>" bit, thought "WTF?" and just blindly carried on.  Yeah: Doh!

 

Jim

 

New Member
Posts: 18
Registered: ‎09-05-2015
Kudos: 8

Re: UDP packet loss with EdgeRouter Lite

UPDATE: Installed 1.8.0alpha1, the loss is almost gone, so that's a relief (was on the verge of buying myself a R7000 and forgetting about all this). The worst I've seen so far is 14 lost out of 1511 at 10 Mbit/s, but usually it's only 1-2 packets per 1511. You did something right, so thanks for that, but there's still room for improvement: direct connection consistently produces 0% loss.

New Member
Posts: 18
Registered: ‎09-05-2015
Kudos: 8

Re: UDP packet loss with EdgeRouter Lite

UPDATE 2: Installed 1.8.0beta3 just now, back to the old 20-30% packet loss...

Established Member
Posts: 2,536
Registered: ‎10-13-2012
Kudos: 670
Solutions: 89

Re: UDP packet loss with EdgeRouter Lite

-b, --bandwidth n[KM]set target bandwidth to n bits/sec (default 1 Mbit/sec for UDP, unlimited for TCP). If there are multiple streams (-P flag), the bandwidth limit is applied separately to each stream. You can also add a '/' and a number to the bandwidth specifier. This is called "burst mode". It will send the given number of packets without pausing, even if that temporarily exceeds the specified bandwidth limit. Setting the target bandwidth to 0 will disable bandwidth limits (particularly useful for UDP tests).

 

New Member
Posts: 18
Registered: ‎09-05-2015
Kudos: 8

Re: UDP packet loss with EdgeRouter Lite

Sorry, I don't understand what you're getting at.

Established Member
Posts: 2,536
Registered: ‎10-13-2012
Kudos: 670
Solutions: 89

Re: UDP packet loss with EdgeRouter Lite

Test with "-b 0"

New Member
Posts: 18
Registered: ‎09-05-2015
Kudos: 8

Re: UDP packet loss with EdgeRouter Lite

I don't understand how that helps. As described in the original post, my ERL can't even handle 10 Mbit/s; why would I try maxing out my network interface, which is ~100 times more bytes per second?

Established Member
Posts: 2,536
Registered: ‎10-13-2012
Kudos: 670
Solutions: 89

Re: UDP packet loss with EdgeRouter Lite

[ Edited ]

To find out where your speed limit is.

 

Anyway, something is obviously wrong. It could be the NIC in the computer you test with, its driver, or a damaged ERL port, or a damaged ERL as such, or ... have you checked the negotiated speed, for example on both sides of the Ethernet cable? Did you try a different ERL port for a) WAN and b) LAN? Did you try a different computer/NIC? Why do you get packet loss when directly on WAN already after 100+ when your speed is supposed to be 400? How long is your cable, is it copper or an aluminum alloy? How far is

ping.online.net

 from you (traceroute)? What if you do "mtr ping.online.net"? ...

New Member
Posts: 18
Registered: ‎09-05-2015
Kudos: 8

Re: UDP packet loss with EdgeRouter Lite

[ Edited ]

Thanks for guesses.

 

  • Packet loss keeps growing if I go above 10 Mbit/s, up to ~84% loss at 952 Mbits/s (which is actually interesting, because that means at least 150 Mbit/s of traffic does get through).
  • I'm testing with two completely different pieces of hardware: I217-V on my PC and 82579LM on my laptop, both producing similar results.
  • I don't think my ERL is damaged in any way, because the packet loss was below 1% with 1.8.0alpha1, as noted above.
  • I'm downloading / uploading a lot as a part of my workflow (pulling / pushing Docker images, libraries / packages / modules, etc.), I'd notice if my connection had degraded (a speed test I ran just now).
  • ERL ports shouldn't matter, since the same configuration worked fine under 1.8.0alpha1, but I did try using different ports, to no avail.
  • I don't know who exactly is losing my packets on a direct connection, but there are many potential reasons for that, since UDP isn't reliable. I've ran some speed tests to a few French targets, and it seems that my ISP doesn't have a fast upload route there, the best result being 88 Mbit/s (the rest being around 40 Mbit/s), so the loss above 100 Mbit/s is understandable.
  • Both cables are Cat 5e, 3 meters long, different manufacturers, one is shielded and the other one isn't, copper. The short cables, connecting all my network devices, are the same as the shielded one, but 30cm long.
  • Traceroute says that ping.online.net is 7 hops away from me, ~50ms latency, MTR agrees (and its packet loss detection is useless, since those aren't UDP packets, AFAIK).

But main point is, it's almost certainly ERL's fault, since there are no issues with a direct connection (I don't consider UDP loss on 100+ Mbit/s as an issue) and 1.8.0alpha1 did make things better, while 1.8.0beta3 has returned them to the initial state.

Established Member
Posts: 2,536
Registered: ‎10-13-2012
Kudos: 670
Solutions: 89

Re: UDP packet loss with EdgeRouter Lite

[ Edited ]

If you have two computers available, then how about you test with one before and one behind the ERL. Ah, and before doing that try with and without hardware offloading. What test results do get with tcp instead of udp? And what speeds to you see reported on the ERL itself (dashboard, if that remains somewhat responsive)?

 

See if there is a change if you disable all firewall rules for the duration of the test.

 

There are many people using 1.8beta1-3 with normal speeds, which means there is something different with your setup. You could try a factory reset as well, and then start with a clean and absolutely (!) minimal config (do not upload old settings).

New Member
Posts: 18
Registered: ‎09-05-2015
Kudos: 8

Re: UDP packet loss with EdgeRouter Lite

[ Edited ]

Alright, makes sense, I'll try doing a LAN test sometime, probably in the second half of February (and a clean setup too). Can't afford wasting more time on this right now, unfortunately, since this is affecting only gaming. All I wanted to do today was report that 1.8.0beta3 got worse.

 

About "There are many people using 1.8beta1-3 with normal speeds" - my speeds are normal too, which is evident from the speed test links. Ookla's test uses TCP though. I've just ran a TCP iperf test against ping.online.net, out of curiosity, and looks like it's only 30 Mbit/s both ways.

 

Thanks for the help Man Happy

New Member
Posts: 8
Registered: ‎10-30-2015
Kudos: 4

Re: UDP packet loss with EdgeRouter Lite

[ Edited ]

Hi, I noticed this as well on an EdgeRouterPOE with 1.8.5 firmware. It doesn't seem like packet loss per say though. Rather, it's out of order packets, which can be seen easily if you run iperf3 yourself. Client:

$ iperf3 -c somwhere.org -u -b 10M
Connecting to host somewhere.org, port 5201
[  4] local 192.168.82.38 port 36021 connected to 52.30.30.30 port 5201
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec  1.09 MBytes  9.11 Mbits/sec  139  
[  4]   1.00-2.00   sec  1.20 MBytes  10.0 Mbits/sec  153  
[  4]   2.00-3.00   sec  1.19 MBytes  9.96 Mbits/sec  152  
[  4]   3.00-4.00   sec  1.20 MBytes  10.0 Mbits/sec  153  
[  4]   4.00-5.00   sec  1.19 MBytes  9.96 Mbits/sec  152  
[  4]   5.00-6.00   sec  1.20 MBytes  10.0 Mbits/sec  153  
[  4]   6.00-7.00   sec  1.19 MBytes  9.96 Mbits/sec  152  
[  4]   7.00-8.00   sec  1.20 MBytes  10.0 Mbits/sec  153  
[  4]   8.00-9.00   sec  1.20 MBytes  10.0 Mbits/sec  153  
[  4]   9.00-10.00  sec  1.19 MBytes  9.96 Mbits/sec  152  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.00  sec  11.8 MBytes  9.91 Mbits/sec  5.216 ms  389/1511 (26%)  
[  4] Sent 1511 datagrams

iperf Done.

Server:

iperf3: OUT OF ORDER - incoming packet = 1494 and received packet = 1495 AND SP = 6
iperf3: OUT OF ORDER - incoming packet = 1500 and received packet = 1502 AND SP = 6
iperf3: OUT OF ORDER - incoming packet = 1501 and received packet = 1502 AND SP = 6
iperf3: OUT OF ORDER - incoming packet = 1503 and received packet = 1504 AND SP = 6
iperf3: OUT OF ORDER - incoming packet = 1505 and received packet = 1506 AND SP = 6
  snip...
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  6]   0.00-10.05  sec  0.00 Bytes  0.00 bits/sec  5.216 ms  389/1511 (26%)  
[SUM]  0.0-10.1 sec  389 datagrams received out-of-order

For TCP this shows up as retries with TCP_DUP_ACK in wireshark on the client. I'd love to get to the bottom of this, since it manifests itself as packet loss in many things that expect sequence numbers to not jump around. Anything above 1Mbps seems to have a problem.

Reply