Reply
New Member
Posts: 36
Registered: ‎11-12-2015
Kudos: 13
Solutions: 2

Re: USG-3P Smart Queue testing results

@Troon as I understand, the 4P uses a faster Caivum dual core processor (running at 1Ghz instead of 500Mhz on the 3P) what are your thoughts on applying your settings below on a 4P? In theory it should produce a similar increase. 

 

echo 2 > /proc/irq/24/smp_affinity
echo 2 > /sys/module/octeon_ethernet/parameters/rx_cpu_factor
echo 3 > /sys/class/net/eth0/queues/rx-0/rps_cpus
echo 2048 > /proc/sys/net/core/rps_sock_flow_entries
echo 2048 > /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
Member
Posts: 189
Registered: ‎09-01-2017
Kudos: 50
Solutions: 9

Re: USG-3P Smart Queue testing results


@mojo_333 wrote:

@Troon as I understand, the 4P uses a faster Caivum dual core processor (running at 1Ghz instead of 500Mhz on the 3P) what are your thoughts on applying your settings below on a 4P? In theory it should produce a similar increase.  


 

I do have a USG-Pro waiting to be connected in my ongoing church installation. However, as their connection is currently only 40/10, I'm not expecting to be able to test any theoretical improvements.

 

All I can suggest is test it, if you're lucky enough to have a Pro and a connection that is fast enough to cause Smart Queue restrictions.

Home: USG-3P | UAP-AC-Lite
Church: USG-Pro | US-24-250W | UAP-AC-Lite x 6 | UAP-AC-IW x 2
Controller: Linode 4GB VPN on Ubuntu 18.04
New Member
Posts: 28
Registered: ‎12-29-2017
Kudos: 1
Solutions: 2

Re: USG-3P Smart Queue testing results

I've had zero luck getting smart queues to do anything useful.  I have a USG 3P, 24 port POE-250W switch and 3 AP-HDs, all unifi stuff.  I'm on ATT DSL (75down, 7up).   If I turn on smart queues, network down hits ~2M, and things like youtube (with 1080p video) become unusuable.   

 

I've got DPI turned off, no GEOIP, no IPS, etc.  I've tried to get rid of any features that are suggested here.  Nothing seems to matter.

 

I'm measuring with speedtest.net. 

 

Sometimes, weirdly, on some runs, the download jumps up to 20m for part of the time, but mostly stays around 2.    I'm at a loss, and other suggestions would be appreciated.  I've never seen anything in excess of 3-4mbps down any time I've tried it, except for the rare bursts during a speedtest.  The family confirms our network performance goes to hell if I turn it on. 

 

I work from home quite a bit, as does my wife, and sometimes I do things that saturate the upload side (e.g., uploading a huge file), which completely kills the ping time;  so, in this case, my son wanders up and complains that ping is > 400, preventing him from playing games.  Not to mention the impact on the rest of the family.

 

I like the idea of the USG, but if I can't use any of the features, AND it is not giving me anything to deal with network flow, I am unsure if there is any reason to keep using it.  (I also wanted to use the 2nd WAN port as failover, but that doesn't work either -- see "many other posts on this forum" on that topic.)

New Member
Posts: 4
Registered: ‎12-03-2017
Kudos: 1

Re: USG-3P Smart Queue testing results

This probably doesn’t help much, but I have the exact same hardware as you (USG 3G, 24 port switch, 3 AP Pros) and no problems with speed with smart queues, DPI, IPS all turned on. I have vdsl2 so max 60mbs and I get that speed.

 

if I can think of anything to try I’ll post again, if if you want me to try something on my setup or my exact options let me know. 

New Member
Posts: 28
Registered: ‎12-29-2017
Kudos: 1
Solutions: 2

Re: USG-3P Smart Queue testing results

That’s really good to know.  do you have IPv6 turned on?  I’ve seen people mention it as a problem in some contexts?  And firewall rules?

New Member
Posts: 4
Registered: ‎12-03-2017
Kudos: 1

Re: USG-3P Smart Queue testing results

A couple of port forwarding rules turned on. IPv6 turned off.  

Regular Member
Posts: 489
Registered: ‎07-20-2013
Kudos: 241
Solutions: 22

Re: USG-3P Smart Queue testing results

@blairmacintyre  If you have AT&T, I assume you’ve read this thread?

New Member
Posts: 28
Registered: ‎12-29-2017
Kudos: 1
Solutions: 2

Re: USG-3P Smart Queue testing results

I hadn’t seen that thread, but doesn’t seem to match me;  I’m on DSL (75 up, 7 down) and have an oldish gateway.

 

i think I’ll try turning off ipv6

New Member
Posts: 28
Registered: ‎12-29-2017
Kudos: 1
Solutions: 2

Re: USG-3P Smart Queue testing results

[ Edited ]

FYI: I turned off ipv6, had no effect. If I turn on smart queues, I get between 3 and 12 mbps down and 1-2 up. If I turn off smart queues, I get almost 80 down and 7 or more up.

 

This is with no DPI, no IPS.

New Member
Posts: 4
Registered: ‎12-03-2017
Kudos: 1

Re: USG-3P Smart Queue testing results

Hi, sorry I may have accidently misled you with my previous answer.

 

Yes, if you enable Smart Queues your speed will seem to drop from 60mbs to about 20mbs - this is how Smart Queues works. It will fairly allocate bandwidth between all the users on your network, and to do this it will limit your speed so that other users aren't affected. Your overall speed (added up between ALL the users on your network) will be the same. This is unaffected by turning DPI/IPS etc on or off because it's working as intended.

 

You can verify for yourself that your USG3 isn't being overloaded by sshing into it and running 'top' to see the cpu usage. Smart Queues doesn't use very much CPU.

 

Some more info here: https://community.ubnt.com/t5/EdgeRouter/What-is-Smart-Queue-and-when-would-I-use-it/m-p/2547338/hig...

New Member
Posts: 28
Registered: ‎12-29-2017
Kudos: 1
Solutions: 2

Re: USG-3P Smart Queue testing results

Oh, no, not misleading.  I didn't expect full speed with smart queues, I get how it works.  But, having it sometimes cap at a really small amount (3mbps up, 1mbps down) is problematic, esp when the network is otherwise unloaded.  So, what do you get per client when you go to somewhere like http://speedtest.net/ ?

 

I was also pleased to see you say you had it working at reasonable speeds with DPI and IPS.

 

If I can get 10+ mbps down and 2 or more up per client, that's probably enough.  We still want to be able to watch moves and stream video sometimes (e.g., to twitch or youtube, my son is playing with doing some game streaming, to go along with the game videos he already puts on youtube).  Heck, I'm happy if it keeps the stream well below half the max of 7mbps so that the game ping times don't spike, which is (of course) the who reason I started fiddling with this.

 

I'll keep playing.

Member
Posts: 189
Registered: ‎09-01-2017
Kudos: 50
Solutions: 9

Re: USG-3P Smart Queue testing results

You should set your Smart Queues limits (Settings / Networks / WAN / Edit) to about 90-95% of the "raw" values, and test using DSLReports speed test at http://www.dslreports.com/speedtest to ensure you get A or A+ for bufferbloat.

 

On the latest 4.4.36 beta firmware, I get 53.5Mbps down with A+ rating.

Home: USG-3P | UAP-AC-Lite
Church: USG-Pro | US-24-250W | UAP-AC-Lite x 6 | UAP-AC-IW x 2
Controller: Linode 4GB VPN on Ubuntu 18.04
New Member
Posts: 28
Registered: ‎12-29-2017
Kudos: 1
Solutions: 2

Re: USG-3P Smart Queue testing results

Thanks for the pointer! What are your raw values, and what did you set to, out of curiosity? I was setting a bit over (ie, I get 78-80 mbps so I set to 80) and I get terrible numbers.

I’m still running 4.4.29.5124210 on my USG, btw; should the slightly newer one really have an impact?
Emerging Member
Posts: 53
Registered: ‎04-11-2017
Kudos: 46
Solutions: 1

Re: USG-3P Smart Queue testing results

[ Edited ]

@Troon @blairmacintyre A handy note, the Unifi USG already tweaks this slightly. As per https://dl.ubnt.com/guides/UniFi/UniFi_Controller_V5_UG.pdf page 119: "The actual rate limits will be set to 95% of the specified value, so you can experiment with different values if necessary". This is also noted for the update to the EdgeRouter line where the feature was first introduced. Hope this is helpful.

 

Edit: I should also note that personally I "cheat" on the USG3P by only using smart queue in the upload direction. This allows me to get far closer to my maximum download speed (150) while still having the all too important latency control on my limited upload. The traffic is still traversing the CPU in both directions but CPU without queue is far faster at only passing traffic along.

44330904

(A quick test with that current config)

Of course only total solution is a 4Pro or the much pined for ER-4 based USG.

Member
Posts: 189
Registered: ‎09-01-2017
Kudos: 50
Solutions: 9

Re: USG-3P Smart Queue testing results


@blairmacintyre wrote:
Thanks for the pointer! What are your raw values, and what did you set to, out of curiosity? I was setting a bit over (ie, I get 78-80 mbps so I set to 80) and I get terrible numbers.

I’m still running 4.4.29.5124210 on my USG, btw; should the slightly newer one really have an impact?

See the first post in this topic, the first table, and the differences between data rows 2 (with limit about line rate) and 3 (below).

 

I'm on an 80/20 VDSL package that runs to 72/16 observed maximum; I have SQ set to 64/15 and see 53/14 with A+ on everything. Any higher on the limits gives worse ping with no speed benefit.

 

Obviously, these are tuned for my setup and shouldn't be blindly copied.

Home: USG-3P | UAP-AC-Lite
Church: USG-Pro | US-24-250W | UAP-AC-Lite x 6 | UAP-AC-IW x 2
Controller: Linode 4GB VPN on Ubuntu 18.04
New Member
Posts: 28
Registered: ‎12-29-2017
Kudos: 1
Solutions: 2

Re: USG-3P Smart Queue testing results

@And4713@Troon@sebachu

and others, thanks for your help.  I'm posting this "what I did to fix this" just so folks know, even though the cause of my problems is pretty severe "user error" on my part, and pretty mortifying to admit.  (As an aside, for those following along, you now will have another "dumb professor" story, since sadly, I'm a professor of computer science, pretty technically skilled in general, but do graphics/games/vr/etc, not networking).

 

The problem was simple, it turns out, but I'd stared at it for so long I didn't see it.  The hint is that I think of the bandwidths in terms of "down, up" (i.e., I have 76 down, 7 up).  So, here's what I put in the form:

Messages Image(1008047328).jpeg

 

After that was entered once, I fiddled with the values, but never really looked closely at the labels again. Swapping the values to their correct locations, and lowering them a bit to 7000 / 70000 did wonderful things.

 

Prior to smart queues, if I ran DSL reports, I had severe bufferbloat (> 400ms additional latency during the upload test).

dsl-reports-no-queues.png

 

After smart queues, with DPI still turned on, the induced latency during upload shows as +1ms (yippee!) and I get really good speeds.

dsl-reports-queues.png

 

This is with no turning (I haven't tried adjusting the values much) and without any of the small tweaks suggested in this thread.  I also haven't updated the firmware yet (one thing at a time!)

 

So, I posted here because I knew this should work and I must be doing something "dumb" ... just didn't realize how dumb.  Thank you all for being helpful!

New Member
Posts: 28
Registered: ‎12-29-2017
Kudos: 1
Solutions: 2

Re: USG-3P Smart Queue testing results

One day later, things are working great. 

 

Having an evening without my son's fortnite gaming experience sudden 400ms ping because someone else is saturating the network is like we have a whole new network.

 

Will be updating the USG to the latest firmware this weekend, hopefully nothing will break.  Perhaps I'll even try to get failover to my small 3G box working again.

Regular Member
Posts: 537
Registered: ‎12-15-2016
Kudos: 55
Solutions: 13

Re: USG-3P Smart Queue testing results

[ Edited ]

@blairmacintyre

 

If its working well for you now.....just leave things as they are for a bit longer, as in several weeks or months even......unless there is something on the newer firmware updates that you really need?

 

 

Highlighted
New Member
Posts: 28
Registered: ‎12-29-2017
Kudos: 1
Solutions: 2

Re: USG-3P Smart Queue testing results

I’m on 4.4.29.  I’ve been wanting to update to 4.4.34 at least, to see if the load balancing / failover fixes actually let me start using failover. It was one of the features I was initially excited about with the USG, and it’s never worked right.

Emerging Member
Posts: 87
Registered: ‎07-16-2016
Kudos: 6
Solutions: 2

Re: USG-3P Smart Queue testing results

@blairmacintyre& @Troon,

 

Here are some numbers from newer firmware for another data point.

 

150/5 Advertised speed from ISP

132/5 Measured speed from wired client from dslreports before changes (similar result from my phone on wifi using Wifiman speedtest or dslreports)

 

USG3P running 4.4.36

DPI enabled

IPS disabled

SQ enabled and set to 85% (131/4.4)

 

57/2 Wired client from dslreports

59/3 Wired client from speedof.me

57/3.75 Wifi client from dsl reports

73/3.5 Wifi client from Wifiman app (Ubiquiti)

72/4 Wifi client from Ookla Speedtest app

76/3 Wired Xbox (just because I was curious)

 

Since my observed rates before SQ were132/5 I adjust SQ to 85% of these values or 113/4.4. Results below.

 

74/2.75 Wired client from dslreports

87/3.5 Wired client from speedof.me

78/3.1 Wired client using ISP speedtest (which is ookla under the hood)

 

I didn't test over Wifi or from my Xbox. Man Happy

 

Those speeds are plenty fine for what we do, so I am going to leave it at 85% of my top observed speeds for now. I will note that 75 is only half of the 150 I pay for at this time. If I really need to use it, I can disable SQ temporarily. Man Happy

 

Regards!

 

Jason

Reply