05-03-2017 09:24 PM
The USG IPsec VPN seems to max out around 12-15Mbps. Is this expected to improve with further firmware/controller upgrades or this is the hardware limit?
my other (older) VPN devices manage faster speeds:
Synology 215j - 30Mbps
1513+ - 60Mbps
RT1900ac - 50Mbps.
05-04-2017 07:16 AM
05-04-2017 06:27 PM
05-07-2017 04:23 PM
SSH to the USG and:
show ubnt offload
^ to make sure the IPsec offload module is loaded
I can get around 50-60 Mb/s IPsec settings with IKEv2 - aes 128 - sha 1 - dh 2 - pfs (group1) on a USG3p
and around 110-120 Mb/s with the USG Pro 4.
Are your USG's behind NAT? If not, some ISPs / modems handle ESP oddly.
Also, are you testing with iperf3?
05-08-2017 01:41 AM
I'm not getting any more than 10-15 Mbit/s either on my USG3p, using the exact same settings you are using.
Things I checked:
- offload is enabled. Manually disabling the offload and retesting showed the same results.
- This is using 4.3.41 and UniFi 5.6.4 vs a pfSense SG-4860
- The USG is behind a 200/12Mbit/s cable line and the pfSense is on a symmetric gigabit connection inside a Tier-1 datacenter
- iPerf between these without the VPN maxes out the cable line.
05-08-2017 12:41 PM
You can do so on the USG by SSHing in and executing:
sudo tcpdump -npi <wan_interface> esp
You can do the same on pfSense by going to Diagnostics > packet capture > select the WAN interface, filter for ESP and put the USG WAN address in for the host, and press start capture.
Start those captures at the same time, run iperf, and then stop them and see if the packet amounts match up. I would recommend accessing pfSense and the USG "outside" of the vpn, as your SSH or HTTPS traffic flowing through the VPN will count as ESP.
05-09-2017 06:35 AM - edited 05-09-2017 06:35 AM
i ran "show ubnt offload" and saw that the offload module was loaded.
i'm testing the speed by:
1. running a speedtest (provided by the ISP) on a PC on LAN1- it will run at full speed (>900mbps)
2. connecting by VPN to LAN1 from a PC on LAN2 and running speedtest - it will run at 12-15mbps
i'm using USGs for both LAN1 and LAN2 and they are both running 1gbps WAN with public IPs so I don't think they are behind a NAT.
05-09-2017 06:49 AM
I have about 10 WANs setup in my test bench, with 2 real public WANs. I tested from my iphone behind LAN1 (USG1) to an L2TP server directly on USG2 (the path is iphone > AP > switch > usg1 > modem > usg2) and I got exactly 31 Mb/s download / 11 Mb/s upload.
05-09-2017 10:40 AM - edited 05-09-2017 10:43 AM
yes, remote access VPN. L2TP over IPSec.
i'm on 4.3.41, 3.7.57 and 5.5.11
client being tested is on a wired connection.
the path is:
client -> US-8 -> US-8-150 -> US-8 -> USG -> modem -> USG2
with this setup, it's 17 Mbps down / 12 Mbps up
when I am able I will try:
client -> US-8 -> USG -> modem -> USG2
05-12-2017 11:34 AM
05-15-2017 01:29 PM - edited 05-15-2017 01:43 PM
I have set up the test as you requested and have captured the ESP traffic at both ends:
pfSense: 19968 Packets
USG: 10793 packets captured
19968 packets received by filter
9175 packets dropped by kernel
I still only get around 10-12Mbit/s in the direction that can carry 200 Mbit/s. Any ideas?
Update: Using UDP I can get the throughput up to ~40Mbit/s but anything more than 25 Mbit/s and it's getting very very lossy... and some TCP love would be nice too.
05-15-2017 01:56 PM - edited 05-15-2017 01:57 PM
@olbjan Half of your ESP packets are getting dropped by the USG it looks like. What's your WAN_IN ruleset there?
show firewall name WAN_IN
05-15-2017 02:41 PM
IPv4 Firewall "WAN_IN": Active on (eth0,IN) (eth2,IN) rule action proto packets bytes ---- ------ ----- ------- ----- 3001 accept all 433750 72281012 condition - state RELATED,ESTABLISHED 3002 drop all 0 0 condition - state INVALID 3003 accept tcp_udp 220 11096 condition - daddr x.x.x tcp dpt:3389 LOG enabled 3004 accept tcp 690 35791 condition - daddr x.x.x tcp dpt:8443 LOG enabled 3005 accept tcp_udp 26 1288 condition - daddr x.x.x tcp dpt:http-alt LOG enabled 3006 accept tcp 0 0 condition - daddr x.x.x tcp dpt:8443 LOG enabled 3007 accept tcp 0 0 condition - daddr x.x.x tcp dpt:8843 3008 accept tcp 0 0 condition - daddr x.x.x tcp dpt:8880 10000 drop all 0 0
@UBNT-jaffeThis is my ruleset
05-15-2017 02:53 PM
I'm testing from the outside.
I'm actually testing between my own sites..
from my personal home network and to a customer usg..
All unified setup.
Both sitting on a 500/500.. are w SUPPOSED to be able to get higher than 10-12 mbit?
05-15-2017 03:52 PM - edited 05-15-2017 03:53 PM
@olbjan Looks like I had the wrong interpretation of "packets dropped by kernel" Apparently that's tcpdump dropping the packets due to lack of buffer space. They aren't being filtered by the firewall. You can check top -SH while running iperf as well to see if you're hitting some sort of hardware limitation with memory or if something else is eating up the CPU.
@skandshus Yeah I would say you should be getting more than 10-12 Mb/s since I'm able to get >30Mb/s even on wifi, although I believe I tested on a USG pro for that. However, there's a lot of factors when testing from the outside (traversing home network > ISP1 > ISP2 > remote network > ISP2 > speedtest server, then the same path on the reply). Is there any limitations you're experiencing with 10-12 Mb/s being on a remote access VPN? You can always try iperf directly from your home network straight to the remote network for a more accurate result.
05-16-2017 03:30 AM
I tried again while running "top -SH":
There are no processes that immediately jump out to me, nothing really goes above 3-8% CPU. The top process is ksoftirqd during the test.
For fun I also tried while disabling offload and I get pretty much the exact same result, with ksoftirqd topping the list around 8%. I also get the same throughput with and without the offload.
Is it possible that the ipsec offload engine isn't working at all in the current build? What should I be looking for?
05-16-2017 07:20 AM - edited 05-16-2017 07:26 AM
In comparison, my tiny fanless pfSense box (fitlet-XA10-LAN) with 4 Gigabit ports, running on AMD A10 Micro-6700T CPU with 4 GB of RAM can do 160 Mbps via IPSec VPN up/down. This box does about 700 Mbps up/down LAN-to-WAN and WAN-to-LAN in clear text with NAT enabled. I thought this was a relatively poor VPN throughput provided that Gigabit speeds are becoming increasingly pervasive in the US as home Internet access speeds.
I purchased this box for $315 barebones in July 2016, when my Cisco ASA5505 was hit by a lightning. I paid about $375 altogether (including SSD and RAM). USG Pro is $264 on B&H. Let me tell you that one cannot buy an enterprise-level router/firewall for $264. Period. Nothing to add here.
If you want to get an enterprise-level router/firewall, be prepared to pay north of $500 for sure, and I use the word "enterprise" loosely here. The real enterprise-level routers cost thousands for dollars.
Ubiquiti needs to stop calling this an enterprise-level router. There's little reason to release a USG enterprise-level router right now becasue the feature set exposed in UniFi controller is nowhere close to the enterprise feature set.
Ubiquiti folks: Please make USG and USG Pro into the DPI probe and then work on expanding the feature set. Maybe in a couple years a real enterprise-level USG router/firewall can be released. In the mean time, I would advise folks to use pfSense unless you are prepared to go with Cisco or Juniper gear, which is very expensive.
If you want to get the cheapest Cisco router that can push 1 Gbps up/down across an IPSec VPN tunnel, get Cisco ISR 4431 with the high performarnce license FL-44-PERF-K9 ($759), which increases the clear text throughput from 500 Mbps to 1 Gbps. Then, you would have to purchase FL-44-HSEC-K9(=) license ($480) for 4431, which allows the encrypted througput to increase from 80 Mbps to the line speed (up to 1 Gbps). Of course before you purchase these two licenses, you will actually have to purchase a Cisco 4431 router with SEC license ($2,374). So, you will spend $3600 to be able to push 1 Gbps via IPSec with Cisco, and that's the cheapest Cisco router that can push up to 1 Gbps in encrypted throughput.
Let's not kid ourselves. UniFi is not enterprise-level gear. This is SMB-level, which is fine with me. The problem is that even for SMB, encrypted througput of 30 Mbps is pretty bad.