Emerging Member
Posts: 69
Registered: ‎03-13-2016
Kudos: 14
Solutions: 2

USG IPsec VPN speed

The USG IPsec VPN seems to max out around 12-15Mbps. Is this expected to improve with further firmware/controller upgrades or this is the hardware limit?

 

my other (older) VPN devices manage faster speeds:

Synology 215j - 30Mbps

1513+ - 60Mbps

RT1900ac - 50Mbps.

Member
Posts: 188
Registered: ‎03-07-2016
Kudos: 54
Solutions: 10

Re: USG IPsec VPN speed

[ Edited ]

I don't think you're seeing a limitation of the USG's VPN - I am able to pass more than 50mbps between locations on an IPSec tunnel with 100mbit connections at each location. This was a crude test done during normal business hours so it was fighting with other traffic as well.

 

edited because I am seeing this issue as well between several sites with both USG's and USG-PRO's.

Emerging Member
Posts: 69
Registered: ‎03-13-2016
Kudos: 14
Solutions: 2

Re: USG IPsec VPN speed

What areas should I focus on troubleshooting? I doubt it's the wan connection. Both sites are running 1gbps symmetric and I can do SSH for transfers between them at speeds much higher than what I'm seeing
New Member
Posts: 35
Registered: ‎08-13-2016
Kudos: 3
Solutions: 1

Re: USG IPsec VPN speed

I just switched to a USG Pro and am getting 40 up / 20 down so hardware seems to be a factor or at least the encryption needs some optimization.

Ubiquiti Employee
Posts: 1,482
Registered: ‎02-28-2017
Kudos: 481
Solutions: 148

Re: USG IPsec VPN speed

SSH to the USG and:

 

show ubnt offload 
^ to make sure the IPsec offload module is loaded

I can get around 50-60 Mb/s IPsec settings with IKEv2 - aes 128 - sha 1 - dh 2 - pfs (group1) on a USG3p
and around 110-120 Mb/s with the USG Pro 4.

Are your USG's behind NAT? If not, some ISPs / modems handle ESP oddly.

Also, are you testing with iperf3?

Brandon Jaffe | UniFi Routing & Switching | Austin, TX
Member
Posts: 245
Registered: ‎08-29-2016
Kudos: 146
Solutions: 2

Re: USG IPsec VPN speed

@UBNT-jaffe
I'm not getting any more than 10-15 Mbit/s either on my USG3p, using the exact same settings you are using.

Things I checked:
- offload is enabled. Manually disabling the offload and retesting showed the same results.
- This is using 4.3.41 and UniFi 5.6.4 vs a pfSense SG-4860
- The USG is behind a 200/12Mbit/s cable line and the pfSense is on a symmetric gigabit connection inside a Tier-1 datacenter
- iPerf between these without the VPN maxes out the cable line.
Ubiquiti Employee
Posts: 1,482
Registered: ‎02-28-2017
Kudos: 481
Solutions: 148

Re: USG IPsec VPN speed

Run a simultaneous packet capture on the USG and pfSense filtering for ESP when you run iperf.
You can do so on the USG by SSHing in and executing:
sudo tcpdump -npi <wan_interface> esp

You can do the same on pfSense by going to Diagnostics > packet capture > select the WAN interface, filter for ESP and put the USG WAN address in for the host, and press start capture.

Start those captures at the same time, run iperf, and then stop them and see if the packet amounts match up. I would recommend accessing pfSense and the USG "outside" of the vpn, as your SSH or HTTPS traffic flowing through the VPN will count as ESP.
Brandon Jaffe | UniFi Routing & Switching | Austin, TX
Member
Posts: 245
Registered: ‎08-29-2016
Kudos: 146
Solutions: 2

Re: USG IPsec VPN speed

Will do. Probably going to take a while to set up.
Emerging Member
Posts: 69
Registered: ‎03-13-2016
Kudos: 14
Solutions: 2

Re: USG IPsec VPN speed

[ Edited ]

i ran "show ubnt offload" and saw that the offload module was loaded.

i'm testing the speed by:

1. running a speedtest (provided by the ISP) on a PC on LAN1- it will run at full speed (>900mbps)

2. connecting by VPN to LAN1 from a PC on LAN2 and running speedtest - it will run at 12-15mbps

 

i'm using USGs for both LAN1 and LAN2 and they are both running 1gbps WAN with public IPs so I don't think they are behind a NAT.

 

Highlighted
Ubiquiti Employee
Posts: 1,482
Registered: ‎02-28-2017
Kudos: 481
Solutions: 148

Re: USG IPsec VPN speed

Ah I see, so you're using a remote access VPN - I assume L2TP over IPsec? What controller version / USG version are you on? Is the client you're testing speedtests on a wired connection?

I have about 10 WANs setup in my test bench, with 2 real public WANs. I tested from my iphone behind LAN1 (USG1) to an L2TP server directly on USG2 (the path is iphone > AP > switch > usg1 > modem > usg2) and I got exactly 31 Mb/s download / 11 Mb/s upload.
Brandon Jaffe | UniFi Routing & Switching | Austin, TX
Emerging Member
Posts: 69
Registered: ‎03-13-2016
Kudos: 14
Solutions: 2

Re: USG IPsec VPN speed

[ Edited ]

yes, remote access VPN. L2TP over IPSec.

i'm on 4.3.41, 3.7.57 and 5.5.11

client being tested is on a wired connection.

 

the path is:

 

client -> US-8 -> US-8-150 -> US-8 -> USG -> modem -> USG2

with this setup, it's 17 Mbps down / 12 Mbps up

 

when I am able I will try:

 

client -> US-8 -> USG -> modem -> USG2

 

 

Established Member
Posts: 1,338
Registered: ‎05-25-2016
Kudos: 243
Solutions: 11

Re: USG IPsec VPN speed

@UBNT-jaffe
Seeing the same thing here.. remote vpn access to the usg using l2tp.. 10-12 mbit but usg is sitting on a500/500 connection
Ubiquiti Employee
Posts: 1,482
Registered: ‎02-28-2017
Kudos: 481
Solutions: 148

Re: USG IPsec VPN speed

@epigram Were you able to test again? I would try taking the modem out of the equation if you can.


@skandshus What path are you clients taking when accessing the Remote Access VPN? Are you testing internally or actually testing from the outside? 

Brandon Jaffe | UniFi Routing & Switching | Austin, TX
Member
Posts: 245
Registered: ‎08-29-2016
Kudos: 146
Solutions: 2

Re: USG IPsec VPN speed

[ Edited ]

@UBNT-jaffe
I have set up the test as you requested and have captured the ESP traffic at both ends:
pfSense: 19968 Packets
USG: 10793 packets captured
19968 packets received by filter
9175 packets dropped by kernel

I still only get around 10-12Mbit/s in the direction that can carry 200 Mbit/s. Any ideas?

 

Update: Using UDP I can get the throughput up to ~40Mbit/s but anything more than 25 Mbit/s and it's getting very very lossy... and some TCP love would be nice too.

Ubiquiti Employee
Posts: 1,482
Registered: ‎02-28-2017
Kudos: 481
Solutions: 148

Re: USG IPsec VPN speed

[ Edited ]

@olbjan Half of your ESP packets are getting dropped by the USG it looks like. What's your WAN_IN ruleset there?
show firewall name WAN_IN

Brandon Jaffe | UniFi Routing & Switching | Austin, TX
Member
Posts: 245
Registered: ‎08-29-2016
Kudos: 146
Solutions: 2

Re: USG IPsec VPN speed

IPv4 Firewall "WAN_IN":

 Active on (eth0,IN) (eth2,IN)

rule  action   proto     packets  bytes
----  ------   -----     -------  -----
3001  accept   all       433750   72281012
  condition - state RELATED,ESTABLISHED                                      

3002  drop     all       0        0
  condition - state INVALID                                                  

3003  accept   tcp_udp   220      11096
  condition - daddr x.x.x tcp dpt:3389 LOG enabled                

3004  accept   tcp       690      35791
  condition - daddr x.x.x tcp dpt:8443 LOG enabled              

3005  accept   tcp_udp   26       1288
  condition - daddr x.x.x tcp dpt:http-alt LOG enabled          

3006  accept   tcp       0        0
  condition - daddr x.x.x tcp dpt:8443 LOG enabled              

3007  accept   tcp       0        0
  condition - daddr x.x.x tcp dpt:8843                          

3008  accept   tcp       0        0
  condition - daddr x.x.x tcp dpt:8880                          

10000 drop     all       0        0

@UBNT-jaffeThis is my ruleset

Established Member
Posts: 1,338
Registered: ‎05-25-2016
Kudos: 243
Solutions: 11

Re: USG IPsec VPN speed

@UBNT-jaffe
I'm testing from the outside.

I'm actually testing between my own sites..
from my personal home network and to a customer usg..

All unified setup.

Both sitting on a 500/500.. are w SUPPOSED to be able to get higher than 10-12 mbit?

Ubiquiti Employee
Posts: 1,482
Registered: ‎02-28-2017
Kudos: 481
Solutions: 148

Re: USG IPsec VPN speed

[ Edited ]

@olbjan Looks like I had the wrong interpretation of "packets dropped by kernel" Apparently that's tcpdump dropping the packets due to lack of buffer space. They aren't being filtered by the firewall.  You can check top -SH while running iperf as well to see if you're hitting some sort of hardware limitation with memory or if something else is eating up the CPU.

@skandshus Yeah I would say you should be getting more than 10-12 Mb/s since I'm able to get >30Mb/s even on wifi, although I believe I tested on a USG pro for that. However, there's a lot of factors when testing from the outside (traversing home network > ISP1 > ISP2 > remote network > ISP2 > speedtest server, then the same path on the reply). Is there any limitations you're experiencing with 10-12 Mb/s being on a remote access VPN? You can always try iperf directly from your home network straight to the remote network for a more accurate result.

Brandon Jaffe | UniFi Routing & Switching | Austin, TX
Member
Posts: 245
Registered: ‎08-29-2016
Kudos: 146
Solutions: 2

Re: USG IPsec VPN speed

@UBNT-jaffe:

I tried again while running "top -SH":
There are no processes that immediately jump out to me, nothing really goes above 3-8% CPU. The top process is ksoftirqd during the test.
For fun I also tried while disabling offload and I get pretty much the exact same result, with ksoftirqd topping the list around 8%. I also get the same throughput with and without the offload.


Is it possible that the ipsec offload engine isn't working at all in the current build? What should I be looking for?
Established Member
Posts: 1,014
Registered: ‎08-22-2016
Kudos: 459
Solutions: 1

Re: USG IPsec VPN speed

[ Edited ]

In comparison, my tiny fanless pfSense box (fitlet-XA10-LAN) with 4 Gigabit ports, running on AMD A10 Micro-6700T CPU with 4 GB of RAM can do 160 Mbps via IPSec VPN up/down. This box does about 700 Mbps up/down LAN-to-WAN and WAN-to-LAN in clear text with NAT enabled. I thought this was a relatively poor VPN throughput provided that Gigabit speeds are becoming increasingly pervasive in the US as home Internet access speeds. 

 

I purchased this box for $315 barebones in July 2016, when my Cisco ASA5505 was hit by a lightning. I paid about $375 altogether (including SSD and RAM). USG Pro is $264 on B&H. Let me tell you that one cannot buy an enterprise-level router/firewall for $264. Period. Nothing to add here. 

 

If you want to get an enterprise-level router/firewall, be prepared to pay north of $500 for sure, and I use the word "enterprise" loosely here. The real enterprise-level routers cost thousands for dollars. 

 

Ubiquiti needs to stop calling this an enterprise-level router. There's little reason to release a USG enterprise-level router right now becasue the feature set exposed in UniFi controller is nowhere close to the enterprise feature set. 

 

Ubiquiti folks: Please make USG and USG Pro into the DPI probe and then work on expanding the feature set. Maybe in a couple years a real enterprise-level USG router/firewall can be released. In the mean time, I would advise folks to use pfSense unless you are prepared to go with Cisco or Juniper gear, which is very expensive.

 

If you want to get the cheapest Cisco router that can push 1 Gbps up/down across an IPSec VPN tunnel, get Cisco ISR 4431 with the high performarnce license FL-44-PERF-K9 ($759), which increases the clear text throughput from 500 Mbps to 1 Gbps. Then, you would have to purchase FL-44-HSEC-K9(=) license ($480) for 4431, which allows the encrypted througput to increase from 80 Mbps to the line speed (up to 1 Gbps). Of course before you purchase these two licenses, you will actually have to purchase a Cisco 4431 router with SEC license ($2,374). So, you will spend $3600 to be able to push 1 Gbps via IPSec with Cisco, and that's the cheapest Cisco router that can push up to 1 Gbps in encrypted throughput.

 

Let's not kid ourselves. UniFi is not enterprise-level gear. This is SMB-level, which is fine with me. The problem is that even for SMB, encrypted througput of 30 Mbps is pretty bad.