New Member
Posts: 26
Registered: ‎09-26-2014
Kudos: 10
Solutions: 1

ERL Performance Testing (with IPSec VPN)

[ Edited ]

I haven't seen many IPSec tests around here. If you know of some please post them here. Here's the only one I've been able to find, but he's pushing the test through Mikrotik gear, which confuses me. 

http://blog.linitx.com/ubiquiti-edgerouter-performance-testing/

 

So I decided to do my own simple VPN performance tests. I put it on my blog, but I'll drop it here too. 

http://ch.illmachine.com/2014/10/edgerouter-lite-performance-tests/

 

Using version 1.5.0

 

Laptop A <-> ERL <-> ERL <-> Laptop B
192.168.20.100 <-> 192.168.20.1 192.168.1.2 <-> 192.168.1.1 192.168.10.1 <-> 192.168.10.100

20141010_22362.jpg

 

 

First a simple routing test to get a baseline without IPSec. I’m using OSPF to expose these two networks to each other for this test. See EdgeRouter Lite Simple OSPF Guide for more on what I did. 


iperf results from one laptop to the other:

 

$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.10.100 port 5001 connected with 192.168.20.100 port 42797
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 1.10 GBytes 940 Mbits/sec
[ 4] local 192.168.10.100 port 5001 connected with 192.168.20.100 port 42820
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-100.0 sec 11.0 GBytes 941 Mbits/sec

Near wire. Great!

 

Next we’ll remove the OSPF routing and setup an IPSec tunnel between the two an retest.

 

I used the GUI’s IPSec site-to-site feature to set this up.

 

And the results with IPSec hardware offload disabled.

 

'set system offload ipsec disable'
[ 4] local 192.168.10.100 port 5001 connected with 192.168.20.100 port 44600
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-100.1 sec 718 MBytes 60.2 Mbits/sec
[ 4] local 192.168.10.100 port 5001 connected with 192.168.20.100 port 44601
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 72.5 MBytes 60.5 Mbits/sec

 

And with IPSec hardware offload enabled.

 

'set system offload ipsec enable'
[ 4] local 192.168.10.100 port 5001 connected with 192.168.20.100 port 44586
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 180 MBytes 151 Mbits/sec
[ 4] local 192.168.10.100 port 5001 connected with 192.168.20.100 port 44590
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 179 MBytes 150 Mbits/sec
[ 4] local 192.168.10.100 port 5001 connected with 192.168.20.100 port 44596
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-100.0 sec 1.75 GBytes 151 Mbits/sec

 

Not bad!

 

However, even with IPSec offload enabled, load average increases a good bit when pushing max data through the IPSec VPN

jdrews@ERL10:~$ uptime
10:23:23 up 23 min, 1 user, load average: 0.58, 0.66, 0.47

 

Compared to without pushing data

jdrews@ERL10:~$ uptime
20:47:29 up 10:47, 1 user, load average: 0.00, 0.01, 0.05

While this didn’t seem to affect the usage of the router, it is a little worrying. I don’t have any firewalls or other services enabled on the router. What happens when we take this into the real world? It's worth noting that this is a dual core router, so really that's only 1/4 of total processing power. 

 

And before people start asking. I've attached the configurations of both routers for the IPSec test. As simple as it gets. 

New Member
Posts: 26
Registered: ‎09-26-2014
Kudos: 10
Solutions: 1

Re: ERL Performance Testing (with IPSec VPN)

[ Edited ]

Does anyone have any tests similar to this for the EdgeRouter Pro? My company is looking to use EdgeRouters to connect a Datacenter to a branch office via IPSec VPN. The branch office will be running a 100mbps fiber connection to the web. While these ERLs could probably do the trick, I'd rather have more headroom on the CPUs. 

New Member
Posts: 21
Registered: ‎09-22-2014
Kudos: 20

Re: ERL Performance Testing (with IPSec VPN)

I've got 2 Edgerouter Pro's sitting on my bench that I was just doing iperf testing with last week using 2 8-core servers as iperf endpoints.  Since jdrews was so nice as to provide his configs I'll give it a shot when I get a chance this week or next.

New Member
Posts: 21
Registered: ‎09-22-2014
Kudos: 20

Re: ERL Performance Testing (with IPSec VPN)

Ok, for some reason I'm getting slower results with IPSec Offload enabled.  I'm running v1.5.0, all links are Gigabit Full-Duplex.  I'm no iperf expert, so mabye I need different iperf options.

 

Configs are attached.

 

With ipsec offload disabled:

[root@Artemis asatnik]# /usr/local/bin/iperf3 -c 192.168.34.2
Connecting to host 192.168.34.2, port 5201
[  4] local 192.168.23.2 port 56676 connected to 192.168.34.2 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  10.8 MBytes  90.4 Mbits/sec   15   74.4 KBytes       
[  4]   1.00-2.00   sec  10.9 MBytes  91.1 Mbits/sec    1   88.0 KBytes       
[  4]   2.00-3.00   sec  10.8 MBytes  90.7 Mbits/sec    2   73.1 KBytes       
[  4]   3.00-4.00   sec  10.8 MBytes  90.4 Mbits/sec    1   83.9 KBytes       
[  4]   4.00-5.00   sec  10.7 MBytes  90.2 Mbits/sec    5   70.4 KBytes       
[  4]   5.00-6.00   sec  10.9 MBytes  91.1 Mbits/sec    6   86.6 KBytes       
[  4]   6.00-7.00   sec  10.9 MBytes  91.1 Mbits/sec    8   71.7 KBytes       
[  4]   7.00-8.00   sec  10.8 MBytes  90.8 Mbits/sec    1   86.6 KBytes       
[  4]   8.00-9.00   sec  10.8 MBytes  90.3 Mbits/sec    9   74.4 KBytes       
[  4]   9.00-10.00  sec  10.9 MBytes  91.5 Mbits/sec    1   88.0 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   108 MBytes  90.8 Mbits/sec   49             sender
[  4]   0.00-10.00  sec   108 MBytes  90.6 Mbits/sec                  receiver

iperf Done.

 With ipsec offload enabled:

[root@Artemis asatnik]# /usr/local/bin/iperf3 -c 192.168.34.2
Connecting to host 192.168.34.2, port 5201
[  4] local 192.168.23.2 port 56674 connected to 192.168.34.2 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  10.1 MBytes  85.0 Mbits/sec   15   73.1 KBytes       
[  4]   1.00-2.00   sec  10.0 MBytes  84.0 Mbits/sec    3   51.4 KBytes       
[  4]   2.00-3.00   sec  9.51 MBytes  79.8 Mbits/sec    2   59.6 KBytes       
[  4]   3.00-4.00   sec  10.0 MBytes  84.1 Mbits/sec    2   71.7 KBytes       
[  4]   4.00-5.00   sec  9.79 MBytes  82.1 Mbits/sec    3   54.1 KBytes       
[  4]   5.00-6.00   sec  10.5 MBytes  88.0 Mbits/sec    2   65.0 KBytes       
[  4]   6.00-7.00   sec  10.7 MBytes  89.9 Mbits/sec    3   52.8 KBytes       
[  4]   7.00-8.00   sec  9.94 MBytes  83.4 Mbits/sec    2   59.6 KBytes       
[  4]   8.00-9.00   sec  9.92 MBytes  83.2 Mbits/sec    2   69.0 KBytes       
[  4]   9.00-10.00  sec  10.1 MBytes  84.8 Mbits/sec    3   55.5 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   101 MBytes  84.4 Mbits/sec   37             sender
[  4]   0.00-10.00  sec   101 MBytes  84.3 Mbits/sec                  receiver

iperf Done.

 

 

 

Attachment
Attachment
New Member
Posts: 26
Registered: ‎09-26-2014
Kudos: 10
Solutions: 1

Re: ERL Performance Testing (with IPSec VPN)

Thanks asat for testing! 

 

That is a very curious result. How were they wired up? Like my picture? Was there anything in between the ERPros?

 

It's also suspicious that you're near wire speed of Fast Ethernet (100BASE-TX). Maybe one of the cables isn't gigabit rated? 

 

The iperf command you ran is exactly the same as mine (meaning defaults). 

 

I did restart each ERL after changing the IPsec offload. Maybe try that? 

New Member
Posts: 21
Registered: ‎09-22-2014
Kudos: 20

Re: ERL Performance Testing (with IPSec VPN)

[ Edited ]

Both routers are directly connected on eth0 using a cat5e crossover cable.  I was able to get 950Mbps routed without issue using the same iperf command and offload enabled, so I know it's not the cable.  I also restarted after toggling offload - I was forced to since I couldn't pass traffic over the tunnel if I didn't.  I'll troubleshoot more maybe tonight or tomorrow - I ran out of time earlier.  I might try 1.6.0rc1 and see if that makes a difference.

New Member
Posts: 26
Registered: ‎09-26-2014
Kudos: 10
Solutions: 1

Re: ERL Performance Testing (with IPSec VPN)

Awesome. Very eager to hear how it goes. Let us know! 

Highlighted
New Member
Posts: 21
Registered: ‎09-22-2014
Kudos: 20

Re: ERL Performance Testing (with IPSec VPN)

Derp.  I had fq_codel applied at 100Mbit on both routers.  I didn't even think to remove it before since it doesn't show up in the config.  Here's what I'm seeing now:

 

Routed OSPF - ipv4 offload enabled (~0-3% CPU on each)

[root@Artemis asatnik]# /usr/local/bin/iperf3 -c 192.168.34.2
Connecting to host 192.168.34.2, port 5201
[  4] local 192.168.23.2 port 56718 connected to 192.168.34.2 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   113 MBytes   947 Mbits/sec    0    434 KBytes       
[  4]   1.00-2.00   sec   112 MBytes   943 Mbits/sec    0    441 KBytes       
[  4]   2.00-3.00   sec   112 MBytes   942 Mbits/sec    0    441 KBytes       
[  4]   3.00-4.00   sec   112 MBytes   942 Mbits/sec    0    441 KBytes       
[  4]   4.00-5.00   sec   112 MBytes   941 Mbits/sec    0    441 KBytes       
[  4]   5.00-6.00   sec   112 MBytes   942 Mbits/sec    0    443 KBytes       
[  4]   6.00-7.00   sec   112 MBytes   941 Mbits/sec    0    443 KBytes       
[  4]   7.00-8.00   sec   112 MBytes   942 Mbits/sec    0    444 KBytes       
[  4]   8.00-9.00   sec   112 MBytes   941 Mbits/sec    0    444 KBytes       
[  4]   9.00-10.00  sec   112 MBytes   941 Mbits/sec    0    447 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.10 GBytes   942 Mbits/sec    0             sender
[  4]   0.00-10.00  sec  1.10 GBytes   942 Mbits/sec                  receiver

iperf Done.

Routed OSPF - ipv4 offload disabled (~50-70% CPU on each)

[root@Artemis asatnik]# /usr/local/bin/iperf3 -c 192.168.34.2
Connecting to host 192.168.34.2, port 5201
[  4] local 192.168.23.2 port 56729 connected to 192.168.34.2 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   111 MBytes   935 Mbits/sec   58   74.9 KBytes       
[  4]   1.00-2.00   sec   112 MBytes   940 Mbits/sec   54   89.1 KBytes       
[  4]   2.00-3.00   sec   112 MBytes   941 Mbits/sec   23    113 KBytes       
[  4]   3.00-4.00   sec   112 MBytes   942 Mbits/sec   33   82.0 KBytes       
[  4]   4.00-5.00   sec   112 MBytes   938 Mbits/sec   43   87.7 KBytes       
[  4]   5.00-6.00   sec   112 MBytes   941 Mbits/sec    5    156 KBytes       
[  4]   6.00-7.00   sec   112 MBytes   939 Mbits/sec   92    112 KBytes       
[  4]   7.00-8.00   sec   112 MBytes   941 Mbits/sec   19    147 KBytes       
[  4]   8.00-9.00   sec   112 MBytes   940 Mbits/sec   67    113 KBytes       
[  4]   9.00-10.00  sec   111 MBytes   935 Mbits/sec  137   86.3 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.09 GBytes   939 Mbits/sec  531             sender
[  4]   0.00-10.00  sec  1.09 GBytes   939 Mbits/sec                  receiver

iperf Done.

 

Routed through IPSec VPN (AES128/SHA1) - IPSec Offload Enabled/ipv4 forwarding offload enabled (~90-100% CPU on each)

 

[root@Artemis asatnik]# /usr/local/bin/iperf3 -c 192.168.34.2
Connecting to host 192.168.34.2, port 5201
[  4] local 192.168.23.2 port 56743 connected to 192.168.34.2 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  47.4 MBytes   398 Mbits/sec    1    138 KBytes       
[  4]   1.00-2.00   sec  57.1 MBytes   479 Mbits/sec    0    202 KBytes       
[  4]   2.00-3.00   sec  56.5 MBytes   474 Mbits/sec    0    246 KBytes       
[  4]   3.00-4.00   sec  56.0 MBytes   470 Mbits/sec    0    284 KBytes       
[  4]   4.00-5.00   sec  55.6 MBytes   467 Mbits/sec    0    319 KBytes       
[  4]   5.00-6.00   sec  55.3 MBytes   464 Mbits/sec    0    349 KBytes       
[  4]   6.00-7.00   sec  55.6 MBytes   467 Mbits/sec    0    378 KBytes       
[  4]   7.00-8.00   sec  54.8 MBytes   459 Mbits/sec    0    403 KBytes       
[  4]   8.00-9.00   sec  54.9 MBytes   461 Mbits/sec    0    428 KBytes       
[  4]   9.00-10.00  sec  54.4 MBytes   456 Mbits/sec    0    567 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   548 MBytes   459 Mbits/sec    1             sender
[  4]   0.00-10.00  sec   547 MBytes   459 Mbits/sec                  receiver

iperf Done.

 

Routed through IPSec VPN (AES128/SHA1) - IPSec Offload Enabled/ipv4 forwarding offload disabled (~90-100% CPU on each)

[root@Artemis asatnik]# /usr/local/bin/iperf3 -c 192.168.34.2
Connecting to host 192.168.34.2, port 5201
[  4] local 192.168.23.2 port 56731 connected to 192.168.34.2 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  52.9 MBytes   444 Mbits/sec    7    149 KBytes       
[  4]   1.00-2.00   sec  61.1 MBytes   512 Mbits/sec    0    210 KBytes       
[  4]   2.00-3.00   sec  60.2 MBytes   505 Mbits/sec    0    257 KBytes       
[  4]   3.00-4.00   sec  60.0 MBytes   503 Mbits/sec    0    296 KBytes       
[  4]   4.00-5.00   sec  59.7 MBytes   501 Mbits/sec    0    332 KBytes       
[  4]   5.00-6.00   sec  58.5 MBytes   491 Mbits/sec    0    363 KBytes       
[  4]   6.00-7.00   sec  58.8 MBytes   493 Mbits/sec    0    391 KBytes       
[  4]   7.00-8.00   sec  57.3 MBytes   481 Mbits/sec    0    418 KBytes       
[  4]   8.00-9.00   sec  57.2 MBytes   480 Mbits/sec    0    443 KBytes       
[  4]   9.00-10.00  sec  57.0 MBytes   478 Mbits/sec    0    583 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   583 MBytes   489 Mbits/sec    7             sender
[  4]   0.00-10.00  sec   582 MBytes   488 Mbits/sec                  receiver

iperf Done.

 

Routed through IPSec VPN - IPSec Offload Disabled/ipv4 forwarding offload enabled (~90-100% CPU on each)

 

[root@Artemis asatnik]# /usr/local/bin/iperf3 -c 192.168.34.2
Connecting to host 192.168.34.2, port 5201
[  4] local 192.168.23.2 port 56759 connected to 192.168.34.2 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  14.9 MBytes   125 Mbits/sec    5   92.0 KBytes       
[  4]   1.00-2.00   sec  19.9 MBytes   167 Mbits/sec    0    126 KBytes       
[  4]   2.00-3.00   sec  20.3 MBytes   170 Mbits/sec    0    156 KBytes       
[  4]   3.00-4.00   sec  20.4 MBytes   171 Mbits/sec    0    179 KBytes       
[  4]   4.00-5.00   sec  20.3 MBytes   171 Mbits/sec    0    199 KBytes       
[  4]   5.00-6.00   sec  20.2 MBytes   170 Mbits/sec    0    218 KBytes       
[  4]   6.00-7.00   sec  20.5 MBytes   172 Mbits/sec    0    236 KBytes       
[  4]   7.00-8.00   sec  20.4 MBytes   171 Mbits/sec    0    303 KBytes       
[  4]   8.00-9.00   sec  20.4 MBytes   171 Mbits/sec    0    409 KBytes       
[  4]   9.00-10.00  sec  20.1 MBytes   169 Mbits/sec    0    543 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   197 MBytes   166 Mbits/sec    5             sender
[  4]   0.00-10.00  sec   197 MBytes   165 Mbits/sec                  receiver

iperf Done.

 

Routed through IPSec VPN - IPSec Offload Disabled/ipv4 forwarding offload disabled (~90-100% CPU on each)

[root@Artemis asatnik]# /usr/local/bin/iperf3 -c 192.168.34.2
Connecting to host 192.168.34.2, port 5201
[  4] local 192.168.23.2 port 56763 connected to 192.168.34.2 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  13.9 MBytes   116 Mbits/sec    7   81.2 KBytes       
[  4]   1.00-2.00   sec  19.6 MBytes   165 Mbits/sec    0    118 KBytes       
[  4]   2.00-3.00   sec  20.5 MBytes   172 Mbits/sec    0    149 KBytes       
[  4]   3.00-4.00   sec  20.8 MBytes   174 Mbits/sec    0    173 KBytes       
[  4]   4.00-5.00   sec  20.8 MBytes   175 Mbits/sec    0    195 KBytes       
[  4]   5.00-6.00   sec  20.6 MBytes   173 Mbits/sec    0    214 KBytes       
[  4]   6.00-7.00   sec  20.6 MBytes   173 Mbits/sec    0    233 KBytes       
[  4]   7.00-8.00   sec  20.7 MBytes   174 Mbits/sec   36    246 KBytes       
[  4]   8.00-9.00   sec  21.1 MBytes   177 Mbits/sec    0    261 KBytes       
[  4]   9.00-10.00  sec  21.0 MBytes   176 Mbits/sec    0    276 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   200 MBytes   167 Mbits/sec   43             sender
[  4]   0.00-10.00  sec   199 MBytes   167 Mbits/sec                  receiver

iperf Done.
New Member
Posts: 26
Registered: ‎09-26-2014
Kudos: 10
Solutions: 1

Re: ERL Performance Testing (with IPSec VPN)

[ Edited ]

Thanks! Just the test I was looking for!

Correct me if I'm wrong, but the EdgeRouter Pro has a dual core processor. So when you say 90-100% is that on one core? Or both?

 

EDIT: Whoops sorry. I read your post again. It says 90-100% on each. I also noticed on the ERL that having the webpage open increased my load average. So I turned it off for the test. Was yours open? 

New Member
Posts: 21
Registered: ‎09-22-2014
Kudos: 20

Re: ERL Performance Testing (with IPSec VPN)

[ Edited ]

Its 90-100% according to 'top'.  System load average is over 2 so I'm assuming it's loading both cores.  Web interface closed on this one - IPSec (AES128/SHA1), ipv4 forwarding and ipsec offload enabled.

 

top - 21:58:55 up 11:59,  1 user,  load average: 3.42, 1.13, 0.43
Tasks:  64 total,   4 running,  60 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.2%us,  0.3%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi, 99.5%si,  0.0%st
Mem:   2044664k total,   151592k used,  1893072k free,    21120k buffers
Swap:        0k total,        0k used,        0k free,    80880k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                    
    9 root      20   0     0    0    0 R   53  0.0   0:36.80 ksoftirqd/1                                                 
    3 root      20   0     0    0    0 R   44  0.0   0:39.30 ksoftirqd/0                                                 
   10 root      20   0     0    0    0 R   44  0.0   0:27.21 kworker/0:1                                                 
    8 root      20   0     0    0    0 S   22  0.0   1:06.54 kworker/1:0                                                 
 3695 asatnik   20   0 11072 1388  876 S   11  0.1   0:07.35 sshd                                                        
  496 root      20   0  4732  748  584 S    5  0.0   0:08.48 ubnt-infctld                                                
 1548 www-data  20   0  6524 2692 1716 S    4  0.1   0:02.45 lighttpd                                                    
   13 root      20   0     0    0    0 S    3  0.0   0:05.86 kworker/u:1                                                 
  423 root      20   0  2004  312  244 S    3  0.0   0:09.21 rngd                                                        
  509 root      20   0  2824  276  164 S    3  0.0   0:04.35 ubnt-daemon                                                 
 4381 asatnik   20   0  3276 1280 1056 R    2  0.1   0:03.41 top                                                         
 1018 ntp       20   0  4920 1492 1236 S    2  0.1   0:07.58 ntpd                  
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-30.00  sec  1.60 GBytes   458 Mbits/sec   18             sender
[  4]   0.00-30.00  sec  1.60 GBytes   458 Mbits/sec                  receiver

 

 

New Member
Posts: 21
Registered: ‎09-22-2014
Kudos: 20

Re: ERL Performance Testing (with IPSec VPN)

Let me know if there's anything else you'd like me to test while I have these available.  They should be available for another week or two.  

New Member
Posts: 5
Registered: ‎07-08-2015
Kudos: 1

Re: ERL Performance Testing (with IPSec VPN)

I was wondering how quite fast ER lite for its cost.

Using about 20 boxes as vpn terminators for small branches led me see performance drops with newer firmwares.

 

Here is what i found (same config as above):

v1.8 approx 38.6 Mbps

 

Server listening on 1500
-----------------------------------------------------------
Accepted connection from 172.27.144.194, port 17530
[ 5] local 172.27.244.194 port 1500 connected to 172.27.144.194 port 17531
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 3.67 MBytes 30.8 Mbits/sec
[ 5] 1.00-2.00 sec 4.40 MBytes 36.9 Mbits/sec
[ 5] 2.00-3.00 sec 4.23 MBytes 35.6 Mbits/sec
[ 5] 3.00-4.00 sec 4.75 MBytes 39.7 Mbits/sec
[ 5] 4.00-5.01 sec 4.83 MBytes 40.5 Mbits/sec
[ 5] 5.01-6.00 sec 4.58 MBytes 38.4 Mbits/sec
[ 5] 6.00-7.01 sec 4.91 MBytes 41.1 Mbits/sec
[ 5] 7.01-8.01 sec 4.76 MBytes 39.8 Mbits/sec
[ 5] 8.01-9.01 sec 4.89 MBytes 40.9 Mbits/sec
[ 5] 9.01-10.02 sec 4.95 MBytes 41.4 Mbits/sec
[ 5] 10.02-10.20 sec 957 KBytes 41.9 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.20 sec 0.00 Bytes 0.00 Mbits/sec sender
[ 5] 0.00-10.20 sec 46.9 MBytes 38.6 Mbits/sec receiver

 

v1.7 approx 45.1 Mbps

Accepted connection from 172.27.144.194, port 17532
[ 5] local 172.27.244.194 port 1500 connected to 172.27.144.194 port 17533
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.01 sec 4.56 MBytes 37.7 Mbits/sec
[ 5] 1.01-2.01 sec 5.50 MBytes 46.1 Mbits/sec
[ 5] 2.01-3.01 sec 5.25 MBytes 44.0 Mbits/sec
[ 5] 3.01-4.01 sec 5.56 MBytes 46.8 Mbits/sec
[ 5] 4.01-5.01 sec 5.54 MBytes 46.5 Mbits/sec
[ 5] 5.01-6.01 sec 5.52 MBytes 46.4 Mbits/sec
[ 5] 6.01-7.01 sec 5.35 MBytes 44.9 Mbits/sec
[ 5] 7.01-8.01 sec 5.69 MBytes 47.7 Mbits/sec
[ 5] 8.01-9.01 sec 5.46 MBytes 45.9 Mbits/sec
[ 5] 9.01-10.01 sec 5.36 MBytes 45.0 Mbits/sec
[ 5] 10.01-10.20 sec 1.07 MBytes 47.6 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.20 sec 0.00 Bytes 0.00 Mbits/sec sender
[ 5] 0.00-10.20 sec 54.9 MBytes 45.1 Mbits/sec receiver

 

v1.4 approx 61 Mbps

Accepted connection from 172.27.144.194, port 18346
[ 5] local 172.27.244.194 port 1500 connected to 172.27.144.194 port 18347
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 5.90 MBytes 49.4 Mbits/sec
[ 5] 1.00-2.00 sec 7.53 MBytes 63.2 Mbits/sec
[ 5] 2.00-3.01 sec 7.21 MBytes 59.7 Mbits/sec
[ 5] 3.01-4.02 sec 7.41 MBytes 62.2 Mbits/sec
[ 5] 4.02-5.01 sec 7.12 MBytes 59.8 Mbits/sec
[ 5] 5.01-6.01 sec 7.49 MBytes 63.0 Mbits/sec
[ 5] 6.01-7.01 sec 7.44 MBytes 62.4 Mbits/sec
[ 5] 7.01-8.01 sec 7.67 MBytes 64.5 Mbits/sec
[ 5] 8.01-9.01 sec 7.58 MBytes 63.6 Mbits/sec
[ 5] 9.01-10.01 sec 7.37 MBytes 61.8 Mbits/sec
[ 5] 10.01-10.20 sec 1.38 MBytes 61.9 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.20 sec 0.00 Bytes 0.00 Mbits/sec sender
[ 5] 0.00-10.20 sec 74.1 MBytes 61.0 Mbits/sec receiver

 

 

v1.2 approx 77 Mbps

Accepted connection from 172.27.144.194, port 5292
[ 5] local 172.27.244.194 port 1500 connected to 172.27.144.194 port 5293
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.01 sec 7.69 MBytes 63.6 Mbits/sec
[ 5] 1.01-2.01 sec 9.35 MBytes 78.4 Mbits/sec
[ 5] 2.01-3.01 sec 9.17 MBytes 77.0 Mbits/sec
[ 5] 3.01-4.01 sec 9.46 MBytes 79.5 Mbits/sec
[ 5] 4.01-5.01 sec 9.22 MBytes 77.3 Mbits/sec
[ 5] 5.01-6.01 sec 9.25 MBytes 77.6 Mbits/sec
[ 5] 6.01-7.01 sec 9.47 MBytes 79.2 Mbits/sec
[ 5] 7.01-8.01 sec 9.52 MBytes 79.8 Mbits/sec
[ 5] 8.01-9.01 sec 9.38 MBytes 78.8 Mbits/sec
[ 5] 9.01-10.01 sec 9.26 MBytes 77.6 Mbits/sec
[ 5] 10.01-10.22 sec 2.04 MBytes 83.4 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.22 sec 0.00 Bytes 0.00 Mbits/sec sender
[ 5] 0.00-10.22 sec 93.8 MBytes 77.0 Mbits/sec receiver

Member
Posts: 113
Registered: ‎08-09-2014
Kudos: 122
Solutions: 1

Re: ERL Performance Testing (with IPSec VPN)

[ Edited ]

@UBNT-ancheng Seeing the same as above. UBNT could consider working on reclaiming some of the lost performance in newer releases?

Member
Posts: 113
Registered: ‎08-09-2014
Kudos: 122
Solutions: 1

Re: ERL Performance Testing (with IPSec VPN)

Shameless bump.

Emerging Member
Posts: 95
Registered: ‎11-08-2015
Kudos: 35
Solutions: 4

Re: ERL Performance Testing (with IPSec VPN)

There are probably some optimizations possible (no programmer is perfect), but what you're likely seeing is the effect of adding new features, and especially of default enabling hardware offload functionality. I've specifically seen references to that eventually slowing down the offload to the point where there's no longer any benefit to offloading.

 

Try recreating the test and disabling everything but IPv4 forwarding and IPsec offloads.

New Member
Posts: 30
Registered: ‎01-24-2014
Kudos: 7

Re: ERL Performance Testing (with IPSec VPN)

Hi all,

 

Does anyone have a recent performance test using the latest firmware (v1.9.0 Build ID: 4901118) on the EdgeRouter Pro or EdgeRouter Lite?  If not, I'll setup a system to test.

 

Thanks!

Emerging Member
Posts: 72
Registered: ‎06-11-2016
Kudos: 12
Solutions: 2

Re: ERL Performance Testing (with IPSec VPN)


@kylebassett wrote:

Hi all,

 

Does anyone have a recent performance test using the latest firmware (v1.9.0 Build ID: 4901118) on the EdgeRouter Pro or EdgeRouter Lite?  If not, I'll setup a system to test.

 

Thanks!


I'd be very interested, especially with NAT-T (UDP transport) enabled on the ERLITE.

I am getting strange numbers, asymmetrical speeds that I had initially ascribed to PMTU issues, but on a closer look are definitely not.

 

Right now I am seeing in one direction 10-15Mbps and about 20 in the other, offload is enabled and CPU usage is negligible, link BW is 100Mbps symmetrical, site-to-site IPsec NAT-T over a plain simple tunnel, no GRE, no VTI.

New Member
Posts: 6
Registered: ‎02-29-2016
Kudos: 1
Solutions: 1

Re: ERL Performance Testing (with IPSec VPN)

I made performance test with same scenario like jdrews.
Routers:
- EdgeRouter X SFP, type ER-X-SFP, EdgeOS v. 1.9.1
- EdgeRouter X, type ER-X, EdgeOS v. 1.9.1

 

HW offload disabled:

[ ID] Interval Transfer Bandwidth
[ 5] 0.00-60.06 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-60.06 sec 273 MBytes 38.2 Mbits/sec receiver

CPU approx. 30-40% according to webinterface

 

 

HW offload enabled:

[ ID] Interval Transfer Bandwidth
[ 5] 0.00-60.06 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-60.06 sec 909 MBytes 127 Mbits/sec receiver

 

CPU approx. 40% according to webinterface

 

I have not seen any difference between webinterface on (dashboard) and off.

SuperUser
Posts: 14,588
Registered: ‎12-08-2008
Kudos: 11381
Solutions: 697
Contributions: 1

Re: ERL Performance Testing (with IPSec VPN)


@janulo wrote:

I made performance test with same scenario like jdrews.
Routers:
- EdgeRouter X SFP, type ER-X-SFP, EdgeOS v. 1.9.1
- EdgeRouter X, type ER-X, EdgeOS v. 1.9.1

 

HW offload disabled:

[ ID] Interval Transfer Bandwidth
[ 5] 0.00-60.06 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-60.06 sec 273 MBytes 38.2 Mbits/sec receiver

CPU approx. 30-40% according to webinterface

 

 

HW offload enabled:

[ ID] Interval Transfer Bandwidth
[ 5] 0.00-60.06 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-60.06 sec 909 MBytes 127 Mbits/sec receiver

 

CPU approx. 40% according to webinterface

 

I have not seen any difference between webinterface on (dashboard) and off.


That's actually pretty impressive, and better than I would have guessed for a $50 device...

Jim

" How can anyone trust Scientists? If new evidence comes along, they change their minds! " Politician's joke (sort of...)
"Humans are allergic to change..They love to say, ‘We’ve always done it this way.’ I try to fight that. "Admiral Grace Hopper, USN, Computer Scientist
"It's not Rocket Science! - Oh wait, Actually it is... "NASA bumper sticker
"Just because you can do something doesn't mean you should."my mantra in the Programming classes I used to teach once upon a time...
Regular Member
Posts: 656
Registered: ‎06-27-2016
Kudos: 228
Solutions: 30

Re: ERL Performance Testing (with IPSec VPN)


@janulo wrote:

I made performance test with same scenario like jdrews.
Routers:
- EdgeRouter X SFP, type ER-X-SFP, EdgeOS v. 1.9.1
- EdgeRouter X, type ER-X, EdgeOS v. 1.9.1

 

 

HW offload enabled:

[ ID] Interval Transfer Bandwidth
[ 5] 0.00-60.06 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-60.06 sec 909 MBytes 127 Mbits/sec receiver

 


 

Which HW offload?  IPSec, HWNAT, or both?  You can check with:

 

show ubnt offload