Reply
New Member
Posts: 27
Registered: ‎12-18-2017

ER-4 performance with VLAN configuration

[ Edited ]

Guys,

 

I've upgraded my whole home network, replaced ERLite-3 with ER-4 and replaced old Cisco and Netgear unmanaged switches with with ES-8Xs. The main driver was to isolate IoT devices from home network and improve security. All good, everything works great except the inter-vlan and vlan to eth0 (external) performance.

 

I have Comcast Xfinity 400/20 service installed, on regular basis I can get about 450 Mbit/s down. Since installed ER-4 I noticed significant difference in performance between regular interface e.g eth2 versus et3.10 or eth3.20 (my VLANs). All tests equally show max throughput about 250-300 Mbit/s when connected to VLAN. I've configured VLAN offload it helps a little bit. It seems I get the peak speed at the beginning of the test, and when the buffers are full, speed goes down. Is this normal and expected? Can I tweak anything to get better results?

 

Thanks
Mario

Established Member
Posts: 751
Registered: ‎02-12-2013
Kudos: 189
Solutions: 59

Re: ER-4 performance with VLAN configuration

Hi @marnow
If you have VLAN offloading enabled - and no other configuration, which disables offloading (QoS, netflow, bridge) - then I would recommend that you post your configuration (sanitize where needed).
https://help.ubnt.com/hc/en-us/articles/115006567467-EdgeRouter-Hardware-Offloading
New Member
Posts: 27
Registered: ‎12-18-2017

Re: ER-4 performance with VLAN configuration

Here is the current running config:

 

firewall {
    all-ping enable
    broadcast-ping disable
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians enable
    name IoT_IN {
        default-action accept
        description ""
        rule 10 {
            action accept
            description "Allow established/related"
            log disable
            protocol all
            state {
                established enable
                invalid disable
                new disable
                related enable
            }
        }
        rule 20 {
firewall {
    all-ping enable
    broadcast-ping disable
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians enable
    name IoT_IN {
        default-action accept
        description ""
        rule 10 {
            action accept
            description "Allow established/related"
            log disable
            protocol all
            state {
                established enable
                invalid disable
                new disable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            log disable
            protocol all
            state {
                established disable
                invalid enable
                new disable
                related disable
            }
        }
        rule 30 {
            action drop
            description "All traffic drop"
            destination {
                address 192.168.0.0/24
            }
            log disable
            protocol all
        }
    }
    name IoT_Local {
        default-action drop
        description ""
        rule 1 {
            action accept
            description "Local DNS"
            destination {
                address 192.168.5.1
                port 53
            }
            log disable
            protocol tcp_udp
        }
    }
    name WAN_IN {
        default-action drop
        description "WAN to internal"
        rule 10 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
    }
    name WAN_LOCAL {
        default-action drop
        description "WAN to router"
        rule 10 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
    }
    receive-redirects disable
    send-redirects enable
    source-validation disable
    syn-cookies enable
}
interfaces {
    ethernet eth0 {
        address dhcp
        description Internet
        dhcp-options {
            default-route update
            default-route-distance 210
            name-server no-update
        }
        duplex auto
        firewall {
            in {
                name WAN_IN
            }
            local {
                name WAN_LOCAL
            }
        }
        speed auto
    }
    ethernet eth1 {
        disable
        duplex auto
        speed auto
    }
    ethernet eth2 {
        address 192.168.2.1/24
        description "Local 2"
        duplex auto
        speed auto
    }
    ethernet eth3 {
        description Local
        duplex auto
        speed auto
        vif 10 {
            address 192.168.0.1/24
            description "Secure LAN"
        }
        vif 20 {
            address 192.168.5.1/24
            description IoT
            firewall {
                in {
                    name IoT_IN
                }
                local {
                    name IoT_Local
                }
            }
        }
        vif 30 {
            address 192.168.30.1/24
            description AP
            mtu 1500
        }
    }
    loopback lo {
    }
}
service {
    dhcp-server {
        disabled false
        hostfile-update disable
        shared-network-name IoT {
            authoritative enable
            subnet 192.168.5.0/24 {
                default-router 192.168.5.1
                dns-server 192.168.5.1
                dns-server 1.1.1.1
                domain-name iot.marnet
                lease 86400
                start 192.168.5.10 {
                    stop 192.168.5.200
                }
            }
        }
        shared-network-name SECLAN {
            authoritative enable
            subnet 192.168.0.0/24 {
                default-router 192.168.0.1
                dns-server 192.168.0.1
                dns-server 1.1.1.1
                domain-name local.marnet
                lease 86400
                start 192.168.0.100 {
                    stop 192.168.0.243
                }
            }
        }
        static-arp disable
        use-dnsmasq disable
    }
    dns {
        dynamic {
            interface eth0 {
                service custom-DyNu {
                    host-name XXXXX
                    login *******
                    password ****************
                    protocol dyndns2
                    server api.dynu.com
                }
            }
        }
        forwarding {
            cache-size 3000
            except-interface eth0
            name-server 1.1.1.1
            name-server 1.0.0.1
            name-server 8.8.8.8
            name-server 208.67.222.222
            options strict-order
        }
    }
    gui {
        http-port 80
        https-port 443
        older-ciphers enable
    }
    nat {
        rule 5010 {
            description "masquerade for WAN"
            outbound-interface eth0
            type masquerade
        }
    }
    ssh {
        port 22
        protocol-version v2
    }
    unms {
        disable
    }
    upnp {
        listen-on eth3 {
            outbound-interface eth0
        }
        listen-on eth3.10 {
            outbound-interface eth0
        }
    }
}
system {
    config-management {
        commit-revisions 20
    }
    conntrack {
        expect-table-size 2048
        hash-size 32768
        table-size 262144
    }
    domain-name marnet.local
    host-name rtr-marnet-1G
    login {
        user ubnt {
            authentication {
                encrypted-password ****************
            }
            level admin
        }
    }
    name-server 1.1.1.1
    name-server 1.0.0.1
    name-server 8.8.8.8
    name-server 208.67.222.222
    ntp {
        server 0.ubnt.pool.ntp.org {
        }
        server 1.ubnt.pool.ntp.org {
        }
        server 2.ubnt.pool.ntp.org {
        }
        server 3.ubnt.pool.ntp.org {
        }
    }
    offload {
        hwnat disable
        ipsec enable
        ipv4 {
            forwarding enable
            table-size 16384
            vlan enable
        }
        ipv6 {
            forwarding disable
        }
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone America/Detroit
    traffic-analysis {
        dpi disable
        export disable
    }
}
traffic-control {
}
Established Member
Posts: 751
Registered: ‎02-12-2013
Kudos: 189
Solutions: 59

Re: ER-4 performance with VLAN configuration

@marnow
It looks like QoS was enabled at some point - I'm pretty sure you have to reboot for offloading to enable, if you just removed QoS.
Which firmware version are you using?
What do you see if you run this command?
show ubnt offload
Highlighted
New Member
Posts: 27
Registered: ‎12-18-2017

Re: ER-4 performance with VLAN configuration

@flamber Thanks for your help!

 

Yes, the QoS SmartQueue was enabled at some point, but I removed it and reboot the router since offload was enabled.

 

I run the latest v1.10.8.

 

IP offload module   : loaded
IPv4
  forwarding: enabled
  vlan      : enabled
  pppoe     : disabled
  gre       : disabled
IPv6
  forwarding: disabled
  vlan      : disabled
  pppoe     : disabled

IPSec offload module: loaded

Traffic Analysis    :
  export    : disabled
  dpi       : disabled
    version       : 1.422

 

 

 

 

New Member
Posts: 27
Registered: ‎12-18-2017

Re: ER-4 performance with VLAN configuration

@flamber is the information sufficient?
Established Member
Posts: 751
Registered: ‎02-12-2013
Kudos: 189
Solutions: 59

Re: ER-4 performance with VLAN configuration

@marnow
I'm not sure what's going on - I was hoping someone else would spot something - let's see if Ben has time? @UBNT-benpin
Member
Posts: 584
Registered: ‎09-13-2018
Kudos: 103
Solutions: 33

Re: ER-4 performance with VLAN configuration

[ Edited ]

Just to be sure we are measuring the inter-vlan routing performance, can you answer the following?

 

How are you measuring the inter-vlan throughput?  What packet size is being used?

 

Are all connections between the test devices wired with cat6 or cat5e cable (no wifi, powerline adapters, etc. involved)?

 

 

New Member
Posts: 27
Registered: ‎12-18-2017

Re: ER-4 performance with VLAN configuration

[ Edited ]

@BuckeyeNet I just copy a large binary file from one laptop to my desktop...both hardwire cat5e cables. MTU size 1500, I tried 9000, but I did't see much difference. For internet speed mesurments I used various testing websites.

Member
Posts: 148
Registered: ‎06-13-2018
Kudos: 15
Solutions: 2

Re: ER-4 performance with VLAN configuration

[ Edited ]

Have you tried iperf between interfaces/VLANs? Internet speedtests offer you little control past your router. Even with all offloads disabled you should easily hit more than the numbers you are seeing.

 

I can hit nearly 1.2Gbps aggregate throughout across VLANs on my ER12 with all offloads off.

New Member
Posts: 27
Registered: ‎12-18-2017

Re: ER-4 performance with VLAN configuration

No, I haven't tried Iperf yet. I understand what you said....but comparing regular port (non-vlan) to port with sub-interfaces I see huge difference. Is not just a single measurement, single site.

Member
Posts: 584
Registered: ‎09-13-2018
Kudos: 103
Solutions: 33

Re: ER-4 performance with VLAN configuration

[ Edited ]

I have to agree with @andybgrant.  If you want significant results, you need to use tools that will give consistent measurements.  Download iperf3 and install on the laptop and desktop.  run one in client mode and one in server mode.

 

iperf uses in memory buffers, so no disk I/O is involved, and uses little CPU, so you are more likely to be measuring the network instead of your PC.

 

Can you configure eth1 with an ip address and dns server, and connect your laptop directly there, run iperf between it and the desktop (on eth2).  Then move the laptop to where the "secure" lan, remove the other vlans from the trunk port on the switch so other traffic can't impact the results, and retest.   Then compare results.

Member
Posts: 584
Registered: ‎09-13-2018
Kudos: 103
Solutions: 33

Re: ER-4 performance with VLAN configuration

[ Edited ]

This should have no effect, but what type of SFP module are you using in the ER4 and switch?  Are you using fiber or an RJ45 SFP module?  I was surprised to see that you are not using eth1, but eth3 (which requires an extra purchase of SFP module).

New Member
Posts: 27
Registered: ‎12-18-2017

Re: ER-4 performance with VLAN configuration

You are correct I use fiber uplink between the router and the switch, both SFPs are Ubiquity brand.  

Member
Posts: 148
Registered: ‎06-13-2018
Kudos: 15
Solutions: 2

Re: ER-4 performance with VLAN configuration

[ Edited ]

@marnow wrote:

No, I haven't tried Iperf yet. I understand what you said....but comparing regular port (non-vlan) to port with sub-interfaces I see huge difference. Is not just a single measurement, single site.


Are those sub-interfaces connected to a switch? [Edit] I type slow and you already answered above [/edit] Can you test without it in order to minimize variables?

 

I have a couple interfaces that can readily be changed to and from sub-interfaces. I'll run some iperfs across them tomorrow to see if there is a decernible difference.

Member
Posts: 148
Registered: ‎06-13-2018
Kudos: 15
Solutions: 2

Re: ER-4 performance with VLAN configuration

[ Edited ]

This was more curiosity than anything else since an ER12 isn't an ER4 but close enough Man Wink

 

Two VM's configured on seperate vswitches each with an ancient Intel e1000 (seriously old, no TSO, LRO and miss-matched fixed rx/tx hardware queue settings).  Ran iperf in both directions using;

  • iperf3 -c 10.10.5.5 -P 2 -M 1500 -t 500 -w 128K
  • iperf3 -c 10.10.5.5 -P 2 -M 1500 -t 500 -w 128K -p 5202 -R 

Here we have straight eth to eth

no sub-int.jpg

 

And here is changing eth8 to a sub-interface with no other changes

sub-int.jpg

 

[edit] PS - offloads off

 

Member
Posts: 584
Registered: ‎09-13-2018
Kudos: 103
Solutions: 33

Re: ER-4 performance with VLAN configuration

What did iperf report (did it agree with the edgerouter?)

 

If would be interesting to compare it to two access ports in the same vlan to see what the pcs can push without routing being done.  I doubt you will see anything on the ER12 dashboard though.  That would be a baseline (not to exceed speed).

 

Also it would be interesting to try with offload enabled for forwarding, and then with forwarding and vlan offloaded, just to see if there is still as much difference.

 

Was there one sender and one receiver process on the same PC with a two port card?

 

Thanks for running this test, it's the first I have seen for the ER12. 

 

The results you got should be very similar to an ER4, since you were using ports that were not part of the switch.

Member
Posts: 148
Registered: ‎06-13-2018
Kudos: 15
Solutions: 2

Re: ER-4 performance with VLAN configuration


@BuckeyeNet wrote:

What did iperf report (did it agree with the edgerouter?)


It was pretty spot on. Hard to capture both in a single screenshot so I chose the ER GUI since it had the interface configuration.

 


@BuckeyeNet wrote:

 

If would be interesting to compare it to two access ports in the same vlan to see what the pcs can push without routing being done.  I doubt you will see anything on the ER12 dashboard though.  That would be a baseline (not to exceed speed).


I have tried this on two switch ports; 930-950Mbps.  Are you suggesting two non-switch enabled ports in a bond?

 


@BuckeyeNet wrote:

Also it would be interesting to try with offload enabled for forwarding, and then with forwarding and vlan offloaded, just to see if there is still as much difference.


I would like to ultimately compare, offloaded, non-offloaded and SFE (once GPL sources released and if I can get it compiled). In addition to that I would like to attempt to verify the CPU to switch interface speed.

 


@BuckeyeNet wrote:

Was there one sender and one receiver process on the same PC with a two port card?


Each VM acted as both a sender and receiver across two seperate cards simultaneously. Depending on the sender/receiver I have seen anywhere from 750-950Mbps in one direction. I found it interesting that a uni-directional iperf of 900+Mbps would be cut in half when starting another in the opposite direction on the same 3 vCPU VM. ACKS alone are only worth ~10Mbps return traffic so there has to be another bottleneck. Perhaps I'll try separate VM's

 

With four iperf client/servers running across four ER12 ports I hit ~1.2Gbps before maxing CPU with offload disabled and other services running.  

Member
Posts: 148
Registered: ‎06-13-2018
Kudos: 15
Solutions: 2

Re: ER-4 performance with VLAN configuration

I will definitely have to experiment with the best combination of iper3 client/server combinations.  Instead of testing bi-directionally I tested two in the same direction while pinning to seperate vCPU's and was able to better saturate a link in a single direction.  Seems it may be a combination of hitting individual VM vCPU saturation (per esxtop) and perhaps some router behaviour since pinning to seperate vCPU's does not change the bi-directional results.

 

better.jpg

New Member
Posts: 27
Registered: ‎12-18-2017

Re: ER-4 performance with VLAN configuration

[ Edited ]

Okey! Thank you guys! I will install iperf and play with it over the coming weekend.

Reply