Reply
New Member
Posts: 8
Registered: ‎12-15-2017

GRE Offload on ER-X

[ Edited ]

I have been testing the performance of various tunnel protocols and encryption on the ER-X platform with offloading enabled and disabled. I have noticed that offloading of GRE does not appear to be working, or at least has no impact on the performance of the tunnel.

 

I have tested this both for gre-bridge and gre tunnels, with and without encryption. The solution is as simple as possible, two routers connected by an ethernet cable and the tunnel between them and a machine on either side for bandwith testing. With simple routing I can, achieve ~1Gbps of throughput. Using a GRE tunnel drops that to ~150Mbps. Is this to be expected?

 

EDIT: Forgot, I am running version 1.10.1

Member
Posts: 219
Registered: ‎02-12-2013
Kudos: 69
Solutions: 18

Re: GRE Offload on ER-X

Hi @jtf6xb
Are you using QoS or NetFlow or bonding? If yes, then those disables the offloading:
https://help.ubnt.com/hc/en-us/articles/115006567467-EdgeRouter-Hardware-Offloading-Explained
Please post your configuration and also run 'show ubnt offload'
New Member
Posts: 8
Registered: ‎12-15-2017

Re: GRE Offload on ER-X

This is in a lab environment so the only thing enabled is the gre tunnel.

 

I rebuilt the lab so I could test again and provide the configuration. I did find that offloading does appear to be working for a simple gre tunnel, but only if the ER-X is rebooted after creating the tunnel. Offloading does still not appear to be working for a bridged gre tunnel config. The configs for the bridged solution are listed below.

 

ubnt@ubnt:~$ show ubnt offload
IPSec offload module: loaded

HWNAT offload module: loaded

Traffic Analysis    :
  export    : disabled
  dpi       : disabled
    version       : 1.354
ubnt@ubnt:~$ show configuration | cat
interfaces {
    bridge br0 {
        aging 300
        bridged-conntrack disable
        hello-time 2
        max-age 20
        priority 32768
        promiscuous disable
        stp false
    }
    ethernet eth0 {
        address 192.168.1.1/24
        duplex auto
        mtu 2018
        speed auto
    }
    ethernet eth1 {
        duplex auto
        mtu 2018
        speed auto
    }
    ethernet eth2 {
        duplex auto
        mtu 2018
        speed auto
    }
    ethernet eth3 {
        address 192.168.0.1/30
        duplex auto
        mtu 2018
        speed auto
    }
    ethernet eth4 {
        bridge-group {
            bridge br0
        }
        duplex auto
        mtu 2018
        speed auto
    }
    ethernet eth5 {
        duplex auto
        mtu 2018
        speed auto
    }
    loopback lo {
    }
    switch switch0 {
        mtu 1500
    }
    tunnel tun0 {
        bridge-group {
            bridge br0
        }
        encapsulation gre-bridge
        local-ip 192.168.0.1
        multicast disable
        remote-ip 192.168.0.2
        ttl 255
    }
}
service {
    gui {
        http-port 80
        https-port 443
        older-ciphers enable
    }
    ssh {
        port 22
        protocol-version v2
    }
}
system {
    host-name ubnt
    login {
        user ubnt {
            authentication {
                encrypted-password ****************
            }
            level admin
        }
    }
    ntp {
        server 0.ubnt.pool.ntp.org {
        }
        server 1.ubnt.pool.ntp.org {
        }
        server 2.ubnt.pool.ntp.org {
        }
        server 3.ubnt.pool.ntp.org {
        }
    }
    offload {
        hwnat enable
        ipsec enable
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone UTC
}
ubnt@ubnt:~$ show ubnt offload
IPSec offload module: loaded

HWNAT offload module: loaded

Traffic Analysis    :
  export    : disabled
  dpi       : disabled
    version       : 1.354
ubnt@ubnt:~$ show configuration | cat
interfaces {
    bridge br0 {
        aging 300
        bridged-conntrack disable
        hello-time 2
        max-age 20
        priority 32768
        promiscuous disable
        stp false
    }
    ethernet eth0 {
        address 192.168.1.2/24
        duplex auto
        mtu 2018
        poe {
            output off
        }
        speed auto
    }
    ethernet eth1 {
        duplex auto
        mtu 2018
        speed auto
    }
    ethernet eth2 {
        duplex auto
        mtu 2018
        speed auto
    }
    ethernet eth3 {
        address 192.168.0.2/30
        duplex auto
        mtu 2018
        poe {
            output off
        }
        speed auto
    }
    ethernet eth4 {
        bridge-group {
            bridge br0
        }
        duplex auto
        mtu 2018
        speed auto
    }
    ethernet eth5 {
        duplex auto
        mtu 2018
        speed auto
    }
    loopback lo {
    }
    switch switch0 {
        mtu 1500
    }
    tunnel tun0 {
        bridge-group {
            bridge br0
        }
        encapsulation gre-bridge
        local-ip 192.168.0.2
        multicast disable
        remote-ip 192.168.0.1
        ttl 255
    }
}
service {
    gui {
        http-port 80
        https-port 443
        older-ciphers enable
    }
    ssh {
        port 22
        protocol-version v2
    }
}
system {
    host-name ubnt
    login {
        user ubnt {
            authentication {
                encrypted-password ****************
            }
            level admin
        }
    }
    ntp {
        server 0.ubnt.pool.ntp.org {
        }
        server 1.ubnt.pool.ntp.org {
        }
        server 2.ubnt.pool.ntp.org {
        }
        server 3.ubnt.pool.ntp.org {
        }
    }
    offload {
        hwnat enable
        ipsec enable
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone UTC
}

 

 

Veteran Member
Posts: 6,640
Registered: ‎03-24-2016
Kudos: 1725
Solutions: 763

Re: GRE Offload on ER-X

GRE offload is for GRE traffic through the router, not for GRE tunnels on router itself

New Member
Posts: 8
Registered: ‎12-15-2017

Re: GRE Offload on ER-X


@16again wrote:

GRE offload is for GRE traffic through the router, not for GRE tunnels on router itself


Not sure what you mean by this statement, and I believe its incorrect. Intermediate routers are agnostic to GRE, so offloading on them is pointless. GRE offloading can only make sense on tunnel endpoint routers, as they are the only ones that process the GRE header.

Veteran Member
Posts: 6,640
Registered: ‎03-24-2016
Kudos: 1725
Solutions: 763

Re: GRE Offload on ER-X

[ Edited ]

Intermediate routers still have to route packets belonging to the GRE protocol.

This routing of GRE traffic is what can be offloaded.  Just like routing TCP or UDP

Offloading GRE/TCP/UDP traffic has speed advantage, as it's done by hardware intead of CPU.

Note, other protocols , like ICMP , and ESP  (=IPSEC) , can't be offloaded while routing those packets

 

 

I understand the confusion, as offloading IPSEC is only active for tunnels terminated on the ER

New Member
Posts: 8
Registered: ‎12-15-2017

Re: GRE Offload on ER-X


@16again wrote:

Intermediate routers still have to route packets belonging to the GRE protocol.

This routing of GRE traffic is what can be offloaded.  Just like routing TCP or UDP

Offloading GRE/TCP/UDP traffic has speed advantage, as it's done by hardware intead of CPU.

Note, other protocols , like ICMP , and ESP  (=IPSEC) , can't be offloaded while routing those packets

 

 

I understand the confusion, as offloading IPSEC is only active for tunnels terminated on the ER


I'm sorry, not to be rude, but this is not correct. I would guess I am not going to get a meaningful response about the offloading of gre-bridge until the confustion is resolved.

 

The simple fact is that intermediate routers are agnostic to all data inside the L4PDU, which includes the GRE header. They do not care about it as they only need the information in the outer ip header to be able to forward the packet. For offloading to even be relevant they would need to strip the outer IP header and process the GRE header\packet. An intermediate router would be wasting CPU cycles if it was doing this because it doesn't need to. Therefore GRE offloading has no impact on intermediate routers. Why would you offload a task from the CPU that it doesn't do in the first place.  It is equivalent to FedEx opening packages to see their contents at each distribution center, then resealing them. They don't need to know what's inside, they just need to look at the shipping label.

 

In an effort to demonstrate this I built a lab using an intermediate router. See the attached network diagram. 

 

After running some basic bandwidth test I got the following resuls. 

 

simple routing(no offload) - ~900Mbps
GRE Tunnel(no offload) - ~500Mbps
GRE Tunnel(offload on R1) - ~500Mbps
GRE Tunnel(offload on all routers) - ~480Mbps
GRE Bridge (offload on all routers)  - ~100Mbps

GRE Bridge (no offload)  - ~100Mbps

 

The interesting thing here is that GRE offloading appears to have very little performance benefit, if it is working as intended. So I wasn't able to demonstrate how offloading works, but I was able to provide some more information about the actual issue.

 

 

gre-lab.jpg
Veteran Member
Posts: 6,640
Registered: ‎03-24-2016
Kudos: 1725
Solutions: 763

Re: GRE Offload on ER-X

Highlighted
New Member
Posts: 8
Registered: ‎12-15-2017

Re: GRE Offload on ER-X

If anything that post confirms my viewpoint that intermediate routers don't offload GRE.

 

My simple point is that intermediate routers don't get to the point where they need to offload GRE. They make a routing decision based upon outer IP header and move on. The fact that the data portion of the IP packet is GRE is irrelevant, because the router doesn't care.

 

How would offloading GRE benefit an intermediate router? It can't make a routing decision based up the information in the GRE packet. In my lab R1 doesnt know how to route to 10.0.2.0/24 or 10.0.3.0/24. It has to look at the outer IP header, which is done in the CPU and is basic IPv4 routing, but once it does that it doesn't need to do any more processing and the packet is sent out. 

 

Intermediate routers don't require GRE offloading because they don't process GRE information. That is the entire point of encapsuation\tunneling. 

 

 

 

 

 

 

 

 

 

Veteran Member
Posts: 6,640
Registered: ‎03-24-2016
Kudos: 1725
Solutions: 763

Re: GRE Offload on ER-X

How would offloading GRE benefit an intermediate router?

 

The CPU can do the routing (=slow), or the hardware (=fast). 

Both don't bother about actual content in GRE packet

New Member
Posts: 8
Registered: ‎12-15-2017

Re: GRE Offload on ER-X


@16again wrote:

How would offloading GRE benefit an intermediate router?

 

The CPU can do the routing (=slow), or the hardware (=fast). 

Both don't bother about actual content in GRE packet


I think maybe we are getting off topic. I understand how offloading in general works and what it’s benefit is.  

 

But your initial point though was that GRE offloading only applies to intermediate routers. My argument is the opposite and GRE offloading is only relevant at tunnel endpoint routers and thus my assessment, based upon my lab testing, that GRE offloading on the ER-X platform does not seem to be working properly in 1.10.1 is correct. 

 

If an intermediate router does not look at the GRE data within the outer IP packet then what are we offloading to hardware? Just the IPv4 routing?

 

 

Senior Member
Posts: 5,146
Registered: ‎01-04-2017
Kudos: 715
Solutions: 256

Re: GRE Offload on ER-X

[ Edited ]

Bridge traffic is not offloaded 

 

https://help.ubnt.com/hc/en-us/articles/115006567467-EdgeRouter-Hardware-Offloading-Explained

 

edit I see they updated the offload page to state that bridges are offloaded.

 

I am pretty sure that is incorrect and is meant to refer to the switch chip.  I think we need clarification on that and the page updated if it is wrong.

 

@UBNT-afomins @UBNT-benpin

Ubiquiti Employee
Posts: 1,820
Registered: ‎05-08-2017
Kudos: 351
Solutions: 289

Re: GRE Offload on ER-X

On the ER-X, ER-X-SFP and EP-R6 models, bridged traffic can in fact be offloaded when hwnat offloading is enabled. On all other EdgeRouter models, bridged traffic is not offloaded.

 

Ben


Ben Pin - EdgeMAX Support

Veteran Member
Posts: 6,640
Registered: ‎03-24-2016
Kudos: 1725
Solutions: 763

Re: GRE Offload on ER-X

@UBNT-benpin,

Are you sure?

afaik , on ER-X switching is hardware based, even without  offload enabled.

But bridging will imho is still software based, and thus cost lots of CPU 

Ubiquiti Employee
Posts: 1,820
Registered: ‎05-08-2017
Kudos: 351
Solutions: 289

Re: GRE Offload on ER-X

When you enable hwnat offloading on the ER-X, bridged interfaces are offloaded yes.

 

Ben


Ben Pin - EdgeMAX Support

SuperUser
Posts: 5,935
Registered: ‎09-03-2013
Kudos: 2046
Solutions: 396

Re: GRE Offload on ER-X


@UBNT-benpin wrote:

When you enable hwnat offloading on the ER-X, bridged interfaces are offloaded yes.

 

Ben


OK, I'd say that this is some really good, undocumented news, because I haven't seen this anywhere else in this forums. But really good news indeed. Cheers2

Redcon IT Solutions - Florianópolis/SC/Brazil -www.redcon.com.br
New Member
Posts: 8
Registered: ‎12-15-2017

Re: GRE Offload on ER-X

All of the offloading discussions taken into account, are the performance figures I am seeing to be expected or could there be an issue? 

Reply