Deleted Account
Posts: 0
Accepted Solution

offload

Hi

 

I often read about offloading and hardware offloading in this forum. Could anyone explain me what it is and what it exactly does. And why there are performance gains if you use it or not. 

I know how to configure, but I want understand more.

 

Greets moustache

 


Accepted Solutions
Member
Posts: 194
Registered: ‎12-11-2013
Kudos: 228
Solutions: 7

Re: offload

When we talk about offload we mean hardware acceleration.  Without offload, EdgeOS has similar performance to Linux on commodity hardware of a similar CPU.  

 

When offload is available, targeted functions can be implimented in a way that reduces latency and increases overall performance (often by serveral times).  You may also hear this mentioned as fast-path.

 

The EdgeRouter Lite, EdgeRouter, and EdgeRouter Pro are based on the Cavium Octeon network CPU.  This CPU provides additional instructions that allow it to perform complex opperations that are typically implimented in software (requiring several cycles of instruction processing to complete).

 

To put thins into perspective here are some test results using ipef.  Note that my test setup is the bottleneck here and not the EdgeRouter Lite.  I wasn't able to generate 64-byte packets fast enough to find the performance limit of the ERL.

 

EdgeRouter Lite with Offload:

 

Packet Len  Throughput
64          343 Mbps
128         645 Mbps
256         788 Mbps
512         886 Mbps
1024        939 Mbps
1500        956 Mbps

 

EdgeRouter Lite without Offload:

 

Packet Len  Throughput
64          62 Mbps
128         124 Mbps
256         255 Mbps
512         505 Mbps
1024        931 Mbps
1500        956 Mbps

 

On paper the ERL can do 1M pps for 64-byte packets, which would be 512 Mbps.  Without offload that drops to almost one tenth the perfromance.  It should be noted that the EdgeRouter and EdgeRouter Pro see almost double the performance for non-offload traffic over the ERL.

 

When it comes to using functionality on a router, you need to be aware of what functionality is offloaded (hardware accelerated) and what traffic is not for the platform being used.  This is true for Cisco and it is true for the EdgeRouter.

 

Based on what I've been able to find, I think the following information is mostly accurate but it would be great if UBNT would maintain a comprehensive KB article that covers this.

 

Things that are offloaded in the EdgeRouter:

 

  • IPv4 forwarding for Ethernet interfaces
  • IPv6 forwarding for Ethernet interfaces
  • 802.1Q VLAN tagging for Ethernet interfaces
  • TCP checksum (not special, offloaded by most server-grade network cards today)
  • IPv4 NAT
  • IPsec encode and decode
  • Linux Netfilter conntrack lookups (I think).
  • It's unclear the extent that Netfilter (firewall) is offloaded, if at all.  But if you use stateful packet inspection, matching by conntrack state seems to be on fast-path.
  • Also unclear if blackhole routing (also called null routing) is offloaded.

 

Things that are not offloaded:

 

  • Ethernet Bridging (using ports on the EdgeRouter as a switch)
  • Ethernet Bonding (also called port-channeling by Cisco or trunking by HP)
  • Tunnel encapsulation (GRE offload support is coming in 1.7 though)
  • Quality of Service (QoS)
  • Policy Routing (in 1.7 this is improved as the existance of a MODIFY rule will no longer disable offload)
  • Unicast Reverse Path Filtering (uRPF)
  • 5-tuple packet filtering (I think).
  • Netflow
  • Port Mirroring

 

In addition, any services that run on the EdgeRouter, like DHCP, DNS, or NTP are not offloaded and just use the normal CPU instruction set.

 

One service of note is the webfilter.  This is very CPU intensive and shouldn't be used for more than a handful of users if you have performance concerns (one of the reasons it's not promoted).

 

The long and short of it is that for simple configurations the EdgeRouter can have impressive performance (often leaving Cisco ISRs that cost over $10,000 in the dust).  But the lack of offload for some more common tasks drops performance down signiciantly so you need to be aware of those limitations.  That said, even without offload the EdgeRouter is more than fine for a 100 Mbps Internet connection.

 

The new EdgeRouter X is not based on the same hardware and currently does not have offload support.  It does feature a hardware switch, though, and can achive line-rate peformance when used that way so it might be better to think of it as an EdgeSwitch that has EdgeOS as the software.

 

Again.  I'm not 100% of what is offloaded and what isn't so if I made any mistakes please correct me. Smiley Happy

 

 

 

 

View solution in original post


All Replies
Member
Posts: 194
Registered: ‎12-11-2013
Kudos: 228
Solutions: 7

Re: offload

When we talk about offload we mean hardware acceleration.  Without offload, EdgeOS has similar performance to Linux on commodity hardware of a similar CPU.  

 

When offload is available, targeted functions can be implimented in a way that reduces latency and increases overall performance (often by serveral times).  You may also hear this mentioned as fast-path.

 

The EdgeRouter Lite, EdgeRouter, and EdgeRouter Pro are based on the Cavium Octeon network CPU.  This CPU provides additional instructions that allow it to perform complex opperations that are typically implimented in software (requiring several cycles of instruction processing to complete).

 

To put thins into perspective here are some test results using ipef.  Note that my test setup is the bottleneck here and not the EdgeRouter Lite.  I wasn't able to generate 64-byte packets fast enough to find the performance limit of the ERL.

 

EdgeRouter Lite with Offload:

 

Packet Len  Throughput
64          343 Mbps
128         645 Mbps
256         788 Mbps
512         886 Mbps
1024        939 Mbps
1500        956 Mbps

 

EdgeRouter Lite without Offload:

 

Packet Len  Throughput
64          62 Mbps
128         124 Mbps
256         255 Mbps
512         505 Mbps
1024        931 Mbps
1500        956 Mbps

 

On paper the ERL can do 1M pps for 64-byte packets, which would be 512 Mbps.  Without offload that drops to almost one tenth the perfromance.  It should be noted that the EdgeRouter and EdgeRouter Pro see almost double the performance for non-offload traffic over the ERL.

 

When it comes to using functionality on a router, you need to be aware of what functionality is offloaded (hardware accelerated) and what traffic is not for the platform being used.  This is true for Cisco and it is true for the EdgeRouter.

 

Based on what I've been able to find, I think the following information is mostly accurate but it would be great if UBNT would maintain a comprehensive KB article that covers this.

 

Things that are offloaded in the EdgeRouter:

 

  • IPv4 forwarding for Ethernet interfaces
  • IPv6 forwarding for Ethernet interfaces
  • 802.1Q VLAN tagging for Ethernet interfaces
  • TCP checksum (not special, offloaded by most server-grade network cards today)
  • IPv4 NAT
  • IPsec encode and decode
  • Linux Netfilter conntrack lookups (I think).
  • It's unclear the extent that Netfilter (firewall) is offloaded, if at all.  But if you use stateful packet inspection, matching by conntrack state seems to be on fast-path.
  • Also unclear if blackhole routing (also called null routing) is offloaded.

 

Things that are not offloaded:

 

  • Ethernet Bridging (using ports on the EdgeRouter as a switch)
  • Ethernet Bonding (also called port-channeling by Cisco or trunking by HP)
  • Tunnel encapsulation (GRE offload support is coming in 1.7 though)
  • Quality of Service (QoS)
  • Policy Routing (in 1.7 this is improved as the existance of a MODIFY rule will no longer disable offload)
  • Unicast Reverse Path Filtering (uRPF)
  • 5-tuple packet filtering (I think).
  • Netflow
  • Port Mirroring

 

In addition, any services that run on the EdgeRouter, like DHCP, DNS, or NTP are not offloaded and just use the normal CPU instruction set.

 

One service of note is the webfilter.  This is very CPU intensive and shouldn't be used for more than a handful of users if you have performance concerns (one of the reasons it's not promoted).

 

The long and short of it is that for simple configurations the EdgeRouter can have impressive performance (often leaving Cisco ISRs that cost over $10,000 in the dust).  But the lack of offload for some more common tasks drops performance down signiciantly so you need to be aware of those limitations.  That said, even without offload the EdgeRouter is more than fine for a 100 Mbps Internet connection.

 

The new EdgeRouter X is not based on the same hardware and currently does not have offload support.  It does feature a hardware switch, though, and can achive line-rate peformance when used that way so it might be better to think of it as an EdgeSwitch that has EdgeOS as the software.

 

Again.  I'm not 100% of what is offloaded and what isn't so if I made any mistakes please correct me. Smiley Happy

 

 

 

 

Deleted Account
Posts: 0

Re: offload

Many thanks for your excellent answer and your explanations. This is really useful (not only for me) and now it's more clear for me what offload means and what is should be. 

 

@UBNT-ancheng or @UBNT-stig please add rps superb explanations to your KB.