Reply
Member
Posts: 230
Registered: ‎11-01-2015
Kudos: 97
Solutions: 5

Qualcomm Fast Path (SFE) for the EdgeRouters

[ Edited ]

From the README:

Shortcut is an in-Linux-kernel IP packet forwarding engine.  It's designed
to offer very high speed IP packet forwarding based on IP connection tracking.
It's dramatically faster than the standard netfilter-based NAT forwarding path
but is designed to synchronise state back to netfilter/conntrack so that it
doesn't need to deal with all of the complexities of special cases.

My main interest in this is that SFE actually works with QoS enabled, while the builtin offloads do not.

 

Benchmarks

When comparing only the Cavium offloads and SFE against each other there's not much off a difference. What's interesting is how CAKE without SFE vs CAKE with SFE ends up. I haven't benchmarked only CAKE here, but previous tests have shown performance to be under 100 Mbit/sec.

 

Some iperf benchmarks with CAKE and SFE. The bandwidth parameter is set to 1Gbit, NAT support and dual-dsthost - dual-srchost are enabled on their respective sides.

 

SFE (fast-classifier) w/CAKE on only egress:

[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   105 MBytes   879 Mbits/sec    3    372 KBytes       
[  4]   1.00-2.00   sec   105 MBytes   882 Mbits/sec  239    496 KBytes       
[  4]   2.00-3.00   sec   107 MBytes   897 Mbits/sec    0    634 KBytes       
[  4]   3.00-4.00   sec   105 MBytes   881 Mbits/sec    0    752 KBytes       
[  4]   4.00-5.00   sec   101 MBytes   849 Mbits/sec    0    850 KBytes       
[  4]   5.00-6.00   sec   101 MBytes   849 Mbits/sec   32    936 KBytes       
[  4]   6.00-7.00   sec   101 MBytes   849 Mbits/sec   76   1008 KBytes       
[  4]   7.00-8.00   sec   100 MBytes   839 Mbits/sec    4    769 KBytes       
[  4]   8.00-9.00   sec  96.2 MBytes   807 Mbits/sec   39    841 KBytes       
[  4]   9.00-10.00  sec   100 MBytes   839 Mbits/sec  1309    444 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1022 MBytes   857 Mbits/sec  1702             sender
[  4]   0.00-10.00  sec  1019 MBytes   855 Mbits/sec                  receiver

SFE (fast-classifier) w/CAKE on egress and ingress:

[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  63.8 MBytes   535 Mbits/sec   33    229 KBytes       
[  4]   1.00-2.00   sec  58.9 MBytes   494 Mbits/sec   47    206 KBytes       
[  4]   2.00-3.00   sec  61.9 MBytes   519 Mbits/sec   35    233 KBytes       
[  4]   3.00-4.00   sec  61.6 MBytes   517 Mbits/sec   45    233 KBytes       
[  4]   4.00-5.00   sec  64.1 MBytes   538 Mbits/sec   41    272 KBytes       
[  4]   5.00-6.00   sec  61.3 MBytes   514 Mbits/sec   49    272 KBytes       
[  4]   6.00-7.00   sec  62.5 MBytes   524 Mbits/sec   36    301 KBytes       
[  4]   7.00-8.00   sec  62.3 MBytes   523 Mbits/sec   51    318 KBytes       
[  4]   8.00-9.00   sec  65.2 MBytes   547 Mbits/sec   42    318 KBytes       
[  4]   9.00-10.00  sec  63.5 MBytes   533 Mbits/sec   39    346 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   625 MBytes   524 Mbits/sec  418             sender
[  4]   0.00-10.00  sec   624 MBytes   523 Mbits/sec                  receiver

This port is based on dissent1's PR against the LEDE repo and adapted for the 3.10 kernel. The source is available on GitHub.

 

WARNING:

here_be_dragons_by_lord_psymon
To use these binaries you'll have to replace the kernel, which should not be taken lightly.
You should also disable the builtin offloads before using this, as they're probably gonna step on each others foot.

 

Installation (ER-Lite/ER-PoE):

# Download for the ERL and ER-PoE
curl -L -O https://dl.lochnair.net/UBNT/E100-SFE/sfe-kernel.tar.bz2 curl -L -O https://dl.lochnair.net/UBNT/E100-SFE/sfe-modules.tar.bz2 # Mount the partition the kernel resides on mkdir /mnt/boot mount /dev/sda1 /mnt/boot # Extract the archives tar -xf sfe-kernel.tar.bz2 -C /mnt/boot/ tar -xf sfe-modules.tar.bz2 -C / umount /dev/sda1
# Update module dependencies
depmod -a
# Reboot reboot # Load SFE modprobe fast-classifier

 Installation (ER-8/ER-Pro):

# Download for the ER-8 and ER-Pro
curl -L -O https://dl.lochnair.net/UBNT/E200-SFE/sfe-kernel.tar.bz2
curl -L -O https://dl.lochnair.net/UBNT/E200-SFE/sfe-modules.tar.bz2

# Mount the partition the kernel resides on
# You can double-check which partition it is with /usr/sbin/ubnt-hal getKernPart mkdir /mnt/boot mount /dev/mmcblk0p1 /mnt/boot # Extract the archives tar -xf sfe-kernel.tar.bz2 -C /mnt/boot/ tar -xf sfe-modules.tar.bz2 -C / umount /dev/mmcblk0p1 # Update module dependencies depmod -a # Reboot reboot # Load SFE modprobe fast-classifier  

 

SuperUser
Posts: 20,370
Registered: ‎09-17-2013
Kudos: 5107
Solutions: 1455

Re: SFE on the ERL

replacing the kernel will nuke all of the cavium-specific stuff that UBNT has put into their firmware ... probably not a good idea in the slightest for any type of "production" setups.

Member
Posts: 230
Registered: ‎11-01-2015
Kudos: 97
Solutions: 5

Re: SFE on the ERL

@dpurgertVery true, this is meant for brave souls that like to live dangerously. However I do have the proprietary modules loaded after replacing the kernel. The changes aren't all that big.

 

But yes do NOT run this in production. Just in case that wasn't clear. This'll eat your dog.

Emerging Member
Posts: 102
Registered: ‎07-25-2016
Kudos: 131

Re: SFE on the ERL

@Lochnair I just noticed your posts with some remarkable results for SFE. It's been almost a year since I ran some RRUL tests on my ER-X with Cake and fq_codel. At a rate limited bandwidth of around 70-80 Mbps or so and above, latencies of the UDP/ICMP flows and TCP RTTs would start to rise. Higher bandwidths could be reached (not sure how high), but SQM would be ineffective at those rates and thus not worthwhile. So I guess we could call 70-80 Mbps the "maximum effective bandwidth" for Cake/fq_codel on the ER-X (without SFE).

 

Unfortunately both ot the ER-Xs I have are now in production and I can't risk replacing the kernel. But what I wonder is, with SFE, what is the maximum effective bandwidth for Cake/fq_codel now? Is it higher than before? I suspect that it may be somewhere between the measured iperf bandwidths and the maximum effective bandwidth without SFE, but it would need some testing. If anyone can try Flent's rrul and/or rrul_be tests at various rate limited bandwidths to find out this info, that would be great to know. Otherwise, one of my ER-X's might become available for testing in the winter when the camp it serves shuts down.

 

Thanks for the great work!

Member
Posts: 230
Registered: ‎11-01-2015
Kudos: 97
Solutions: 5

Re: SFE on the ERL

[ Edited ]

@heistp

Those iperf tests doesn't really show a realistic picture of what performance you can actually expect. If you run Flent with that configuration things fall apart rather rapidly, and ingress performance really suffers. I've been tweaking the bandwidth parameter to find how high you can go before CAKE falls apart. Of course, if you've got an asymmetric link you can get higher download speed compared to a symmetric one.

 

Without SFE 50 Mbit is the highest I can set the bandwidth parameter to, anything higher and download suffers.

rrul_be-2017-08-27T124731.544336.CAKE_wo_SFE

With SFE I've gotten as high as 170 Mbit on the bandwidth setting without things falling apart. So that's at least a 300% increase.

rrul_be-2017-08-27T123344.407780.CAKE_w_SFE

Of course with its beefier CPU the ER-X should be able to get even better results.

Sadly I haven't been able to compile the whole kernel for the ER-X yet. There's probably some patches from MediaTek for Binutils I need, but can't find.

 

Flent result files:

With SFE

Without SFE

Emerging Member
Posts: 65
Registered: ‎04-15-2017
Kudos: 14

Re: SFE on the ERL

Everything works fine until the modprobe commands then it says the fast classifier module isn't found any ideas?

 

I tried it on 1.9.7 hotifx 2 if that matters.

 

Thanks

Member
Posts: 230
Registered: ‎11-01-2015
Kudos: 97
Solutions: 5

Re: SFE on the ERL

@JoeyJoe

I forgot to add it to the instructions, so in case you haven't done it you need to run:

## Update module dependencies
depmod -a

before loading the module.

Emerging Member
Posts: 65
Registered: ‎04-15-2017
Kudos: 14

Re: SFE on the ERL

Thanks

Emerging Member
Posts: 65
Registered: ‎04-15-2017
Kudos: 14

Re: SFE on the ERL

[ Edited ]

So now I get, when doing 'sudo modprobe fast-classifier'

 

ERROR: could not insert 'fast_classifier': Unknown symbol in module, or unknown parameter (see dmesg)

 

Any ideas?

Emerging Member
Posts: 102
Registered: ‎07-25-2016
Kudos: 131

Re: SFE on the ERL

[ Edited ]

@Lochnair Those are outstanding results. SFE could make these devices usable with SQM for very high speed broadband. I also like how the latency over time with SFE is flat- that's a great sign that things are working well. Even if you're not running at such high speeds, it looks like latency control will be more consistent with SFE than without it.

 

I've just ordered another ER-X as a "backup". I hope to get some time to test this on the ER-X as well and post some results...

Member
Posts: 230
Registered: ‎11-01-2015
Kudos: 97
Solutions: 5

Re: SFE on the ERL

@JoeyJoe

I need the output of the dmesg command to know more.

Member
Posts: 230
Registered: ‎11-01-2015
Kudos: 97
Solutions: 5

Re: SFE on the ERL

@heistp

FTTH has become pretty common here in Norway, and many typically has 100/100 or 150/150, so this makes the ERL actually usable for SQM purposes with these links. It'd be interesting to test with low enough speeds to compare CAKE with/without SFE to see how much latency difference there are.

 

If you didn't see it in the other post, I haven't been able to compile for the ER-X just yet. Binutils complains about some instruction in the kernel, so I need to figure out which patches I need for it to work.

Emerging Member
Posts: 65
Registered: ‎04-15-2017
Kudos: 14

Re: SFE on the ERL

[ Edited ]

@LochnairHere is dmesg (Sorry I gave you the wrong stuff at first).  If you need more let me know.

 

fast_classifier: Unknown symbol br_dev_update_stats (err 0)
fast_classifier: Unknown symbol br_dev_update_stats (err 0)

 

Thanks for your help.

 

 

 

Member
Posts: 230
Registered: ‎11-01-2015
Kudos: 97
Solutions: 5

Re: SFE on the ERL

@JoeyJoe

That function is part of the patched kernel, so it looks like you didn't put it in the right place.

Emerging Member
Posts: 65
Registered: ‎04-15-2017
Kudos: 14

Re: SFE on the ERL

[ Edited ]

I followed the instructions that were posted, is there any other way to load the kernal.  I.E. manually unpack it?  I'm kinda at a loss as to what to do as Linux is not my strong suit.

 

Thanks

 

Edit so when I do umount /dev/sda1 the kernal disappears from /mnt/boot. 

 

Not sure where I'm going wrong but I'm going to start again.

 

Emerging Member
Posts: 65
Registered: ‎04-15-2017
Kudos: 14

Re: SFE on the ERL

@LochnairI reloaded stock firmware 1.9.7 hotifx 2 and did a factory reset and still get

 

fast_classifier: Unknown symbol br_dev_update_stats (err 0)

 

I did follow the instructions as posted except adding sudo in front of the commands that required me to do so.

 

I'm lost at this point so I'm not sure why the new Kernel isn't being loaded to the right place.

Member
Posts: 230
Registered: ‎11-01-2015
Kudos: 97
Solutions: 5

Re: SFE on the ERL

@JoeyJoe

That's weird. I've verified that the kernel image I uploaded indeed is the correct one, so you shouldn't be having any issues.

 

It's easy to check if you have the right image however. After mounting /dev/sda1 run:

md5sum /mnt/boot/vmlinux.64

and compare the output against this:

d7c8b7ea66a27d0ac4b9809241bdf8f7
Emerging Member
Posts: 65
Registered: ‎04-15-2017
Kudos: 14

Re: SFE on the ERL

[ Edited ]

@LochnairSo my MD5 checks out d7c8b7ea66a27d0ac4b9809241bdf8f7 .  It shows a kernel date of Aug 25 when I logon with putty.

 

What firmware version are you using to try this out on?

Member
Posts: 230
Registered: ‎11-01-2015
Kudos: 97
Solutions: 5

Re: SFE on the ERL

@JoeyJoe

I'm on 1.9.7.

 

Just a shot in the dark, what's the output of 'show system image'?

Emerging Member
Posts: 65
Registered: ‎04-15-2017
Kudos: 14

Re: SFE on the ERL

[ Edited ]

@LochnairThis is the output: v1.9.7+hotfix.2.5010182.170818.0413-1 (running image) (default boot)

 

Edit: I just check dmesg and it shows that it's loading your kernel on boot.

 

If you can think if anything else let me know.

Also I have all offloads disable but it still show the offload module is loaded, could that be a problem?

 

Thanks for all your help

Reply