Radio Shootout Pt.2 - let's try a whole bunch of them...

by ‎04-18-2015 01:55 PM - edited ‎04-23-2015 02:06 PM

Since the results of these tests (see Pt. 1 ) were providing some surprising results against the one Ubiquiti competitor's radio, it seemed like a good idea to see how some other types of radios fared too. And to make it more interesting, how about testing the radios for interference rejection on nearby channels?

ShootoutRadios.jpg

So we put together a group of radios that many WISPs use today or are looking at using – the Mimosa B5C (yes, I'm giving up the thinly veiled “Brand M” thing...), the Cambium EPMP 1000-GPS, the Mikrotik RB922, and from Ubiquiti the AF5X, Rocket AC Lite, and the Rocket AC-PTP (with the Prism Chip).

 

Just as a point of reference, here are the Prices of the various radios as of today:

InterferenceTestRadioMSRPs.png

Here's a diagram of the test setup:

InterferenceTestSetupDiag.png

For those who may not be familiar with more advanced microwave components, some elaboration is in order. To do these tests an interfering signal needed to be introduced at a specifically defined power level without either completely disrupting the link between the radios under test, or harming them by coupling the interfering signal directly to the receiver and blowing something up. This was done with a combination of fixed attenuators and a pair of Directional Couplers. These are devices that are designed to couple or extract a reduced signal level from a RF signal path, but in only one direction and at a specific coupling ratio. The directionality is veery useful when you want to look at things like reflected signals from terminating components (like antennas or amplifiers) and make it easy to add signals to an RF path without disrupting things. They work by arranging specific stripline paths or waveguides with coupling ports into a passive device – for more detailed info see http://www.microwaves101.com/encyclopedias/directional-couplers

The coupling port will either extract a portion of the signal going through the Coupler, or can add signals to the straight through path, at some fixed attenuation level, typically 20-30 dB down. Internally, these are 4 port devices, with the two coupling ports being directionally isolated – the coupling port picks up or inserts the signal in one direction only. For the three-port versions (like used in these tests) the 4th port is just terminated with a 50 ohm load internally. The math behind this is complex, but it works very well.

 

For this test, a pair of directional couplers were set up with the direct through ports connected to the radios being tested, and the interference source connected to the coupling port. This way, a specific level of interference can be injected into the test path.

 

To generate the interference, a standard RocketM5 was used. Atheros (the chipset manufacturer) has a test program which can be used to make the radio do all kinds of things it normally wouldn't do, and in this case it was used to make it transmit continuously with full modulation to simulate very strong interference from a nearby radio. Unless you are doing testing like this, don't try this yourself, and definitely don't do it on a live radio in the field, unless your goal is to interfere with everyone else.

 

The tests

 

For this series of tests, each radio pair was set up on a 20MHz channel at a nominal received signal level, and the interfering signal coupled in at differing levels from the same level as the data channel to 30dB more than the data carrier. Each test series was run with the interfering signal offset from the test radio's carrier frequency by different distances: 10MHz edge-to-edge (30MHz Center-to-Center), 30MHz EtoE (50MHz CtoC), 50MHz EtoE (70MHz CtoC) and 70MHz EtoE (90MHz CtoC) were all tested. The throughput using Iperf in tcp mode was plotted vs. The interfering ratio which is how much stronger in dB the interfering signal is than the received signal. So if the received signal is -60 dBm and the interferer is at -30 at the tested unit's receiver, emulating a very nearby transmitter, and the plotted level would be 30dBc (the worst the test tried, since all the radios were basically unusable by then.)

InterferenceTestSetupChans.png

Each radio pair was tested at the 4 different interferer frequency offset values across a power range of 0dB (interferer same level as the desired signal) to 30dB (30dB more interferer than desired). All 6 radios were plotted together for each frequency offset value, and are shown below.

Throughput_bidir_cfg9.png

 

Throughput_bidir_cfg8.png

Throughput_bidir_cfg7.png

Throughput_bidir_cfg6.png

Results:

The clear winner here is the AF5X, followed by the RocketAC PtP and the Rocket AC Lite. The AF5X's superior filtering ability really shines, with it's ability to ignore interference at all but the closest offset being identical. The AC PtP is actually better at 30MHz offset or more due to the active filtering in the Prism Chip, but it starts to degrade significantly with 10MHz spacing from the main carrier edge.

The MiktoTik is next best with heavier interference levels, and is actually slightly ahead of the AC Lite and the AC PtP at the closest spacing and highest interference levels, although that's of dubious usefulness, as the channel throughput is down under 20Mb at that point.

The Cambium is a surprise here, both for it's relative suceptibility to interferrence at all levels and rather poor performance overall. To look a little deeper as to why, opening it up revealed that it uses the same chipset as the UBNT Rocket5M, so that's probably why. So much for Cambium...

The Mimosa is not too far behind the AC Lite speed wise at lower interference levels, but quickly drops behind the others as the interference level increases. So it's overall performance in heavy interference puts it at the back of the pack, unless you use it's dual channel function to approximately double the throughput (while eating up 2x the bandwidth...) But even doubling the throughput under heavy interference won't get you up to the level of the other radios except the EPMP, which is pretty bad.

 

So what does this tell us? Well, the best performance overall in both throughput and interference handling is the AF5X, followed by the Rocket5 AC PtP and AC Lite. Even at moderately high interference levels, they all outperform the competition handily, and under heavy interference where the other radios have basically given up, the AF5X and PTP are still able to function remarkably well

 

The two biggest surprises here were the B5c and the EPMP, and neither in a good way. For an $839 radio to perform so much worse than a $135 AC Lite was a shock.   Part of this is possibly due to their having to make everything work over such a wide bandwidth all the time to support the dual channel operation, but it's still not good.   And for the EPMP to so drastically underperform across the board, well I suppose if you're a Cambium shop...

 

So in the end, what does this prove?   Well, IMHO it means the Ubiquiti gear is much better engineered, for one thing.   And it's real world performance (when things are kept legal - more on that in a minute) is going to be better than the competition.   I'm not going to say any more than that our decision to keep to Ubiquiti gear keeps being proven to us to be the correct decision over and over again.

 

Now about those folks who have been reporting about how great the B5c radios are working for them - in at least one case I know of that's because they are being run at illegal power levels.   One link in particular I'm aware of is operating at 56dB EIRP on DFS (30dBm EIRP max legal) channels to get 2x80 to work.   Now there are probably people out there doing this with other kinds of radios too, but if that's what it takes to get those "amazing" numbers, then that's not a real solution.   So beware of claims about how well Brand X is working over some huge distance without knowing exactly what the parameters are.

Jim

 

 

EDIT - 

Several people asked for more results for on-channel and adjacent channel interference after I posted this. Initially I wasn't going to do this, but it made sense to include this information as well as all the previous data. So here is the comparison of the AF5X and the B5.

 

To look at a worst case scenario of what happens when the interfering signal is directly adjacent to the operating channels with zero guard space between, in this test, the received signal levels of each test receiver were carefully measured (not from the radio's GUI) to insure they both had a received level of -52dBm. That way any differences or inaccuracies in the radios' conducted power, EIRP calculations or RSSI values would be eliminated.

 

The channel offset (center frequency of the operating channel to the interferer) was varied from 0 (directly on top of each other) to 2 times the channel bandwidth, which leaves a guard band between them of between zero and 1x the bandwidth – 20MHz in this case as that was the channel width used for this test. The interfering signal level was increased until the throughput of the radio channel began to be affected, (in this case a 20% reduction) and was plotted at that point for both the AF5X and the B5. The chart below shows the dB level the interferer needs to be higher than the desired signal at an offset of 1x – no spacing between the two signals at all – literally edge-to-edge – for both above and below the channel.

AF5B5IntChart.png

It's clear that the AF5X can handle quintuple (7dB more or 5x) the interference level of the B5 in both cases. This is very significant in a heavily congested RF environment like we're all finding more and more today.

 

Here is the complete plot of the test results – the x-axis is the channel offset in channel width from right on top of the channel in the center to 2x the channel width above and below the center frequency. The y-axis is the interferer level in dB above the received signal (-52dBm) needed to begin to degrade the throughput of the link.

AF5B5IntPlot.png

So to summarize, the AF5X has far superior (5 times) the zero-guard band interference rejection of the B5 at exactly the same received signal strength. That makes a huge difference in operations at congested sites, and allows for much easier frequency planning as well.

" How can anyone trust Scientists? If new evidence comes along, they change their minds! " Politician's joke (sort of...)
"Humans are allergic to change..They love to say, ‘We’ve always done it this way.’ I try to fight that. "Admiral Grace Hopper, USN, Computer Scientist
":It's not Rocket Science! - Oh wait, Actually it is... "NASA bumper sticker
":The biggest problem in tech I see right now is that most users don't want to do things that are hard. That doesn't bode well for the industry or the society.": (me. actually ;-)
Comments
by
on ‎04-18-2015 02:12 PM

WOW Jim this is just blowing my Mind !!! Kudos for that ! Very well done !

by
on ‎04-18-2015 02:40 PM

Cracking read. Very enlightening.

by
on ‎04-18-2015 02:56 PM

An N chip (ePMP) should be expected to perform worse than an AC chip in throughput tests.

by
on ‎04-18-2015 03:23 PM

You do realize the ePMP is the only N radio on the list, right? It should not be such a surprise that an N radio can't match the throughput of an AC radio.

by
on ‎04-18-2015 03:28 PM
N or not, it degrades much faster than anything but the B5C.
by
on ‎04-18-2015 03:55 PM

And it "costs" more than a "custom radio" (AF5X) or both "AC Radios" from UBNT Biggrin

 

Leaving

by
‎04-18-2015 04:43 PM - edited ‎04-18-2015 04:45 PM

Jim, this is an awesome writeup. But your results don't hold up in the real world when you throw distance into the equation. AFx is a clear winner. Airmax AC falls on its face when distance comes into the equation.

 

However, adjacent channel info is spot on.

by
on ‎04-18-2015 05:32 PM

I am surprised to see how well the lite held up to some noise!

by
on ‎04-18-2015 05:46 PM

Thanks for doing this writeup Jim! Great insight!

by
on ‎04-18-2015 10:13 PM

Very valuable data, few people have a chance compare all of them!

by
on ‎04-18-2015 10:23 PM

Hi,  Im considered a non experted user and with very basics about RF.  But Im a user of cambium and ubnt gear.

 

My question is why to use/choose the more expensive radio from ubnt , i guess the sincronization feature do not provides any advantage on ths scenario , and prices is almos the 3 times againts the non GPS radio. the epmp  radio is not more like a rocket m5 radio ? 

 

Whata are the results from a Rocket M5 radios results on this test ?

Thanks in advanced.

 

 

 

 

 

by
on ‎04-18-2015 11:07 PM

Could you please list what firmware you use on each radio, along with some of the basic configurations on each link... e.g. depending on what ePMP firmware you're running there are 3 possible PtP modes, each with different performance and latency characteristics.

by
‎04-19-2015 02:00 AM - edited ‎04-19-2015 02:06 AM

 eejimm in these  tests simply measured  Adjacent Channel Rejection  of various radios  . ACR is important parameter, but it does not give any estimation of interference resistance  of ePMP , ubnt  and Mikrotik radio. Really  these tests showed following :

- ACR  of Atheros  802.11n  and ubnt/mikrotik  АС  radio  at 2.5  adjacent channel - 30 MHz guard interval (GI)  in 20MHz channel bandwidth , roughtly  are  about  15-17 dB. It is not good, but is not so  bad.

- Mimosa  radio ACR  is much worse – less than  12 dB at 30 MHz guard interval. It is  not  good.

- ACR of ubnt AF5X   it seems due to Airprizm filter is about 30 dB at  (GI  >30 MHz). But we see that at GI= 20 MHz  and less  ACR is the same as  other systems  without filter. So airprizm does not works at 1st  and probably at 2nd  adjacent channels, so  it makes this type of filter at  low GI  unefficient.

In regards to measure of thoughput.

 ePMP aggregate throughput  60-70 mbps without interference ( 70 mHz GI) is not correct. It should have been 110-120Mbps. Something   wrong   in configuration, may be   DL/UL ratio  is 50/50 and  simplex ( one way) speed  is measured .   Also   ePMP works at  64QAM  and  comparison    with  other system at   256QAM  is not correct.  

So these tests  are   usual  throughput  comparison  at  64QAM 5/6 and 256QAM modulation and ACR measurement  various systems and nothing more.

 eejimm should provide tests  with   interference  at  the same frequency channels  (co-channel interference)   with   C/I = 10, 20,30 dB. These results will be more interesting and usefull

by
on ‎04-19-2015 02:07 AM

epmp is a suprise here , but ok gsp model is not needed in this secenario so price is more like rocketM5 or aclite, and still in wide use ... meMan Happy. I woud like to se RocketM5 in coparations. All radios are atheros wifi based except arifiber .. I think !?

 

 

by
on ‎04-19-2015 05:37 AM

@Makoto Mimosa is not Atheros based, it is Quantenna based.

 

I would like to see the PMP450 and PTP650 series run through the same tests. yes, I know they're expensive and no, I don't have any to donate to the test. Maybe someone else can?

by
on ‎04-19-2015 07:59 AM

Everyone - I understand some of the frustration of only seeing limited testing of a limited number of radios here, but I do have a business to run, and a lot of other things take up the majority of my time (which is why I'm writing this on Sunday morning - only so many hours in the week...)   So there's no way I can do more tests on more radios right now - this just happened to be somethig that was started for some other reasons, and was fairly easy to do at the time.

 

Please feel free to do whatever additional testing/comparisons you feel approrpiate, and post your results too - more data is always better than less.  I'm sure I'll do more in the future - this really started by wanting to compare AF5X with B5c, and sort expanded from there - we're going to be doing some testing on long links in the field soon, and I'll post those results when I get them.

Jim

by
on ‎04-19-2015 08:02 AM

@eejimm Agreed that more data is better than less and that time is precious. Not everyone has your skills in creating whatever devices it is you have in the middle. Hopefully others can piece these things together and do some further testing.

by
on ‎04-19-2015 02:00 PM

Awesome tests.

 

I'd be curious to see what the results are of on-channel interference as well (albeit at much lower levels I'm sure). Would  be a great reference to point people to when they think that airPrism is going to solve their on-channel interference issues. Man Very Happy - I also have a suspicion that the AF5x would cope better than the airMAX AC units with on-channel interference, but it would be good to actually have some numbers rather than a gut feeling on the matter haha.

by
‎04-19-2015 02:28 PM - edited ‎04-19-2015 07:48 PM

How close do you think the Cambium proximates a standard Rocket M5 or Airmax N radio since they use the same chipsets? Do you think the Cambium results would be about the same for the "N" Airmax line? Like if you replaced the Cambium with another rocket m5 radio?

by
on ‎04-20-2015 06:00 AM

I like seeing test comparing multiple vendors equipment. It really helps the end user decide on which equipment to use because we all know you can't trust marketing. And not to take anything away from Jim, like he said, he has a business to run and this was done on his spare time, but from my experience these results don't relate well to real world performance.  At least for the Rocket AC Lite and the epmp 1000.  We've tested the mikrotik netmetal, epmp 1000, rocket AC Lite, and now most recently the AF5X on real world links ranging from 5 miles to just shy of 20 miles and while the AF5x blows the rest of them out of the water, the next best performing radio is the epmp 1000, followed by the mikrotik, and then the AC Lite(mind you we didn't test the AC PTP or Brand "M").  

Our 20 mile link had very little noise and on a 20mhz channel the rocket ac lite would only push 70mbps tcp, only slightly better than the rocket M it replaced.  The epmp 1000 pushed 110mbps tcp.   On a 10 mile link with heavy noise, the AC lite would barely pass traffic, maybe 10 mbps,  whereas the mikrotik netmetal was able to push 60mbps tcp.   Maybe it's a software issue or something to do with ACK calculation but the AC Lites left us disappointed, on the other hand we were very impressed with the epmp considering it has an 802.11N chipset and it was outperforming AC radios.

 

In the end, it really doesn't matter though, I can't imagine using any 5ghz radio for a backhaul other than the AF5x or AF5 at this point.

by
on ‎04-20-2015 10:28 AM

@popcorrin there's ack issues in the current AC stuff, others have complained about this as well

by
on ‎04-20-2015 10:30 AM

@esseph ACK!

by
on ‎04-20-2015 10:32 AM

@mhammett "*silence*" - Current UBNT AC Firmware

by Ubiquiti Employee
on ‎04-21-2015 10:15 AM

@Alver  just to make it clear related to the AirFiber line.  We do not use AirPrism in the design of the product.  AirFiber radios are purpose-built from the ground up with a custom design.  Just wanted to make that clear.

by
on ‎04-21-2015 05:38 PM

@UBNT-Alex just on that note, AirPrism on airMAX lines are to improve the filtering on the otherwise off-the-shelf WiFi chipsets used. Does the AirFiber line have equivalent fancy filtering as standard as a part of the purpose-built chips so adding AirPrism in front would change nothing? Or would there still be some advantage to having an AirPrism equivalent sitting in front of the custom AF radio?

by
on ‎04-21-2015 06:31 PM

NVX, In my testing, the AF5X has an even better adjacent channel filter than AirPrism radios do.   Jim's testing here backs my field observations on this.

by Ubiquiti Employee
on ‎04-21-2015 06:40 PM

NVX,

 

I wish I could figure out how to find my old posts on the forum because I addressed this in detail some months ago.  The answer is no.  

 

The AF radio (in this particular case) uses a Zero IF (ZIF) architecture that has a built-in configurable very high order digital filter for selectivity.  It results in a filter passband that is not realizable through analog means (like airPrism) so there would be no point to it.  The filter inside the AFX platform can adapt to different operating conditions (like interference) and has far more equivalent poles than airPrism.  As has been pointed out, airPrism is a way to improve the relatively modest selectivity associated with standard WiFi chips as used by others. The addition of external analog filters enhances the performance and provides a significant performance benefit.  I hope this makes some sense?

 

 

by
‎04-21-2015 07:17 PM - edited ‎04-21-2015 07:20 PM

Makes sense, although I thought the Atheros WiFi chipsets were Zero IF as well, so that wouldn't be the differing factor in this case, but the built in configurable high order digital filter would be (which, from my understanding is what airPrism is anyway, just in a separate chip whereas from the sounds of it AirFiber has it integrated in with the tuner itself?)

 

In any case you guys should really harass your marketing guys so that it's clearer that the AF gear has superior selectivity than airPrism devices, cause the amount airPrism is praised in the airMAX AC datasheets then seeing its absence in the AF datasheets the uneducated could think it's a poorer choice for noisy environments!

 

Edit: Guessing this is the other post you're referring to? http://community.ubnt.com/t5/airFiber/Aifiber-little-spectrum-left/m-p/1181371#M18098

by
on ‎04-22-2015 09:00 AM

Awesome very very good that you did these tests!

by
on ‎04-27-2015 12:16 PM

Great writeup, *but* as has been aluded to, the real world negates some of these results.  Over long distances, the RocketM5 has shown to be superior to the RocketAC-Lite for me as the M5 maintains usable throughput to lower signal levels.  The flip side, the AF5X is many times better at long distance plus noise than the RocketM5 or AC-lite.

 

A brief example is that I replaced an AC-lite link w/ an AF5X recently.  The AC-lite link actually performed worse than the RocketM5 link it recently replaced (about a month previous) only pulling in around 20Mbps aggregate across 13 miles.  The Rocket M5 was around 30Mbps aggregate.  The AF5X is 100Mbps aggregate on the same channel as the RocketM5 (5180) same 40Mhz.  The AC-lite was up in UNII3 for lack of access to 5180Mhz.

 

From my experience the AF5X is even more superior than the tests suggest, and that the RocketAC line's superiority falls off at around 6 miles and the RocketM5 takes over #2 spot.  I've helped install a B5 and a B5c and the owner of those is happy with them, but now that I'm running AF5X I wouldn't consider the B5*.

 

Jim, I think what your tests lack is variables.  Though an excellent test, your decouplers and attenuators are essentially static.  What the real world is throwing in the mix is the variability from random partial-duty cycle links for which my observations tell me the RocketM5 is beating the non-airprism AC-lite, and that the cambium epmp link accells at.  Still a fantastic test, but it's definitely 'lab conditions'

by
on ‎04-29-2015 02:09 AM

Very interesting test indeed. IMHO, the main point missing in this test is the duty cycle of the interferring signal. You are using an emtpy Rocket M5 AP to generate the interference, thus yielding in a very low duty cycle.

 

When serious interference test is to be done, you need to control not only the interference level in dBc, but also the time that this interference lasts for. For instance, a AirFiber like device would be more useful.

 

I say this, because when you get to real field, the interferring APs are usually not quiet, just the contrary, they are busy, thus transmitting a great part of the time.

 

Regards!

 

    Iratxo. 

by
on ‎04-29-2015 03:21 AM
He has a special parameter set in the RM5 to cause it to transmit all the time.
by
on ‎04-29-2015 06:47 AM

I've testing AF5X on a REALLY heavy interfered enviroment and my conclusion are a bit far from the conclusions shown here: in my tests, AF5X doesn't work properly on this kind of situations. However, I'm yet on early testing period and many things can change.

 

@eejimm excellent job!

 

by
on ‎04-29-2015 06:56 AM

      I've testing AF5X on a REALLY heavy interfered enviroment and my conclusion are a bit far from the conclusions shown here: in my tests, AF5X doesn't work properly on this kind of situations. However, I'm yet on early testing period and many things can change.

 

You mean like in this case -

http://community.ubnt.com/t5/airFiber-Stories/AF5X-on-21-mile-link-with-very-heavy-interference-stil...

At a site where there are 16 Cambium APs all in the 5GHz band operating at the same time, and the AF5X still gets 280Mb actual throughput at 21 miles?

Jim

by
on ‎04-29-2015 07:11 AM
Oh snap! Haha
by
on ‎04-29-2015 07:21 AM

@eejimm 16 Cambium radios that only probably used two channels due to sync?

by
on ‎04-29-2015 07:33 AM
 

16 Cambium radios that only probably used two channels due to sync?

 

Oh, I wish - they are using every single frequency possible in UNII-2 and UNII-3 - there's literally nothing open except UNII-1.   Fortunately we only backhaul through the site, otherwise we never would have even looked at it...

Jim

by
on ‎07-30-2015 12:36 PM

Brilliant write and experiment...welldone Man Happy