Please help, UAP Outdoor+ rebooting every 5 mins in busy wifi environment - "too much interference"

[andris]

We operate a network of ~50 Outdoor+ units running 3.7.9 firmware.  These generally work well for us.

However, I'm seeing some really weird behavior on the 5 Outdoor+ units that are in the noisiest/busiest wifi downtown urban environment

The access points reboot themselves very frequently.  The one that does this the most does it about every 5 minutes.

The last entry in /var/log/messages prior to the reboot is always:  "user.warn kernel: [  271.747000] Too much interference, Trying to change channels"

It seems that the AP is detecting a lot of noise - and trying to change channels, causing a reboot.

To avoid this, I tried to manually force to channel 1, 6, 11.  Even after forcing a channel - the ap still seems to execute the automatic channel change.  In other words, after setting to channel 11, I'll come back a few minutes later to find it has rebooted and changed to another channel.

1. How do I combat this problem?
2. Is there a way to turn off the "Too much interference, Trying to change channels" action since setting a manual channel doesn't seem to work

Any help greatly appreciated!

[UBNT-BenBuckley · Accepted solution]

@andris, here is a new binary for you.  This has the fixes that are being released with the next Alpha release, but are based on 3.7.9, which I recall is the version you're currently running.  If I messed that up, let me know:

http://dl.ubnt-ut.com/uap-odplus-busy-env3.bin

@Amaravati, it looks like you did a factory reset... not really sure how that would solve your issue.  Regardless, here is a binary for the UAP-OD as you requested (this image is also valid for  UAP, UAP-LR, UAP-OD5, PicostatonM2).  Please give it a try and let me know how it goes.

http://dl.ubnt-ut.com/uap-busy-env3.bin

[Dave-D]

Are you sure you have actually committed the setting?

First 'change' the settting, then 'apply'.   Dave

[andris]

I'm running controller version 4.8.14, which doesn't have change/apply.  (Is this something new that's been added?)

I have only an apply button, and that's what I clicked to apply the setting.

[Dave-D]

Fair enough.

When you return after a refresh to check the channel

settings, do they show the channel you selected?

Consider a controller update?   Dave

[andris]

Hi @Dave-D

I'm somewhat reluctant to upgrade the controller.  The ~45 other access points in my network are working great, and I don't need the new features of the new controller version.  I have some fairly heavy API integrations / custom portal stuff - and I'd have to do a full function test prior to upgrading to verify that all works properly.

Using the 3.7.9 firmware with my current controller version was recommended to me by a Ubiquiti employee to address the issues described here http://community.ubnt.com/t5/UniFi-Wireless/Connectivity-Problems-UAP-UAP-LR-UAP-Pro-UAP-Outdoor5-UA...

We had verified that the settings had in fact applied (unit went to "provisioning" then "connected" with the new static channel setting....but then after the next reboot with "too much interference" message, the channel had automatically changed to another.  This behavior seemed very similar to what was described/experienced here http://community.ubnt.com/t5/UniFi-Wireless/UAP-AC-PRO-Interference-Restart-Link-Down-Link-Up-Channe... by another user on some of the AC units.

Would love to hear some input from someone at Ubiquiti on this - @UBNT-BenBuckley maybe?

[deedee89]

Try swapping it out with some other unit from another area where it's not as busy, just to see if it acts the same. Might just be a defective unit.

[andris]

Thanks for the tip... One step ahead of you. :-) i tried a new unit and immediately got the same behavior.

Also, exact same behavior on 5 different units in different places.

I may have a solution. I have downgraded to 3.2.12 on one unit. The issues seem to have gone away. I have not seen a reboot for almost 6 hours.

I believe it was discovered that wifi driver issues were introduced in 3.3.x that were not resolved until 3.7.9. I am wondering if i have encountered a hanger on issue with this new driver?

[Dave-D]

Instead of downgrading, I suggest you upgrade. The 'extra features'

include bug and stability fixes you should have. Read the release

notes if that will help you decide.   Dave

[UBNT-BenBuckley]

@andris, sorry for not replying to this sooner.  I was working on it Friday and wanted to get you a debug firmware, but ended up not gettimg all the way through it. 

You are correct that the 10.2 driver causes a reboot in this situation, whereas the old driver did not.  I'm looking into a fix.  Are you willing to run debug firmware for troubleshooting since you are able reproduce the issue easily?

[andris]

Thanks for confirming @UBNT-BenBuckley.  Definitely willing to run debug firmware or provide other feedback to help understand the issue better and get it sorted.

If you have a moment - could you check out my questions in this post: http://community.ubnt.com/t5/UniFi-Wireless/Connectivity-Problems-UAP-UAP-LR-UAP-Pro-UAP-Outdoor5-UA...

This will help me choose the best course of action throughout my network while the debug fix is in process.

Thanks!

[UBNT-BenBuckley]

@andris

---

Can you confirm if the good wifi connectivity but no ip can pass" issue is unique to 3.3.x?  I think it wasn't present in 3.2.x?

---

3.3.X introduced a number of regressions that appear to have been corrected at this point.  That is to say, the latest releases of 3.7.X are as stable/more stable than 3.2.X was.  I assume you are referring to "connected with no internet", and at this point the remaining cases of that behavior appear to be environment/configuration related, so we are looking at them on an individual basis.

---

Can you help me understand if 3.7.9 is a vast improvement over 3.2.12 in other ways?  Or if 3.2.12 is more wifi stable than 3.7.9?

3.7.9 does improve on 3.2.12 in many ways... things have changed enough that I can't really list them all here.  But based on community feedback, 3.7.X is more stable than 3.2.12.

---

Do you think I should downgrade firmware network wide to 3.2.12 until any remaining wifi driver issues are sorted?  Or do you feel that 3.7.9 is a big enough advancement that i should only downgrade to 3.2.12 in locations where i observe the "Too much interference, Trying to change channels" and retain 3.7.9 in all other locations?

I strongly encourage to use 3.7.X rather than 3.2.12.  Reporting issues in 3.2.12 is useless to us for finding bugs.  If you're on 3.2.12, our first suggestion will be to upgrade to 3.7.X and see if it helps.  So basing all reports on 3.7.X is definitly the best course of action.

---

Do you have any idea of timeline when the "Too much interference, Trying to change channels" might be resolved?  I know you said you are still debugging...so no worries if not.

I'm working on a debug build right now.  I should have something for you today.

---

Is there any debug info i can provide that might assist in debugging further?

The info you provided was very helpful, led me straight to the issue.  Thanks for providing ip up front (looks like you paid attention to information requests in previous posts )  I'll let you know if I need more information, but at this point I have what I need.

[UBNT-BenBuckley]

@andris,

Try the debug firmware here: <bad url>

You can paste that url into the custom upgrade option in the controller.

That is based on 3.7.9, so it should look just like your firmware, except for the changes I've made. I tried to reproduce the environment here in the lab, but didn't have any success.   I know you will have better "luck"

Will you run this firmware, give it 10 or 15 minutes, and then SSH in and get the following:

1) dmesg | grep Too

    This is to see if you are getting the "Too much interference" message.

2) dmesg | grep bjb

    This is to isolate a couple of debug lines I put in.

Hopefully you will see mulitple "Too much interference" messages but experience zero actual reboots.  Looking forward to the results.

[andris]

Thanks @UBNT-BenBuckley will try this today

Just confirming...the firmware has "od5" in its name....our units are outdoor+, this firmware will work?

[UBNT-BenBuckley]

@andris thanks for paying attention.  That is a typo on my part.  I changed the name to avoid confusion.  New link is 

http://dl.ubnt-ut.com/uap-odplus-busy-env.bin

[andris]

Copy that @UBNT-BenBuckley -  we've deployed the debug firmware in 2 locations in our network.  Both of these locations that:

--are extremely busy wifi environments (downtown / urban)

--were rebooting very frequently with "too much interference" when running 3.7.9

--did not reboot when running 3.2.12

--(though i did find one of these units "hung" yesterday when running 3.2.12 after 23 hours of uptime in the state where it showed connected, was not rebooting, but users who connected to the wifi could not access any sites via IP.)

--are now still *not* rebooting frequently when running your debug firmware

This morning, I logged into one of the debug units.  I did not find any "bjb" lines or "Too" lines, though i did find many lines like this.  What do these mean?  On other access points, this has meant the unit is in a bad state.  I am going to reboot it:

Jan  1 17:40:08 CBD-UNION-TOWERS1 user.err libubnt[1871]: wevent.ubnt_check_wireless_index(): get_iface_index fail
Jan  1 17:40:08 CBD-UNION-TOWERS1 user.err libubnt[1871]: libubnt_util.get_iface_index(): unable to open sock
Jan  1 17:40:08 CBD-UNION-TOWERS1 user.err libubnt[1871]: wevent.ubnt_check_wireless_index(): get_iface_index fail
Jan  1 17:40:08 CBD-UNION-TOWERS1 user.err libubnt[1871]: libubnt_util.get_iface_index(): unable to open sock
Jan  1 17:40:08 CBD-UNION-TOWERS1 user.err libubnt[1871]: wevent.ubnt_check_wireless_index(): get_iface_index fail
Jan  1 17:40:08 CBD-UNION-TOWERS1 user.err libubnt[1871]: libubnt_util.get_iface_index(): unable to open sock
Jan  1 17:40:08 CBD-UNION-TOWERS1 user.err libubnt[1871]: wevent.ubnt_check_wireless_index(): get_iface_index fail
Jan  1 17:40:10 CBD-UNION-TOWERS1 user.err libubnt[1871]: libubnt_util.get_iface_index(): unable to open sock
Jan  1 17:40:10 CBD-UNION-TOWERS1 user.err libubnt[1871]: wevent.ubnt_check_wireless_index(): get_iface_index fail
Jan  1 17:40:10 CBD-UNION-TOWERS1 user.err libubnt[1871]: libubnt_util.get_iface_index(): unable to open sock
Jan  1 17:40:10 CBD-UNION-TOWERS1 user.err libubnt[1871]: wevent.ubnt_check_wireless_index(): get_iface_index fail

[andris]

@UBNT-BenBuckley

Investigated the second unit in busy wifi environment running debug firmware.  This one did produce "too much interference" messages, but did not produce any bjb lines here are the details:

[77859.110000] Too much interference, Trying to change channels
[77889.110000] Too much interference, Trying to change channels
[77919.111000] Too much interference, Trying to change channels
[77949.115000] Too much interference, Trying to change channels
[77979.116000] Too much interference, Trying to change channels
[78009.117000] Too much interference, Trying to change channels
[78309.120000] Too much interference, Trying to change channels
[78339.123000] Too much interference, Trying to change channels
[78369.123000] Too much interference, Trying to change channels
[78399.126000] Too much interference, Trying to change channels
[78429.126000] Too much interference, Trying to change channels
[78459.127000] Too much interference, Trying to change channels
[78489.127000] Too much interference, Trying to change channels
[78519.127000] Too much interference, Trying to change channels
[78549.127000] Too much interference, Trying to change channels
[78579.130000] Too much interference, Trying to change channels
[78609.130000] Too much interference, Trying to change channels
[78639.130000] Too much interference, Trying to change channels
[78669.131000] Too much interference, Trying to change channels
[78759.134000] Too much interference, Trying to change channels
[78819.137000] Too much interference, Trying to change channels
[78849.138000] Too much interference, Trying to change channels
[78879.144000] Too much interference, Trying to change channels
[78909.144000] Too much interference, Trying to change channels
[78939.145000] Too much interference, Trying to change channels
[78969.146000] Too much interference, Trying to change channels
[78999.146000] Too much interference, Trying to change channels
[79029.146000] Too much interference, Trying to change channels
[79059.146000] Too much interference, Trying to change channels
[79089.146000] Too much interference, Trying to change channels
[79119.147000] Too much interference, Trying to change channels
[79149.149000] Too much interference, Trying to change channels
[79179.149000] Too much interference, Trying to change channels
[79209.149000] Too much interference, Trying to change channels
[79239.150000] Too much interference, Trying to change channels
[79269.152000] Too much interference, Trying to change channels
[79299.157000] Too much interference, Trying to change channels
[79329.158000] Too much interference, Trying to change channels
[79359.158000] Too much interference, Trying to change channels
[79389.158000] Too much interference, Trying to change channels
[79419.159000] Too much interference, Trying to change channels
[79449.159000] Too much interference, Trying to change channels
[79479.162000] Too much interference, Trying to change channels
[79509.162000] Too much interference, Trying to change channels
[79539.162000] Too much interference, Trying to change channels
[79569.163000] Too much interference, Trying to change channels
[79599.163000] Too much interference, Trying to change channels
[79629.163000] Too much interference, Trying to change channels
[79659.166000] Too much interference, Trying to change channels
[79689.166000] Too much interference, Trying to change channels
[79719.166000] Too much interference, Trying to change channels
[79749.166000] Too much interference, Trying to change channels
[79779.166000] Too much interference, Trying to change channels
[79809.167000] Too much interference, Trying to change channels
[79839.168000] Too much interference, Trying to change channels
[79869.168000] Too much interference, Trying to change channels
[79899.168000] Too much interference, Trying to change channels
[79929.169000] Too much interference, Trying to change channels
[79959.173000] Too much interference, Trying to change channels
[79989.173000] Too much interference, Trying to change channels
[80019.174000] Too much interference, Trying to change channels
[80049.174000] Too much interference, Trying to change channels
[80079.174000] Too much interference, Trying to change channels
[80109.176000] Too much interference, Trying to change channels
[80139.177000] Too much interference, Trying to change channels
[80169.177000] Too much interference, Trying to change channels
[80199.179000] Too much interference, Trying to change channels
[80229.183000] Too much interference, Trying to change channels
[80259.187000] Too much interference, Trying to change channels
[80289.187000] Too much interference, Trying to change channels
[80319.187000] Too much interference, Trying to change channels
[80349.188000] Too much interference, Trying to change channels
[80379.188000] Too much interference, Trying to change channels
[80409.188000] Too much interference, Trying to change channels
[80439.190000] Too much interference, Trying to change channels
[80469.191000] Too much interference, Trying to change channels
[80499.191000] Too much interference, Trying to change channels
[80529.191000] Too much interference, Trying to change channels
[80559.191000] Too much interference, Trying to change channels
[80589.192000] Too much interference, Trying to change channels
[80619.193000] Too much interference, Trying to change channels
[80649.194000] Too much interference, Trying to change channels
[80679.196000] Too much interference, Trying to change channels
[80709.196000] Too much interference, Trying to change channels
[80739.196000] Too much interference, Trying to change channels
[80769.197000] Too much interference, Trying to change channels
[80799.199000] Too much interference, Trying to change channels

also noted that there are lots of "failed to open sock" messages intersperced:

Jan  1 21:49:18 CBD-WELBEX-IMENTI user.err libubnt[2444]: wevent.ubnt_check_wireless_index(): get_iface_index fail
Jan  1 21:49:18 CBD-WELBEX-IMENTI user.err libubnt[2444]: libubnt_util.get_iface_index(): unable to open sock
Jan  1 21:49:18 CBD-WELBEX-IMENTI user.err libubnt[2444]: wevent.ubnt_check_wireless_index(): get_iface_index fail
Jan  2 00:49:18 CBD-WELBEX-IMENTI user.warn kernel: [78558.182000] vap: 44:d9:e7:c7:ac:14
Jan  2 00:49:18 CBD-WELBEX-IMENTI user.warn kernel: [78558.183000] valid pRc->validRateIndex :
Jan  2 00:49:18 CBD-WELBEX-IMENTI user.warn kernel: [78558.183000] 13 14 15 16 17 18 19
Jan  2 00:49:39 CBD-WELBEX-IMENTI user.warn kernel: [78579.130000] Too much interference, Trying to change channels
Jan  1 21:49:58 CBD-WELBEX-IMENTI user.err libubnt[2444]: libubnt_util.get_iface_index(): unable to open sock
Jan  1 21:49:58 CBD-WELBEX-IMENTI user.err libubnt[2444]: wevent.ubnt_check_wireless_index(): get_iface_index fail
Jan  2 00:49:58 CBD-WELBEX-IMENTI user.warn kernel: [78598.966000] vap: 44:d9:e7:c7:ac:14
Jan  2 00:49:58 CBD-WELBEX-IMENTI user.warn kernel: [78598.966000] valid pRc->validRateIndex :
Jan  2 00:49:58 CBD-WELBEX-IMENTI user.warn kernel: [78598.966000] 13 14 15 16 17 18 19
Jan  1 21:49:58 CBD-WELBEX-IMENTI user.err libubnt[2444]: libubnt_util.get_iface_index(): unable to open sock
Jan  1 21:49:58 CBD-WELBEX-IMENTI user.err libubnt[2444]: wevent.ubnt_check_wireless_index(): get_iface_index fail
Jan  1 21:49:58 CBD-WELBEX-IMENTI user.err libubnt[2444]: libubnt_util.get_iface_index(): unable to open sock
Jan  1 21:49:58 CBD-WELBEX-IMENTI user.err libubnt[2444]: wevent.ubnt_check_wireless_index(): get_iface_index fail
Jan  2 00:50:09 CBD-WELBEX-IMENTI user.warn kernel: [78609.130000] Too much interference, Trying to change channels
Jan  2 00:50:39 CBD-WELBEX-IMENTI user.warn kernel: [78639.130000] Too much interference, Trying to change channels

[UBNT-BenBuckley]

@andris, I'll look into the socket failure messages you're seeing.  In the meantime, with the debug firmware, does the network work well?  Are users able to pass traffic, even though it's so congested?

[andris]

@UBNT-BenBuckley

Good question - sorry for not being more clear.

Of the two units running debug:

1. I suspect one with the "unable to open sock" messages was probably not passing IP traffic at the time.  This message has been indicative of that in other situations.  Unfortunately I didn't have anyone onsite at the time to verify.  I had to reboot as a precaution.
2. The second unit, the one that had all the "Too much interference" messages got extensive testing today.  Users were able to pass traffic.  So i'd say this one is performing well.

I'd say i am cautiously optimistic, though the "unable to open sock" messages have me a little worried.  I deployed the debug firmware on a few more units today, and will continue monitoring.

[UBNT-BenBuckley]

@andris also, can you ssh into the AP and show me the output of this command:

dmesg | grep bjb

There should be a few debug lines in there.

[andris]

Hi @UBNT-BenBuckley

I now have 3 access points running the debug firmware.  Of them, only one got "Too much interference" messages today, and that one had no lines containing "bjb".  

Here is the output of these commands on the unifi that did observe "Too much interference" lines.

BZ.bjb-debug-379-odp_ben@toad.ubnt-ut.com.-1# dmesg | grep bjb
BZ.bjb-debug-379-odp_ben@toad.ubnt-ut.com.-1# dmesg | grep Too
[45319.234000] Too much interference, Trying to change channels
[45349.234000] Too much interference, Trying to change channels
[45379.236000] Too much interference, Trying to change channels
[45409.237000] Too much interference, Trying to change channels
[45439.239000] Too much interference, Trying to change channels
[45469.242000] Too much interference, Trying to change channels
(many more of these)

(the other two units had no output of either command:

dmesg | grep bjb

dmesg | grep Too