Scheduled maintenance: Community will be offline Monday June 17th, 1:00 AM - 6:00 AM (PT)
Reply
Emerging Member
Posts: 44
Registered: ‎01-10-2017
Kudos: 3

Re: TONS of errors, and frequent service drops Mesh network

Thursday afternoon or friday , but link me the debug ver and I'll try it on my problem child AP.  

 

Member
Posts: 188
Registered: ‎05-31-2018
Kudos: 25
Solutions: 4

Re: TONS of errors, and frequent service drops Mesh network

Hi @UBNT-BenBuckley 

 

I had another AP reset yetsterday. A quick 

grep 'received packet on' /var/log/messages

 shows that not all loops detected result in a AP reset, in case you are interetsed.

 

Spoiler

Sun May 12 13:00:39 2019 kern.warn kernel: [4448544.987545] br0: received packet on ath3 with own address as source address

Wed May 15 21:24:01 2019 kern.warn kernel: [4737944.954826] br0: received packet on ath3 with own address as source address

Thu May 16 11:28:20 2019 kern.warn kernel: [4788603.591753] br0: received packet on ath3 with own address as source address

 

I will be back on location this weekend for a couple of weeks if you'd like me to try any debug firmware. I'll be updating my devices in any case.

Ubiquiti Employee
Posts: 1,471
Registered: ‎12-16-2015
Kudos: 310
Solutions: 10

Re: TONS of errors, and frequent service drops Mesh network

@Pian what do you mean when you say you had an AP reset?  Does that mean reboot?  Or disconnect/reconnect?

 

Can you try 4.0.39?  It has some specific fixes to mesh that have improved the reliability at my site, and has been confirmed at some other sites.  I'd like to know if it helps your's as well.

If you'd like to try our latest features, consider signing up for beta: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Sign-Up-for-Beta-Access

If you're trying to debug a network issue, take a look at this help article: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
Member
Posts: 188
Registered: ‎05-31-2018
Kudos: 25
Solutions: 4

Re: TONS of errors, and frequent service drops Mesh network

@UBNT-BenBuckley - I mean a disconnect/reconnect.

 

OK, I'll update this weekend (all APs + switch, plus controller + firmware, and USG) + see how things go next week. What could possibly go wrong? Man Happy 

Ubiquiti Employee
Posts: 1,471
Registered: ‎12-16-2015
Kudos: 310
Solutions: 10

Re: TONS of errors, and frequent service drops Mesh network

@Pian that loop detection log entry will cause the wireless uplink to disconnect.  But if it comes back online fast enough, the controller won't register the disconnect event (because it's based on some sort of timeout).  

 

I appreciate your contiuned optimism Man Happy .  It should absolutely be getting more stable though, so if it doesn't help your site then it means there is still something else to be addressed.

If you'd like to try our latest features, consider signing up for beta: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Sign-Up-for-Beta-Access

If you're trying to debug a network issue, take a look at this help article: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
New Member
Posts: 22
Registered: ‎10-06-2017
Kudos: 1

Re: TONS of errors, and frequent service drops Mesh network

@UBNT-BenBuckley is the new debug firmware stable?

Ubiquiti Employee
Posts: 1,471
Registered: ‎12-16-2015
Kudos: 310
Solutions: 10

Re: TONS of errors, and frequent service drops Mesh network

@marshall1367 I've seen the latest FW increase stabilty in at least 2 different sites (one of them is my house) and I haven't seen any new regressions.  If you can try it, maybe on an AP or two, and provide some feedback, that would be useful.

If you'd like to try our latest features, consider signing up for beta: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Sign-Up-for-Beta-Access

If you're trying to debug a network issue, take a look at this help article: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
New Member
Posts: 22
Registered: ‎10-06-2017
Kudos: 1

Re: TONS of errors, and frequent service drops Mesh network

@UBNT-BenBuckley yes please send it to me

Member
Posts: 188
Registered: ‎05-31-2018
Kudos: 25
Solutions: 4

Re: TONS of errors, and frequent service drops Mesh network

Hi @UBNT-BenBuckley 

 

I just saw that I had an AP disconnect/reconnect yesterday (running 4.0.42)

 

Screenshot 2019-05-22 at 20.38.17.png

 

Sadly, my /var/log/messages starts at 23.25 on May 21, so I missed it. Unless you know if rotated logs are kept? Feel free to log in.)

Member
Posts: 188
Registered: ‎05-31-2018
Kudos: 25
Solutions: 4

Re: TONS of errors, and frequent service drops Mesh network

Hi @UBNT-BenBuckley 

 

I had another event on a different AP this morning - 

 

Screenshot 2019-05-23 at 17.31.38.png

That was only ~5 hours ago, so the /var/log/messages has not rolled off, and I can confirm that no "received packet on" strings are in there.

 

Again, if you see this in good time please feel free to dial in and have a look.

New Member
Posts: 22
Registered: ‎10-06-2017
Kudos: 1

Re: TONS of errors, and frequent service drops Mesh network

@UBNT-BenBuckley can you send me the most recent debug firmware?

Ubiquiti Employee
Posts: 1,471
Registered: ‎12-16-2015
Kudos: 310
Solutions: 10

Re: TONS of errors, and frequent service drops Mesh network

@Pian and @marshall1367  what is the latest version you've tested?  The changes that would be in the debug firmware are now actually part of the 4.0.42 release FW, which is released to stable.  Can you upgrade to that and see how it goes?

If you'd like to try our latest features, consider signing up for beta: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Sign-Up-for-Beta-Access

If you're trying to debug a network issue, take a look at this help article: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
Member
Posts: 188
Registered: ‎05-31-2018
Kudos: 25
Solutions: 4

Re: TONS of errors, and frequent service drops Mesh network

@UBNT-BenBuckley - as I mentioned, I'm on 4.0.42 (not the SPF53 version for my XG)

Ubiquiti Employee
Posts: 1,471
Registered: ‎12-16-2015
Kudos: 310
Solutions: 10

Re: TONS of errors, and frequent service drops Mesh network

@Pian sorry I missed it.  What about the Base Station?  Is it also on 4.0.42?

If you'd like to try our latest features, consider signing up for beta: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Sign-Up-for-Beta-Access

If you're trying to debug a network issue, take a look at this help article: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
Member
Posts: 188
Registered: ‎05-31-2018
Kudos: 25
Solutions: 4

Re: TONS of errors, and frequent service drops Mesh network

@UBNT-BenBuckley - see attatched. I updated my USG yesterday

 

 

Screenshot 2019-05-24 at 21.50.03.png
Member
Posts: 188
Registered: ‎05-31-2018
Kudos: 25
Solutions: 4

Re: TONS of errors, and frequent service drops Mesh network

[ Edited ]

@UBNT-BenBuckley wrote:

 But if it comes back online fast enough, the controller won't register the disconnect event (because it's based on some sort of timeout).  

 


Hi @UBNT-BenBuckley 

 

I have been trying to diagnose a fault with some IoT devices repeatedly going offline. I am starting to wonder if my problem is some of these "micro-disconnects" ie a short mesh failure which is not reported by the controller. (I am not seeing any `received packet on` loops, so any micro-disconnect seems not be caused by the loopback you are investigating.) 

 

For example, I had one device reporting itself as having gone offline and then reconnecting at both 5.37am and 5.39am this morning (two of many such events last night). Looking at `/var/log/messages` on the AP that device usually connects to - just a few feet away - I find the following for the period straddling those two times (with MAC's redacted):

 

Spoiler

Tue May 28 05:31:41 2019 user.info : wevent[1520]: wevent.ubnt_handle_custom_alert_sta_assoc(): EVT_AP_STA_ASSOC_TRACKER_DBG: event_id: 10 vap: ath1 sta_mac: b8:xxxxxxxxxxxxx auth_ts: 0.0 aut

h_delta: -1 assoc_delta: -1 wpa_auth_delta: -1 radius_auth_delta: -1 radius_auth_status: N/A ip_delta: -1 ip_assign_type: N/A disassoc_count: 0 auth_failures: 0 assoc_failures: 0 wpa_auth_fail

ures: 0 ip_failures: 0 assoc_status: 0 arp_status: yes dns_status: N/A event_type: fixup

Tue May 28 05:37:53 2019 daemon.info hostapd: ath1: STA 60:xxxxxxxxxxxxxIEEE 802.11: sta_stats

Tue May 28 05:37:53 2019 daemon.info hostapd: ath1: STA 60:xxxxxxxxxxxxx IEEE 802.11: disassociated

Tue May 28 05:37:53 2019 user.info : wevent[1520]: wevent.ubnt_custom_event(): EVENT_STA_LEAVE ath1: 60:0xxxxxxxxxxxxxx

Tue May 28 05:37:53 2019 daemon.info hostapd: ath1: STA 7a:xxxxxxxxxxxxxDRIVER: Sead AUTH addr=60:01:94:84:8c:f5 status_code=0

Tue May 28 05:37:53 2019 daemon.info hostapd: ath1: STA 60:xxxxxxxxxxxx IEEE 802.11: associated

Tue May 28 05:37:53 2019 daemon.info hostapd: ath1: STA 60xxxxxxxxxxxxx WPA: pairwise key handshake completed (RSN)

Tue May 28 05:37:53 2019 user.info : wevent[1520]: wevent.ubnt_custom_event(): EVENT_STA_JOIN ath1: 60:xxxxxxxxxxxx / 2

Tue May 28 05:37:53 2019 user.info : wevent[1520]: wevent.ubnt_custom_event(): EVENT_STA_IP ath1: 60:xxxxxxxxxxxxx / 192.168.99.117

Tue May 28 05:38:03 2019 user.info : wevent[1520]: wevent.ubnt_handle_custom_alert_sta_assoc(): EVT_AP_STA_ASSOC_TRACKER_DBG: event_id: 216 vap: ath1 sta_mac: 60:01:94:84:8c:f5 auth_ts: 123447

.440399 auth_delta: 0 assoc_delta: 10000 wpa_auth_delta: 20000 radius_auth_delta: -1 radius_auth_status: N/A ip_delta: 60000 ip_assign_type: dhcp disassoc_count: 0 auth_failures: 0 assoc_failu

res: 0 wpa_auth_failures: 0 ip_failures: 0 assoc_status: 0 arp_status: yes dns_status: yes event_type: success

Tue May 28 05:38:03 2019 daemon.info hostapd: ath1: STA 60:xxxxxxxxxxxxx RADIUS: starting accounting session 600443A994B5273E

Tue May 28 05:39:03 2019 daemon.info hostapd: ath1: STA 60xxxxxxxxxxxxx IEEE 802.11: sta_stats

Tue May 28 05:39:03 2019 daemon.info hostapd: ath1: STA 60:xxxxxxxxxxxxxx IEEE 802.11: disassociated

Tue May 28 05:39:03 2019 daemon.info hostapd: ath1: STA 7a:xxxxxxxxxxxxx DRIVER: Sead AUTH addr=60:01:94:84:8c:f5 status_code=0

Tue May 28 05:39:03 2019 user.info : wevent[1520]: wevent.ubnt_custom_event(): EVENT_STA_LEAVE ath1: 60:xxxxxxxxxxxxx / 2

Tue May 28 05:39:03 2019 daemon.info hostapd: ath1: STA 60xxxxxxxxxxxxx IEEE 802.11: associated

Tue May 28 05:39:03 2019 daemon.info hostapd: ath1: STA 60:xxxxxxxxxxxxx WPA: pairwise key handshake completed (RSN)

Tue May 28 05:39:03 2019 user.info : wevent[1520]: wevent.ubnt_custom_event(): EVENT_STA_JOIN ath1: 60:xxxxxxxxxxxxx / 2

Tue May 28 05:39:03 2019 user.info : wevent[1520]: wevent.ubnt_custom_event(): EVENT_STA_IP ath1: 60:xxxxxxxxxxxxx / 192.168.99.117

Tue May 28 05:39:13 2019 user.info : wevent[1520]: wevent.ubnt_handle_custom_alert_sta_assoc(): EVT_AP_STA_ASSOC_TRACKER_DBG: event_id: 217 vap: ath1 sta_mac: 60:xxxxxxxxxxxxx auth_ts: 123517

.638702 auth_delta: 0 assoc_delta: 0 wpa_auth_delta: 20000 radius_auth_delta: -1 radius_auth_status: N/A ip_delta: 50000 ip_assign_type: dhcp disassoc_count: 0 auth_failures: 0 assoc_failures:

 0 wpa_auth_failures: 0 ip_failures: 0 assoc_status: 0 arp_status: yes dns_status: N/A event_type: success

Tue May 28 05:39:13 2019 daemon.info hostapd: ath1: STA 60:xxxxxxxxxxxxx RADIUS: starting accounting session F771A90A4xxxxxxx

Tue May 28 05:56:26 2019 daemon.info hostapd: ath1: STA 60xxxxxxxxxxxxx5 IEEE 802.11: sta_stats

 My question is - what in the logs definitively shows that an AP has fallen off the mesh and no longer has network connection? eg is it a log entry like `Tue May 28 05:39:03 2019 daemon.info hostapd: ath1: STA 60:xxxxxxxxxxxxxx IEEE 802.11: disassociated` (I hope not, as there are a lot of these in the log!)

Ubiquiti Employee
Posts: 1,471
Registered: ‎12-16-2015
Kudos: 310
Solutions: 10

Re: TONS of errors, and frequent service drops Mesh network

@Pian The log messages you showed are from a normal client connecting and disconnecting from the AP.  

 

In general, however, similar log entries can show you when a wireless AP disconnects.

 

If you're looking at the logs on the Mesh AP, you can grep -nr 'EVENT_STA_LEAVE athX`where athX can be found with iwconfig 2>&1 | grep vwire.  It should be ath0

 

If you're looking at logs on the Base Station, you can grep -nr 'EVENT_STA_LEAVE vwireand this will reveal when one of it's Mesh AP's has disconnected from it.

 

If you'd like to try our latest features, consider signing up for beta: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Sign-Up-for-Beta-Access

If you're trying to debug a network issue, take a look at this help article: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
Member
Posts: 188
Registered: ‎05-31-2018
Kudos: 25
Solutions: 4

Re: TONS of errors, and frequent service drops Mesh network

[ Edited ]

Thanks for that info, @UBNT-BenBuckley - using this does confirm that I am suffering  from these micro-disconnects, though perhaps in a slightly unusual way.

 

I looked at my logs for both my Unifi and my home automation system. I found that my HA system had an anomaly at around 5.05am (some 2 1/2 hours ago at time of writing.):

Screenshot 2019-05-31 at 07.34.57.png 

 

I look at the events reported in my controller log and I find that a mesh AP did indeed show a disconnect/reconnect at both 5:04am and 5:08am, eg:

 

Screenshot 2019-05-31 at 07.37.13.png

 

By way of reminder, please recall my wireless AP topology is as follows:

 

Screenshot 2019-05-31 at 07.40.32.png

 

So, looking at the AP 3-Lite-Ballroom, I find 

 

Spoiler

  ___ ___      .__________.__

 |   |   |____ |__\_  ____/__|

 |   |   /    \|  ||  __) |  |   (c) 2010-2019

 |   |  |   |  \  ||  \   |  |   Ubiquiti Networks, Inc.

 |______|___|  /__||__/   |__|

            |_/                  https://www.ui.com/

 

      Welcome to UniFi UAP-AC-Lite!

 

3-Lite-Ballroom-BZ.v4.0.42# iwconfig 2>&1 | grep vwire

ath2      IEEE 802.11ac  ESSID:"vwire-6e1f2cec0e6840c8"  

3-Lite-Ballroom-BZ.v4.0.42# cat /var/log/messages | grep  -nr 'EVENT_STA_LEAVE ath2'

91:Fri May 31 04:56:48 2019 user.info : wevent[1463]: wevent.ubnt_custom_event(): EVENT_STA_LEAVE ath2: 8a:8a:20:d2:85:5b / 2

211:Fri May 31 04:59:21 2019 user.info : wevent[1463]: wevent.ubnt_custom_event(): EVENT_STA_LEAVE ath2: 8a:8a:20:d2:85:5b / 2

342:Fri May 31 05:01:54 2019 user.info : wevent[1463]: wevent.ubnt_custom_event(): EVENT_STA_LEAVE ath2: 8a:8a:20:d2:85:5b / 2

467:Fri May 31 05:04:26 2019 user.info : wevent[1463]: wevent.ubnt_custom_event(): EVENT_STA_LEAVE ath2: 8a:8a:20:d2:85:5b / 2

593:Fri May 31 05:06:59 2019 user.info : wevent[1463]: wevent.ubnt_custom_event(): EVENT_STA_LEAVE ath2: 8a:8a:20:d2:85:5b / 2

713:Fri May 31 05:09:31 2019 user.info : wevent[1463]: wevent.ubnt_custom_event(): EVENT_STA_LEAVE ath2: 8a:8a:20:d2:85:5b / 2

874:Fri May 31 05:12:04 2019 user.info : wevent[1463]: wevent.ubnt_custom_event(): EVENT_STA_LEAVE ath2: 8a:8a:20:d2:85:5b / 2

3-Lite-Ballroom-BZ.v4.0.42# 

Looking at 2-Pro-Study, I find, unfortunately, that the logs have rotated and /var/log/messages starts  05:08:28 and 

cat /var/log/messages | grep -nr 'EVENT_STA_LEAVE vwire' produces no output

New Member
Posts: 22
Registered: ‎10-06-2017
Kudos: 1

Re: TONS of errors, and frequent service drops Mesh network

@UBNT-BenBuckley my system is still experiencing intermittant drops.  Is there another version of firmware that can be tried or do you need more information?

Ubiquiti Employee
Posts: 1,471
Registered: ‎12-16-2015
Kudos: 310
Solutions: 10

Re: TONS of errors, and frequent service drops Mesh network

@marshall1367 @Pian thanks for testing, I'll discuss internally and get back to you.

If you'd like to try our latest features, consider signing up for beta: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Sign-Up-for-Beta-Access

If you're trying to debug a network issue, take a look at this help article: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
Reply