Scheduled maintenance: Community will be offline Monday June 17th, 1:00 AM - 6:00 AM (PT)
Reply
Ubiquiti Employee
Posts: 1,471
Registered: ‎12-16-2015
Kudos: 310
Solutions: 10

Re: TONS of errors, and frequent service drops Mesh network

@marshall1367 let me know what you find.  If you're interested in the debug firmware too, just let me know.

 

If you'd like to try our latest features, consider signing up for beta: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Sign-Up-for-Beta-Access

If you're trying to debug a network issue, take a look at this help article: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
Emerging Member
Posts: 44
Registered: ‎01-10-2017
Kudos: 3

Re: TONS of errors, and frequent service drops Mesh network

Have it installed and will proceed as instructed.  I imagine it wont take long.  Will that diag file stay a while if I'm unable to ssh in right away ?

Ubiquiti Employee
Posts: 1,471
Registered: ‎12-16-2015
Kudos: 310
Solutions: 10

Re: TONS of errors, and frequent service drops Mesh network

yes the diagnostic file will stay as long as the device isn't rebooted.

 

it will get updated every time the received packon on ... message happens.  So you can grab the file, then if it happens again, grab the file again.  That will give us extra data points.

If you'd like to try our latest features, consider signing up for beta: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Sign-Up-for-Beta-Access

If you're trying to debug a network issue, take a look at this help article: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
Emerging Member
Posts: 44
Registered: ‎01-10-2017
Kudos: 3

Re: TONS of errors, and frequent service drops Mesh network

So far I have not seen this file show up on any of my aps. As mentioned, the duplicate mac thing is much more rare than all the drops, which are the real issue, since they happen many times per day, and seem to have shifted to outages being HOURS long instead of just a 3-4 min outage.

 

Botton line, Its getting worse.

 

Is there data I should be trying to collect focused on the big outage?

 

New Member
Posts: 22
Registered: ‎10-06-2017
Kudos: 1

Re: TONS of errors, and frequent service drops Mesh network

@UBNT-BenBuckley 

Yes please send the debug software.  I was unable to access the system today as planned will try again either later this afternoon or later in the week.

Ubiquiti Employee
Posts: 1,471
Registered: ‎12-16-2015
Kudos: 310
Solutions: 10

Re: TONS of errors, and frequent service drops Mesh network

@robpatton long disconnects sound like a distinct and different issue.  Has anything changed in the environment lately that can be causing high interference or poor signal strength?

 

Are you willing to give me controller access so I can take a look?

If you'd like to try our latest features, consider signing up for beta: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Sign-Up-for-Beta-Access

If you're trying to debug a network issue, take a look at this help article: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
Emerging Member
Posts: 44
Registered: ‎01-10-2017
Kudos: 3

Re: TONS of errors, and frequent service drops Mesh network

The only changes made to the enviroment recently is the change in firmware.

 

Can I push syslog data to a syslog server of yours?  Will that help?

 

Otherwise I could set up a Teamviewer session on Friday?

 

Emerging Member
Posts: 44
Registered: ‎01-10-2017
Kudos: 3

Re: TONS of errors, and frequent service drops Mesh network

Ok, here is a list of up.down over the past week, based on my snmp collector.
 
2019-05-01 01:21:20 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-05-01 01:17:20 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-30 18:52:27 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-30 18:45:02 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-30 08:47:05 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-30 08:42:02 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-30 06:16:55 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-30 06:11:51 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-30 01:01:57 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-30 00:36:17 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-30 00:31:26 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-30 00:25:08 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-29 07:20:09 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-29 07:16:57 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-28 20:11:57 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-28 20:07:13 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-28 19:56:58 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-28 18:52:11 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-28 18:47:13 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-28 15:27:45 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-28 15:22:31 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-28 10:17:31 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-28 10:13:08 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-28 06:07:20 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-28 06:02:23 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-28 05:27:17 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-28 05:22:22 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-28 04:27:20 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-28 04:22:24 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-28 02:22:26 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-28 02:17:37 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-28 01:26:20 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-28 01:21:21 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-28 00:22:23 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-28 00:17:25 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-27 23:22:20 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-27 23:17:25 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-27 22:22:20 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-27 22:17:25 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-27 21:22:19 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-27 21:17:24 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-27 20:22:19 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-27 20:17:26 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-27 19:22:25 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-27 19:17:25 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-27 18:22:20 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-27 18:17:25 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-27 17:22:18 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-27 17:17:27 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-27 06:07:10 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-27 05:57:22 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-27 05:21:51 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-27 05:16:58 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-27 05:11:51 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-27 05:06:57 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-26 19:36:22 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-26 19:31:24 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-26 18:35:02 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-26 11:05:09 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-26 11:02:24 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-26 10:27:23 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-26 08:12:07 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-26 08:08:09 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-26 06:02:50 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-26 06:00:41 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-26 04:02:47 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-26 03:52:42 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-26 03:37:37 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-26 03:33:07 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-26 02:32:33 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-26 02:27:25 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-26 00:27:08 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-26 00:22:27 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-25 12:17:35 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-25 12:12:30 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-25 09:07:37 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-25 09:02:46 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-25 04:02:45 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-25 03:57:37 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-24 23:56:51 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-24 23:51:55 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-24 23:29:36 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-24 23:22:17 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-24 23:17:01 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-24 23:06:56 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-24 22:56:52 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-24 22:51:56 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-24 22:45:03 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-24 22:35:08 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-24 22:02:21 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-24 21:57:31 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-24 21:02:27 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-24 20:57:22 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-24 20:02:28 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-24 19:57:29 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-24 19:02:27 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-24 18:57:23 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-24 17:01:22 ubnt-ap-ac-mesh-4 Device status changed to Up
2019-04-24 16:55:59 ubnt-ap-ac-mesh-4 Device status changed to Down (snmp)
2019-04-24 15:57:16 ubnt-ap-ac-mesh-4 Device status changed to Up
Ubiquiti Employee
Posts: 1,471
Registered: ‎12-16-2015
Kudos: 310
Solutions: 10

Re: TONS of errors, and frequent service drops Mesh network

@robpatton you can enable syslog in the controller and point it to 192.76.255.30 (ubnt-ut.com), that's a log server in our office.  From there, I need to know your public IP address.

 

What might be easier though, and catch more info, is that after you're device goes down and comes back, just ssh in and grab /var/log/messages off the device and send it to me.

 

It's the same stuff that get's recorded to syslog, but in the case where your AP is going offline, the syslog messages won't make it.  So make sure that you have some down/up events, whithout a device reboot (as that will wipe out /var/log/messages), and collect that log and send it to me.

If you'd like to try our latest features, consider signing up for beta: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Sign-Up-for-Beta-Access

If you're trying to debug a network issue, take a look at this help article: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
Emerging Member
Posts: 44
Registered: ‎01-10-2017
Kudos: 3

Re: TONS of errors, and frequent service drops Mesh network

Ok, lets see if the forum will let me post this.

 

I dunno, it just wont let me upload the diag files.  This not the first time I've had a problem attaching files

 

Attachment
Emerging Member
Posts: 44
Registered: ‎01-10-2017
Kudos: 3

Re: TONS of errors, and frequent service drops Mesh network

Or mabee it worked ,  I'll ignore the weird errors

 

Ubiquiti Employee
Posts: 1,471
Registered: ‎12-16-2015
Kudos: 310
Solutions: 10

Re: TONS of errors, and frequent service drops Mesh network

@robpatton I've got the logs. I need some additional info from you, I'll touch bases directly.

If you'd like to try our latest features, consider signing up for beta: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Sign-Up-for-Beta-Access

If you're trying to debug a network issue, take a look at this help article: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
Emerging Member
Posts: 44
Registered: ‎01-10-2017
Kudos: 3

Re: TONS of errors, and frequent service drops Mesh network

Upgraded to the 2nd diag firmware, about 10am est this day, by 12:30 it had dropped 3 times.  Had to cold boot device after firmware.

 

All 3 of the mesh remote aps have this problem but this one AP drops about 5 times more than the rest.

 

 

Ubiquiti Employee
Posts: 1,471
Registered: ‎12-16-2015
Kudos: 310
Solutions: 10

Re: TONS of errors, and frequent service drops Mesh network

@robpatton were you able to get the debug files off of the AP before having to reboot?

 

Also, can I get access to your controller?

If you'd like to try our latest features, consider signing up for beta: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Sign-Up-for-Beta-Access

If you're trying to debug a network issue, take a look at this help article: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
Ubiquiti Employee
Posts: 1,471
Registered: ‎12-16-2015
Kudos: 310
Solutions: 10

Re: TONS of errors, and frequent service drops Mesh network

@robpatton were you able to get the debug files off of the AP before having to reboot?

 

Also, can I get access to your controller?

If you'd like to try our latest features, consider signing up for beta: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Sign-Up-for-Beta-Access

If you're trying to debug a network issue, take a look at this help article: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
Emerging Member
Posts: 44
Registered: ‎01-10-2017
Kudos: 3

Re: TONS of errors, and frequent service drops Mesh network

I am happy announce that after a remote session and some (what seem like simple) changes, (which I need to verify what was changed for sure)   I have experienced the longest stretch of uninterrupted service since the ver 4.x mesh architecuture changes.

 

FINALLY!

 

Will test over the weekend, but so far its a record.

 

 

New Member
Posts: 21
Registered: ‎01-06-2019
Kudos: 6
Solutions: 1

Re: TONS of errors, and frequent service drops Mesh network

Please let us know what settings were changed!
Emerging Member
Posts: 44
Registered: ‎01-10-2017
Kudos: 3

Re: TONS of errors, and frequent service drops Mesh network

Well, I had a good 10 hours, then had about 25 instancs of "AP Became Isolated" about 2am,   1 instance at 8am, 6:30pm, 6:30pm and the ap vanished at 11pm, and hasnt come back yet. 

 

So, perhaps its "better" but its still currently completely down.

 

 

 

Member
Posts: 188
Registered: ‎05-31-2018
Kudos: 25
Solutions: 4

Re: TONS of errors, and frequent service drops Mesh network

[ Edited ]

Hi @UBNT-BenBuckley 

 

After more than a week since the last disconnect, I checked today and saw that a node fell off the mesh at 1.06pm LDN local time. Sadly, my syslog failed, but I have grabbed /var/log/messages below.

 

Spoiler

Sun May  5 11:20:45 2019 kern.warn kernel: [3837755.741650] [wifi1] FWLOG: [37493116] WAL_DBGID_SECURITY_ALLOW_DATA ( 0x43a5d0 )

Sun May  5 12:07:44 2019 kern.warn kernel: [3840574.004921] [wifi1] FWLOG: [40378360] WAL_DBGID_BB_WDOG_TRIGGERED ( 0xea681ff8, 0x4300000a, 0x0, 0x7b3c )

Sun May  5 12:07:49 2019 kern.warn kernel: [3840579.005542] [wifi1] FWLOG: [40384066] WAL_DBGID_BB_WDOG_TRIGGERED ( 0xea683642, 0x4300000a, 0x0, 0x7b3e )

Sun May  5 12:20:46 2019 kern.warn kernel: [3841356.078120] [wifi1] FWLOG: [41179415] WAL_DBGID_SECURITY_ALLOW_DATA ( 0x43a5d0 )

Sun May  5 13:07:54 2019 kern.warn kernel: [3844184.343127] [wifi1] FWLOG: [44075905] WAL_DBGID_BB_WDOG_TRIGGERED ( 0xeaa08b81, 0x4300000a, 0x0, 0x7b45 )

Sun May  5 13:07:58 2019 kern.warn kernel: [3844188.343656] [wifi1] FWLOG: [44080119] WAL_DBGID_BB_WDOG_TRIGGERED ( 0xeaa09bf7, 0x4300000a, 0x0, 0x7b4e )

Sun May  5 13:07:59 2019 kern.warn kernel: [3844189.343746] [wifi1] FWLOG: [44080821] WAL_DBGID_BB_WDOG_TRIGGERED ( 0xeaa09eb5, 0x4300000a, 0x0, 0x7b4f )

Sun May  5 13:07:59 2019 kern.warn kernel: [3844189.343783] [wifi1] FWLOG: [44080953] WAL_DBGID_BB_WDOG_TRIGGERED ( 0xeaa09f39, 0x4300000a, 0x0, 0x7b50 )

Sun May  5 13:08:06 2019 kern.warn kernel: [3844196.344274] [wifi1] FWLOG: [44088327] WAL_DBGID_BB_WDOG_TRIGGERED ( 0xeaa0bc07, 0x4300000a, 0x0, 0x7b54 )

Sun May  5 13:20:46 2019 kern.warn kernel: [3844956.414538] [wifi1] FWLOG: [44865714] WAL_DBGID_SECURITY_ALLOW_DATA ( 0x43a5d0 )

EDIT: Also - 

 

Spoiler

3-Pro-Kitchen-BZ.v4.0.21# grep 'received packet on' /var/log/messages

Sun May  5 09:10:17 2019 kern.warn kernel: [3829927.068836] br0: received packet on ath3 with own address as source address

 

Screenshot 2019-05-05 at 20.33.28.png
Ubiquiti Employee
Posts: 1,471
Registered: ‎12-16-2015
Kudos: 310
Solutions: 10

Re: TONS of errors, and frequent service drops Mesh network

@robpatton can you check for `/var/run/loop_*` files on the AP again?  Also 

grep 'received packet on' /var/log/messages

Let's see if you're still getting loop packets or not.  This may be the same issue (with a different root cause), or it might be something else.

 

Also, feel free to email/dm me to resume, if you prefer.

If you'd like to try our latest features, consider signing up for beta: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Sign-Up-for-Beta-Access

If you're trying to debug a network issue, take a look at this help article: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
Reply