Reply
Ubiquiti Employee
Posts: 3,898
Registered: ‎01-11-2016
Kudos: 1160
Solutions: 29

Re: Intermittent connectivity loss with UAP-AC-PRO

@adelman If you use ssh for +30 seconds or so, does the situation improve, or is it always slow no matter how long you use it? I've noticed that MacBooks can have poor latency until it senses that you are using a latency-intense application, like ssh.

Want to try out new features or fixes before they're released as Stable? Sign up for Beta here: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Signup-for-Beta-Access
Having connectivity issues? See: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
Ubiquiti Employee
Posts: 3,898
Registered: ‎01-11-2016
Kudos: 1160
Solutions: 29

Re: Intermittent connectivity loss with UAP-AC-PRO

@evharten No disconnects today either? Using a static IP really shouldn't affect battery life too much - there are plenty of other broadcasts that will be coming out of your phone.

 

Thinking about it some more, this issue could potentially occur with a DHCP server that replies using broadcast packets (some do). Since Apple devices aggressively power-save and we know they miss broadcast packets (due to our low DTIM period setting of 1), they could be missing the DHCP response.

 

Any chance you can get a pcap/Wireshark capture of your DHCP server's lease response? If it's broadcast, this may be why you see issues. We may be able to work around it by setting the DTIM period temporarily:

 

iwpriv athX dtim_period 4

killall hostapd

 

(replace athX with your AP interface, using iwconfig to see what that is)

 

Want to try out new features or fixes before they're released as Stable? Sign up for Beta here: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Signup-for-Beta-Access
Having connectivity issues? See: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
Member
Posts: 235
Registered: ‎12-27-2014
Kudos: 69
Solutions: 1

Re: Intermittent connectivity loss with UAP-AC-PRO

Up to two minutes of continuously typing into an ssh "into" the MacBookPro, no change. >1s response to the character echos.

Regular Member
Posts: 755
Registered: ‎10-23-2013
Kudos: 341
Solutions: 7

Re: Intermittent connectivity loss with UAP-AC-PRO


@UBNT-jeff wrote:

@evharten No disconnects today either? Using a static IP really shouldn't affect battery life too much - there are plenty of other broadcasts that will be coming out of your phone.

 

Thinking about it some more, this issue could potentially occur with a DHCP server that replies using broadcast packets (some do). Since Apple devices aggressively power-save and we know they miss broadcast packets (due to our low DTIM period setting of 1), they could be missing the DHCP response.

 

Any chance you can get a pcap/Wireshark capture of your DHCP server's lease response? If it's broadcast, this may be why you see issues. We may be able to work around it by setting the DTIM period temporarily:

 

iwpriv athX dtim_period 4

killall hostapd

 

(replace athX with your AP interface, using iwconfig to see what that is)

 


So far no disconnects, i have tried the dtim_period 4 before when you said it first in this topic, to see if that would make a difference, sadly it didnt.

I'll put on an pcap/wireshark later on today.

Ubiquiti Employee
Posts: 3,898
Registered: ‎01-11-2016
Kudos: 1160
Solutions: 29

Re: Intermittent connectivity loss with UAP-AC-PRO

@evharten So even with the DTIM period set to 4, you still see an auto-IP of 169.254.x.x sometimes?

Want to try out new features or fixes before they're released as Stable? Sign up for Beta here: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Signup-for-Beta-Access
Having connectivity issues? See: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
Regular Member
Posts: 755
Registered: ‎10-23-2013
Kudos: 341
Solutions: 7

Re: Intermittent connectivity loss with UAP-AC-PRO


@UBNT-jeff wrote:

@evharten So even with the DTIM period set to 4, you still see an auto-IP of 169.254.x.x sometimes?


Yes exactly.

Member
Posts: 235
Registered: ‎12-27-2014
Kudos: 69
Solutions: 1

Re: Intermittent connectivity loss with UAP-AC-PRO

I see this too, but it usually clear in 5-10 seconds and I get an address from the DHCP server.

Ubiquiti Employee
Posts: 3,898
Registered: ‎01-11-2016
Kudos: 1160
Solutions: 29

Re: Intermittent connectivity loss with UAP-AC-PRO

@evharten Can you describe what switches and DHCP server you are using?

Want to try out new features or fixes before they're released as Stable? Sign up for Beta here: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Signup-for-Beta-Access
Having connectivity issues? See: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
Member
Posts: 235
Registered: ‎12-27-2014
Kudos: 69
Solutions: 1

Re: Intermittent connectivity loss with UAP-AC-PRO

So what are your plans to address this DTIM related problem with Macs?  

 

Why is 2.4Ghz not affected?

 

Can you loan me some UAP-PROs while you work on a solution to this problem?

Regular Member
Posts: 755
Registered: ‎10-23-2013
Kudos: 341
Solutions: 7

Re: Intermittent connectivity loss with UAP-AC-PRO

[ Edited ]

@UBNT-jeff wrote:

@evharten Can you describe what switches and DHCP server you are using?


Ofcourse:

 

Router (which also does the DHCP server): ERPro-8

Switches: ES48-750W, ES-(NDA), US-8-150W, and an ES48-500W

All switches are connected to eachother with Fiber, the ES48's and the NDA switch are with 10Gbit, the US-8 is connected with 1Gbit fiber, the ERPro is connected thru 1G UTP

 

You can ask @UBNT-MikeD i've send him an diagram of my network a few months back Man Wink

New Member
Posts: 20
Registered: ‎02-19-2016
Kudos: 1

Re: Intermittent connectivity loss with UAP-AC-PRO

Hi There.

 

I had 5 UAP-LR on my site. For performance improwments of dense zones I replaced two of them with AP-AC-PRO. But after that, I started to have Internet connection issues every 2 or 3 days from clinets wich sits around AP-AC-PRO's. No problem with wired clients or UAP-LR's 

 

Is this the same problem you discus? 

 

Controller version  4.8.15

 

Regular Member
Posts: 755
Registered: ‎10-23-2013
Kudos: 341
Solutions: 7

Re: Intermittent connectivity loss with UAP-AC-PRO

@UBNT-jeff correction, even with an fixed ip traffic stops, just noticed it.

 

Emerging Member
Posts: 102
Registered: ‎03-03-2014
Kudos: 59
Solutions: 2

Re: Intermittent connectivity loss with UAP-AC-PRO

[ Edited ]

@UBNT-jeff - not sure about the other poster here, but we're using a basic MS DHCP server.

Our lease time for the VLAN that's affecting me is 8 days. I experience the issue about every other day, once or twice.

 

Speaking of VLANs, I'm running 3. One 'main,' one 'guest,' and one other.

 

edit: switch is a Cisco 3560, trunked back to a 6500 series core.

Highlighted
New Member
Posts: 32
Registered: ‎04-28-2016
Kudos: 8

Re: Intermittent connectivity loss with UAP-AC-PRO

[ Edited ]

My situation seems to fall in line with all the 4.8.14/15 problems.

 

My quick run-down :

 

debian VM

 

19 Access points 7 uap-standard and 11 uap-pro

450 ipads and 50 pc laptops

Access points are at least 3 years old.

Ran 3.2.10 controller forever with zero problems

Upgraded to 3.8.14 2 weeks ago users started mentioning connection issues

Enabled min rssi on all access point last Friday at 3pm

All access points started disconnecting at 3:45

Switches do not acknowledge that ethernet ever goes down but ping does not respond nor ssh

This happens every hour on the nose and sometimes skips an hour

Access point is still broadcasting ssid's  but all clients are disconnected

Remains like this 10 minutes and then reconnects  

 

What I have done.

Turned off min rssi

Factory reset through ssh and physically with button

Tried to revert to a snapshot but it would not adopt access points. AP status was decrypt error regardless of resetting access point

Upgraded reverted snapshot vm back to 4.8.15

 

Current situation :

Turned all logging to debug mode

Monitoring /var/log/unfi/server.log and mongo.log

Monitoring /var/log/messages on an AP

***Stood up a new debian install and installed unifi fresh. I moved one access point from old controller to the new one and I also installed a new access point that was baught maybe 3 months ago.

I have log files but debug mode was not turned on during a disconnect moment

 

 

 

The white line in graph are times the AP is not reachable

 

Ubiquiti Employee
Posts: 3,898
Registered: ‎01-11-2016
Kudos: 1160
Solutions: 29

Re: Intermittent connectivity loss with UAP-AC-PRO

@jdavis4155 Can you get the kernel/syslogs from several of the APs at the moment that the disconnection occurs? To do this, ssh into the AP (after you can ssh again to it) and copy the file /var/log/messages off of it.

 

When the clients are disconnected from the APs each hour, does it still show WiFi bars on the clients, or are the WiFi bars gone, too? You said that the APs are still beaconing, but it would be strange if they are still beaconing for the WiFi bars to be gone on the clients (i.e. for them to be disconnected from WiFi).

 

Want to try out new features or fixes before they're released as Stable? Sign up for Beta here: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Signup-for-Beta-Access
Having connectivity issues? See: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
New Member
Posts: 32
Registered: ‎04-28-2016
Kudos: 8

Re: Intermittent connectivity loss with UAP-AC-PRO

[ Edited ]

login as: admin
admin@192.168.1.36's password:

 

BZ.v3.3.20# tail -f /var/log/messages
Apr 29 08:11:25 UBNT daemon.notice hostapd: AP-STA-DISCONNECTED 48:d7:05:4e:a1:3a
Apr 29 08:11:57 UBNT authpriv.info dropbear[4395]: Child connection from 192.168.1.10:53623
Apr 29 08:12:01 UBNT daemon.info hostapd: ath0: STA 5c:f5:da:7b:a6:11 IEEE 802.11: sta_stats
Apr 29 08:12:01 UBNT daemon.info hostapd: ath0: STA 5c:f5:da:7b:a6:11 IEEE 802.11: disassociated
Apr 29 08:12:01 UBNT daemon.notice hostapd: AP-STA-DISCONNECTED 5c:f5:da:7b:a6:11
Apr 29 08:12:03 UBNT daemon.info hostapd: ath0: STA 78:fd:94:3a:e5:40 IEEE 802.11: associated
Apr 29 08:12:03 UBNT daemon.notice hostapd: CTRL-EVENT-EAP-STARTED 78:fd:94:3a:e5:40
Apr 29 08:12:03 UBNT daemon.notice hostapd: CTRL-EVENT-EAP-PROPOSED-METHOD vendor=0 method=1
Apr 29 08:12:08 UBNT daemon.notice hostapd: CTRL-EVENT-EAP-STARTED 78:fd:94:3a:e5:40
Apr 29 08:12:12 UBNT authpriv.notice dropbear[4395]: Password auth succeeded for 'admin' from 192.168.1.10:53623
Apr 29 08:12:46 UBNT daemon.info hostapd: ath0: STA 74:81:14:79:2e:e6 IEEE 802.11: associated
Apr 29 08:12:46 UBNT daemon.notice hostapd: CTRL-EVENT-EAP-STARTED 74:81:14:79:2e:e6
Apr 29 08:12:46 UBNT daemon.notice hostapd: CTRL-EVENT-EAP-PROPOSED-METHOD vendor=0 method=1
Apr 29 08:12:51 UBNT daemon.notice hostapd: CTRL-EVENT-EAP-STARTED 74:81:14:79:2e:e6
Apr 29 08:12:51 UBNT daemon.info hostapd: ath0: STA 9c:04:eb:7f:6d:d2 IEEE 802.11: associated
Apr 29 08:12:51 UBNT daemon.notice hostapd: CTRL-EVENT-EAP-STARTED 9c:04:eb:7f:6d:d2
Apr 29 08:12:51 UBNT daemon.notice hostapd: CTRL-EVENT-EAP-PROPOSED-METHOD vendor=0 method=1
Apr 29 08:12:56 UBNT daemon.info hostapd: ath0: STA 9c:35:eb:4e:6a:fa IEEE 802.11: associated
Apr 29 08:12:56 UBNT daemon.notice hostapd: CTRL-EVENT-EAP-STARTED 9c:35:eb:4e:6a:fa
Apr 29 08:12:56 UBNT daemon.notice hostapd: CTRL-EVENT-EAP-PROPOSED-METHOD vendor=0 method=1
Apr 29 08:12:56 UBNT daemon.info hostapd: ath0: STA 9c:04:eb:7f:6d:d2 IEEE 802.11: sta_stats
Apr 29 08:12:56 UBNT daemon.info hostapd: ath0: STA 9c:04:eb:7f:6d:d2 IEEE 802.11: disassociated
Apr 29 08:12:56 UBNT daemon.notice hostapd: AP-STA-DISCONNECTED 9c:04:eb:7f:6d:d2
Apr 29 08:12:57 UBNT daemon.info hostapd: ath0: STA 9c:04:eb:7f:6d:d2 IEEE 802.11: associated
Apr 29 08:12:57 UBNT daemon.notice hostapd: CTRL-EVENT-EAP-STARTED 9c:04:eb:7f:6d:d2
Apr 29 08:12:57 UBNT daemon.notice hostapd: CTRL-EVENT-EAP-PROPOSED-METHOD vendor=0 method=1
Apr 29 08:12:58 UBNT daemon.info hostapd: ath0: STA 5c:f5:da:b5:02:2a IEEE 802.11: associated
Apr 29 08:12:58 UBNT daemon.notice hostapd: CTRL-EVENT-EAP-STARTED 5c:f5:da:b5:02:2a
Apr 29 08:12:58 UBNT daemon.notice hostapd: CTRL-EVENT-EAP-PROPOSED-METHOD vendor=0 method=1
^C
BZ.v3.3.20#

 

 

 

I will verify Wireless bars ect.. on next disconnect

Ubiquiti Employee
Posts: 3,898
Registered: ‎01-11-2016
Kudos: 1160
Solutions: 29

Re: Intermittent connectivity loss with UAP-AC-PRO

@jdavis4155 Sorry, can I get the full logs, and can you try to correlate an outage by marking the outage time in the log file?

Want to try out new features or fixes before they're released as Stable? Sign up for Beta here: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Signup-for-Beta-Access
Having connectivity issues? See: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
New Member
Posts: 32
Registered: ‎04-28-2016
Kudos: 8

Re: Intermittent connectivity loss with UAP-AC-PRO

[ Edited ]

Access points are disconnecting right now. I do not show any ssid from access points as they disconnect. So my previous statement was incorrect. I will have logs shortly when AP's come back online

 

Edit :  I am sorry its hard to catch them. It seems like the standards quit broadcasting ssid during "heartbeat missed" and then "disconneted" but the uap-pro continued to broadcast the entire time.

 

Problems started at 8:36 I highlighted them. Its not letting me attach a file...

 

Edit2 : So in my test controller the access point that is 3 years old is disconnecting, the access point that is months old is rock solid

 

 

Ubiquiti Employee
Posts: 3,898
Registered: ‎01-11-2016
Kudos: 1160
Solutions: 29

Re: Intermittent connectivity loss with UAP-AC-PRO

@jdavis4155 It's interesting that you see the message: ipready.br0 = 192.168.1.20 at about the same time as the failure. Are you using DHCP on these APs? If you extend your DHCP leases to beyond an hour, does the failure take longer to happen? Do you see the DHCP requests on your DHCP server near the time of the failure?

Want to try out new features or fixes before they're released as Stable? Sign up for Beta here: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Signup-for-Beta-Access
Having connectivity issues? See: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
New Member
Posts: 32
Registered: ‎04-28-2016
Kudos: 8

Re: Intermittent connectivity loss with UAP-AC-PRO

[ Edited ]

We use dhcp reservations. I will investigate 

 

DHCP lease time changed to 6 hours.

 

We are using pfsense to handle dhcp and reservations do not have a renew/release time.. I will tcpdump | grep 1982.168.1.36 to watch for a dhcp request.

 

Edit : Something that just struck me as add is that 192.168.1.20 is.. nothing.... I have no idea where it is even pulling that from. Thats the default access point IP so its basically saying it can not get an IP and reverting to that. Let me manually set an ip on a problem access point and see what we get.

Reply