Reply
New Member
Posts: 10
Registered: ‎03-24-2018
Kudos: 9

Re: [FIRMWARE] 4.0.30.10217 for UAP/USW has been released | Stable Candidate

Hiya,

 

There's appears to be a common theme here in the issues we both describe. Broadcast traffic and VLANs (not dynamic in my case) and the 4.0.x release stream.

 

Really irritatingly, it appears to be intermittent. (related to # of clients?) I have 15/16 on the smallest network I have available - and the issue is present there too.

 

I used to have issues with DHCPv4 - however found a comment somewhere here about switching away from broadcast replies to unicast replies. Once I made that config change (DHCPv4 replies with unicast) my customer base has no further problem with IPv4. I did't consider this further until I read your post. (it worked, so I stoped getting angry calls).

 

However, IPv6 relies heavily on multicast traffic (which is broadcast across a whole VLAN set of ports when IGMP support is off). So - I'm not seeing broadcast traffic from wired -> wireless = no RA messages = broken IPv6.

 

I will test some more tonight (10AM here now) with some of the following and report back: (intermittent behaviour will make this frustrating)

 

  1. Wired sender -> wired receiver (prove my tests) - unicast and broadcast IPv4 ping
  2. Wired sender -> wireless receiver - unicast and broadcast - 5GHz radio and then 2.4GHz radio
  3. Wireless sender -> wired receiver (both radios again)
  4. Wireless sender -> wireless receiver (both radios again)
  5. Repeat 1-4 on VLAN tagged and untagged wired side network.

Impossible to eliminate completely due to intermittent nature of the symptoms, however will try to get as mich info as possible,

 

Dave

 

 

--

 

 

 

New Member
Posts: 10
Registered: ‎03-24-2018
Kudos: 9

Re: [FIRMWARE] 4.0.30.10217 for UAP/USW has been released | Stable Candidate

[ Edited ]

[ duplicate post - removed ]

Senior Member
Posts: 3,333
Registered: ‎05-19-2013
Kudos: 1451
Solutions: 33

Re: [FIRMWARE] 4.0.30.10217 for UAP/USW has been released | Stable Candidate


@wavesound wrote:

@chaicka wrote:

@wavesound wrote:
There are a lot of comments in the Beta forum about DHCP failing to renew when clients roam between APs and clients end up with a 169.x.x.x address. Since most of my devices are Apple devices, I assumed that they were to blame but the issue seems more widespread.

While there are posts on this, I just want to share that there are also non-posted instances where this does not happen at all. In short, the real cause of it does not seem to be a single factor.

 

I have tens of deployments and none are experiencing this for those under our maintenance, and no feedback for those not under our maintenance so far. One difference I can think of is use of USG and perhaps a set of features enabled on UniFi platform (be it on USW or UAP). Reason being ~95% of our deployments do not use USG (a long topic to share on why). They are either on ISP's ONR or EdgeRouter. Perhaps the simplicity setup of all these deployments helped avoid facing this 'DHCP-related' issue.


@chaicka 

 

Others have reported that you need to use VLANs with your SSIDs for the issue to appear. I'm using a USG as well as Ubiquiti switches which is also different from your deployment. Users in the beta forum are also reporting that the issue goes away if you revert back to 3.9.x.


For the one instance I have first-handedly seen, there is no VLAN for entire setup, hence no VLAN for SSID. So, again, there is no fix set of criterias for the '100FDX' issue to appear and reproducible. It is hard to fix something which cannot identify a repeatable root cause.

New Member
Posts: 10
Registered: ‎03-24-2018
Kudos: 9

Re: [FIRMWARE] 4.0.30.10217 for UAP/USW has been released | Stable Candidate

[ Edited ]

heya,

 

Ok. Managed to spend a couple of hours testing this. Initial testing had these consistent results:

 

1. Wired src -> wifi dst unicast ping works fine on both 2G and 5G radio

2. Wired src -> wifi dst broadcast ping works on 2G and does NOT work on 5G radio (and, no ARP seen at all on link when on 5G)

 

Didn't go any further with test list as this is enough. Would work initially (on reboot of all devices) and then a single roam or disconnect/reconnect made the issue reappear. This was repeatable.

 

Connected to a non unifi AP, same IP configuration for the fallback SSID (connected to same VLAN) and both tests work perfectly. again and again.

 

So, went to the controller and thought, "hey, let's just disable guest controls on a different SSID and try again". Repeated the test, and after 30 minutes of disconnect/reconnect/test/rinse/repeate/3 devices all known to show the issue. IT still works properly. ARP received. Ping broadcasts received.

 

So - I immediately jump to the conclusion, "Guest policy (no portal, just the single guest policy tickbox for the SSID) and 5G do not cooperate". Of course, being a responsible induhvidual, switched guest policy  back on and resume testing to see if it can be reproduced again.

 

And now I can not reproduce the issue. Disconnect, reconnect, roam, switch between bands and it is working properly.

 

Was there a configuration hangover during the upgrade from 4.0.21 that clearing and resetting the "Guest policy" for a different SSID was enough to clear? ( 6 SSIDs....)

 

FWIW - some AP association phase logs captured: (bold/underline significant differences perhaps)

 

NOT WORKING:

Apr 15 20:32:35 192.168.2.104 U7PG2,788a20d05fdc,v4.0.30.10217: hostapd: ath11: STA 18:3d:a2:96:d3:c8 IEEE 802.11: associated
Apr 15 20:32:35 192.168.2.104 U7PG2,788a20d05fdc,v4.0.30.10217: kernel: [159684.243049] wmi_unified_event_rx : no registered event handler : event id 0x901b
Apr 15 20:32:35 192.168.2.104 U7PG2,788a20d05fdc,v4.0.30.10217: hostapd: ath11: STA 18:3d:a2:96:d3:c8 WPA: pairwise key handshake completed (RSN)
Apr 15 20:32:35 192.168.2.104 U7PG2,788a20d05fdc,v4.0.30.10217: : wevent.ubnt_custom_event(): EVENT_STA_JOIN ath11: 18:3d:a2:96:d3:c8 / 2
Apr 15 20:32:35 192.168.2.104 U7PG2,788a20d05fdc,v4.0.30.10217: : wevent.ubnt_custom_event(): EVENT_STA_IP ath11: 18:3d:a2:96:d3:c8 / 192.168.2.177
Apr 15 20:32:36 192.168.2.104 U7PG2,788a20d05fdc,v4.0.30.10217: kernel: [159685.176174] [wifi1] FWLOG: [29316961] RATE: ChainMask 3, phymode 1044486, ni_flags 0x00223006, vht_mcs_set 0x0000, ht_mcs_set 0xffff, legacy_rate_set 0x1bf5776
Apr 15 20:32:36 192.168.2.104 U7PG2,788a20d05fdc,v4.0.30.10217: kernel: [159685.176208] [wifi1] FWLOG: [29316982] WAL_DBGID_SECURITY_ALLOW_DATA ( 0x43b708 )
Apr 15 20:32:36 192.168.2.104 U7PG2,788a20d05fdc,v4.0.30.10217: kernel: [159685.176224] [wifi1] FWLOG: [29317102] RATE: ChainMask 3, phymode 1044486, ni_flags 0x00223006, vht_mcs_set 0x0000, ht_mcs_set 0xffff, legacy_rate_set 0x0401
Apr 15 20:32:45 192.168.2.104 U7PG2,788a20d05fdc,v4.0.30.10217: : wevent.ubnt_handle_custom_alert_sta_assoc(): EVT_AP_STA_ASSOC_TRACKER_DBG: event_id: 2 vap: ath11 sta_mac: 18:3d:a2:96:d3:c8 auth_ts: 159684.203448 auth_delta: 0 assoc_delta: 0 wpa_auth_delta: 20000 radius_auth_delta: -1 radius_auth_status: N/A ip_delta: 140000 ip_assign_type: dhcp disassoc_count: 0 auth_failures: 0 assoc_failures: 0 wpa_auth_failures: 0 ip_failures: 0 assoc_status: 0 arp_status: yes dns_status: yes event_type: success
Apr 15 20:32:45 192.168.2.104 U7PG2,788a20d05fdc,v4.0.30.10217: hostapd: ath11: STA 18:3d:a2:96:d3:c8 RADIUS: starting accounting session 30F4D226AAC9753E

 

WORKING:

Apr 15 21:10:17 192.168.2.104 U7PG2,788a20d05fdc,v4.0.30.10217: hostapd: ath11: STA 18:3d:a2:96:d3:c8 IEEE 802.11: associated
Apr 15 21:10:17 192.168.2.104 U7PG2,788a20d05fdc,v4.0.30.10217: kernel: [161946.957589] wmi_unified_event_rx : no registered event handler : event id 0x901b
Apr 15 21:10:17 192.168.2.104 U7PG2,788a20d05fdc,v4.0.30.10217: hostapd: ath11: STA 18:3d:a2:96:d3:c8 WPA: pairwise key handshake completed (RSN)
Apr 15 21:10:17 192.168.2.104 U7PG2,788a20d05fdc,v4.0.30.10217: : wevent.ubnt_custom_event(): EVENT_STA_JOIN ath11: 18:3d:a2:96:d3:c8 / 3
Apr 15 21:10:18 192.168.2.104 U7PG2,788a20d05fdc,v4.0.30.10217: : wevent.ubnt_custom_event(): EVENT_STA_IP ath11: 18:3d:a2:96:d3:c8 / 192.168.2.177
Apr 15 21:10:18 192.168.2.104 U7PG2,788a20d05fdc,v4.0.30.10217: kernel: [161947.491012] [wifi1] FWLOG: [31633936] RATE: ChainMask 3, phymode 1044486, ni_flags 0x00223006, vht_mcs_set 0x0000, ht_mcs_set 0xffff, legacy_rate_set 0x1e2b230
Apr 15 21:10:18 192.168.2.104 U7PG2,788a20d05fdc,v4.0.30.10217: kernel: [161947.491046] [wifi1] FWLOG: [31633968] WAL_DBGID_SECURITY_ALLOW_DATA ( 0x437140 )
Apr 15 21:10:18 192.168.2.104 U7PG2,788a20d05fdc,v4.0.30.10217: kernel: [161947.491062] [wifi1] FWLOG: [31634025] RATE: ChainMask 3, phymode 1044486, ni_flags 0x00223006, vht_mcs_set 0x0000, ht_mcs_set 0xffff, legacy_rate_set 0x1d70401
Apr 15 21:10:27 192.168.2.104 U7PG2,788a20d05fdc,v4.0.30.10217: : wevent.ubnt_handle_custom_alert_sta_assoc(): EVT_AP_STA_ASSOC_TRACKER_DBG: event_id: 8 vap: ath11 sta_mac: 18:3d:a2:96:d3:c8 auth_ts: 161946.917904 auth_delta: 0 assoc_delta: 10000 wpa_auth_delta: 40000 radius_auth_delta: -1 radius_auth_status: N/A ip_delta: 90000 ip_assign_type: dhcp disassoc_count: 0 auth_failures: 0 assoc_failures: 0 wpa_auth_failures: 0 ip_failures: 0 assoc_status: 0 arp_status: yes dns_status: yes event_type: success
Apr 15 21:10:27 192.168.2.104 U7PG2,788a20d05fdc,v4.0.30.10217: hostapd: ath11: STA 18:3d:a2:96:d3:c8 RADIUS: starting accounting session B7CDC534DB6EE717

 

Happy to keep trying to reproduce and report back. 

 

laters!

 

dave

New Member
Posts: 10
Registered: ‎03-24-2018
Kudos: 9

Re: [FIRMWARE] 4.0.30.10217 for UAP/USW has been released | Stable Candidate

Agh, it is back. broadcast frames not being received on 5G radio clients.

 

Disabling guest policy has "made it work" again.

 

Will leave guest policy disabled for the next 48-72 hours and report back.

 

I have a love/hate relationship with intermittent faults.

 

dave.

 

 

--

 

New Member
Posts: 12
Registered: ‎02-03-2018
Kudos: 1

Re: [FIRMWARE] 4.0.30.10217 for UAP/USW has been released | Stable Candidate

[ Edited ]

I want to be clear that my issues with lack of stability occured and I do not, and have never used the guest wireless functionality.

 

Also, having seen the logs posted above by @davetick, I do find it bazzar that I'm also seeing logging on RADIUS accounting sessions being started when I don't have RADIUS authentication enabled on any VLANs, don't have accounting enabled, and I'm not using guest wireless functionality. Why are Radius accounting events even happening here?

 

Capture.PNG

Ubiquiti Employee
Posts: 9,533
Registered: ‎01-28-2013
Kudos: 16717
Solutions: 608
Contributions: 20

Re: [FIRMWARE] 4.0.30.10217 for UAP/USW has been released | Stable Candidate

@davetick thanks for all the details! Certainly an interesting issue. I'm planning to investigate first thing in the morning and will follow up. I don't think I need anything further, but I will reach out if I have any questions. 

 

Cheers,

Mike

Emerging Member
Posts: 96
Registered: ‎12-28-2016
Kudos: 33
Solutions: 3

Re: [FIRMWARE] 4.0.30.10217 for UAP/USW has been released | Stable Candidate

[ Edited ]

Today I was checking why my UAP-AC-M-Pro lost connectivity via it's wireless uplink to my UAP-AC-Pro, and I checked the /var/log/messages and I noticed the following:

 

Tue Apr 16 13:39:40 2019 user.info : wevent.ubnt_custom_event(): EVT_AP_RadarDetected ath3: radar found on channel 128 (5640 MHz)
Tue Apr 16 13:39:40 2019 kern.warn kernel: [92847.711958] dfs_process_radarevent: FILTER 34 - MATCH FOUND.
Tue Apr 16 13:39:40 2019 kern.warn kernel: [92847.711977] dfs_process_radarevent: Radar found on channel 128 (5640 MHz

Knowing the above, I know it's expected to lose connectivity, however my issue is that the event was not logged in the controller...

Now I'm not sure if it's a controller issue or a device firmware issue, but these Radar events used to get logged all the time in the controller, and now nothing (I though it's been a bit quiet recently...)

 

Possibley it's disconnecting the Wireless Uplink before it's sent the message to the controller?

 

Controller - 5.11.10

UAP-AC-M-Pro - 4.0.30.10217

UAP-AC-Pro - 4.0.30.10217

Controller: CloudKey Gen1
Gateway: USG-3P, USG-4P
Switch’s: US-8-150W, US-8-60W
AP’s: UAP-AC-Pro, UAP-AC-M-Pro, UAP-nanoHD
Member
Posts: 245
Registered: ‎03-31-2016
Kudos: 80
Solutions: 2

Re: [FIRMWARE] 4.0.30.10217 for UAP/USW has been released | Stable Candidate

@UBNT-MikeD-- I see exactly the same issue - random clients will not get DHCP lease - DHCP server (DHCPD on Centos7) is sending out DHCPOFFERS but they never reach the client - I have seen it on Macs and Lenovo (Windows 10) PCs

Emerging Member
Posts: 53
Registered: ‎11-06-2017
Kudos: 6
Solutions: 3

Re: [FIRMWARE] 4.0.30.10217 for UAP/USW has been released | Stable Candidate

Just as @spiderweb said, this version broke the portal page. It just won't show up. Downgraded and instantly started working again. There is an issue with this version and the guest portal.

Member
Posts: 245
Registered: ‎03-31-2016
Kudos: 80
Solutions: 2

Re: [FIRMWARE] 4.0.30.10217 for UAP/USW has been released | Stable Candidate

[ Edited ]

@jnew00 - I have encountered this issue on 4.0.29 - are you using hostname or ip address for your guest portal? Because our DNS server was getting blocked even though it is listed in the preauth section (ip/32). I could ping it but no DNS traffic could go through. Using tcpdump on the AP I was seeing that all DNS requests were rejected. But I even got random reports of this behavior on 4.0.21 so we had to disable guest portal for now.

New Member
Posts: 10
Registered: ‎03-24-2018
Kudos: 9

Re: [FIRMWARE] 4.0.30.10217 for UAP/USW has been released | Stable Candidate

[ Edited ]

Brief update ...

 

A few days on after having disabled guest policy (specifically : the guest policy tickbox against the only guest SSID out of 6), only a few occurrences of the "no broadcast traffic received" issue on the 5GHz radio - on any of the SSIDs. 2.4 still appears to be working as normal.

 

Contrasting with a >50% wifi session failure rate previously, this is a massive improvement for the 5G band! 

 

@UBNT-MikeD - you were hoping to have a look - any news that you can share?

 

cheers!

 

dave

 

--

<<edited for clarity>>

 

 

New Member
Posts: 11
Registered: ‎08-11-2017
Kudos: 5

Re: [FIRMWARE] 4.0.30.10217 for UAP/USW has been released | Stable Candidate


@davetick wrote:

Hiya,

 

Pleased to report - 4.0.30 is more stable in many ways, however, it seems to not forward broadcast traffic. Started off noticing that IPv6 wasn't working consistently. Connected a laptop direct (Wired) and can see all broadcast traffic. Connected via wireless (UAP AP AC Pro) - no broadcast traffic received.

 

Able to reproduce with "ping -b 192.168.x.255 -I interface" whist running wireshark on both the wired source and the [wireless|wired] receiver.  All brodcasts received when receiver connected via ethernet/wired, none received on wireless. This is NOT a guest SSID. I do have a guest SSID configured, however it is not this one.

 

Unifi config - as vanilla as I can get whilst not causing a firestorm by breaking my family service contract ;-)   :

 

  • UniFi AP-AC-Pro - running this release,
  • 6 SSIDs, one guest, others are IOT/trusted/dmz/ with schedule controlled
  • Wired backhaul
  • No "advanced features"
  • Auth: WPA personal
  • NOT a guest SSID
  • Rekey interval 3600 seconds (toying with reducing.)
  • "Block LAN to WLAN Multicast and Broadcast Data" not enabled (A test with this enabled and router / broadcast source MAC listed had same result)
  • No fast roaming
  • No vlan (for this SSID)
  • Not a hidden ssid
  • No UAPSD for all SSIDs
  • No wlan schedule for this test SSID
  • Multicast enhancement enabled & disabled - both same result.
  • "High performace devices " not enabled for all
  • DTIM @ 3 seconds for both
  • No data rate controls enabled.
  • No MAC filter

Verified broadcast is working by:

  • Wired network - to same switch - works fine using wireshark on sender&receiver.
  • Wireshark - verified MAC destination is FF:FF:FF:FF:FF:FF (yep, it's really a broadcast frame)

Verified ping is working using unicast ping. Echo request and response working fine. Unicast=ok-cool.

 

Happy to provide any more info on request.

 

Dave

 

--

 


With respect to your IPv6 issue, the problem might be that you need to whitelist the MAC of eth1 on your USG.  Not entirely clear from your post if you're using one or not, but your symptoms sound like you might be.  Under Wireless Networks -> Advanced Options, look for "Block LAN to WLAN Multicast and Broadcast Data", and make sure that the MAC of the internal interface of whatever router is handing out IPv6 RA to your network is in the Excepted Devices list. 

 

The controller only adds the external MAC (eth0) by default, but your wireless clients need to be able to recieve broadcast traffic from the internal interface.

 

 

Ubiquiti Employee
Posts: 4,009
Registered: ‎01-11-2016
Kudos: 1196
Solutions: 29

Re: [FIRMWARE] 4.0.30.10217 for UAP/USW has been released | Stable Candidate

@nrempe Do you have connectivity monitor and allow meshing to other UAPs enabled on the UAPs?
Want to try out new features or fixes before they're released as Stable? Sign up for Beta here: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Signup-for-Beta-Access
Having connectivity issues? See: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
Ubiquiti Employee
Posts: 4,009
Registered: ‎01-11-2016
Kudos: 1196
Solutions: 29

Re: [FIRMWARE] 4.0.30.10217 for UAP/USW has been released | Stable Candidate

@spiderweb Which version of controller are you using, and have you tried a force provision to that UAP, out of curiosity?
Want to try out new features or fixes before they're released as Stable? Sign up for Beta here: https://help.ubnt.com/hc/en-us/articles/204908664-How-To-Signup-for-Beta-Access
Having connectivity issues? See: https://help.ubnt.com/hc/en-us/articles/221029967-UniFi-Debugging-Intermittent-Connectivity-Issues-on-your-UAP
New Member
Posts: 12
Registered: ‎05-06-2016

Re: [FIRMWARE] 4.0.30.10217 for UAP/USW has been released | Stable Candidate

Using 5.9.29. Yes tried a force provision on all the WAPs and couldn't get it to work, then downgraded afterwards.

 

@davetick we are seeing same issue with the broadcast traffic, specifically broken IPv6 on roam. We have had to disable IPv6 on the wireless VLANs for now as clients were complaining about broken g suite etc. / timeouts as their machines wouldn't fallback to IPv4 quickly enough, entire network is BYOD which makes it hard. The few cisco WAPs we have left we don't have this problem, though I'd like to remove them completely !

Ubiquiti Employee
Posts: 211
Registered: ‎06-14-2016
Kudos: 132
Solutions: 6

Re: [FIRMWARE] 4.0.30.10217 for UAP/USW has been released | Stable Candidate

Hi @unifias ,

 

Can you please provide the device info by sending my private message? (Device panel -> Config -> Manage device -> Download device info)

Did you have any config change before you see device reboot?

Thanks.

 

New Member
Posts: 10
Registered: ‎03-24-2018
Kudos: 9

Re: [FIRMWARE] 4.0.30.10217 for UAP/USW has been released | Stable Candidate

[ Edited ]
@spiderweb - agreed - I'm now seeing more and more of the same issue even with all guest features disabled everywhere. IPv4 only working well for me as the DHCP server is unicast replying. If I reenable broadcast responses I have the same issues with IPv4! Strangely though - still appears to happen more often on 5GHz and less (not even sure I've seen it yet) on 2.4. YMMV. dave
New Member
Posts: 10
Registered: ‎03-24-2018
Kudos: 9

Re: [FIRMWARE] 4.0.30.10217 for UAP/USW has been released | Stable Candidate

[ Edited ]

@kf-kenobi- agree with your suggestion - that was one of the things I have tried - both with it enabled (and all MACs listed) and with broadcast/multicast control disabled. Neither behaved any differently in apparent rate of faulting clients.

 

thanks though!!

dave

New Member
Posts: 12
Registered: ‎02-03-2018
Kudos: 1

Re: [FIRMWARE] 4.0.30.10217 for UAP/USW has been released | Stable Candidate

@UBNT-jeff Yes to both.

Reply