05-11-2018 03:07 PM
I'd love to get some move visibility around the suggestion regarding multicast traffic causing the issue from @shpoellet
Is everybody (or a large number of us) that is experiencing the issue using IGMP or something similar with multicast? Or is it a red herring?
05-12-2018 07:40 AM
From my experience we are facing more than the DHCP snooping issue.
I disabled DHCP snooping through config properties. But during my test with the latest firmware I started facing issues with Home Kit and even with my Printer. I think both use mDNS to announce themselfes. I get intermittent connection issues. So it could be a broadcast and or multicast issue beneath it.
If there would be a solution to disable this behaviour somehow I would gladly update to latest firmwares.
07-09-2018 06:26 AM
Just putting my two cents in.
I recently upgraded from 3.9.6 to the latest firmware on my 2 switches (16 and a 48) as well as upgrading the AP AC Pros to the lastest version. I use multicast for internal IPTV on a VLAN.
Randomly; devices connected to the core switch (48) would or wouldnt get an address from the DHCP server (an edgerouter poe 5). But 99% of the time; the devices wouldnt get an address assigned to them. I thought I'd done something wrong during the upgrade, (also went from 5.6 to 5.7 and then to 5.8) and was absolutely certain that in the "Corporate" network in Unifi I had originally set No DHCP as it was handled by the edgerouter etc. I ended up rebuilding my network config (stupidly didnt have easy access to a unifi backup) and hit the issue again. It didnt matter whether I was using Multicast etc.
If I connected the core switch the the 16 to a dumb gigabit switch then they'd both get IP addresses etc but would sporadically allow DHCP leases to be given out but most of the time, if a client didnt already have a lease (ie from me putting it on the dumb gigabit switch first) then it would fail. 100% sporadic. I thought I was going mad.
Decided to dowgrade the switches to the last known OK firmware. Downgraded and instantly everything got given a lease the way it should have all along - it was like a christmas tree being plugged in.
There is 100% a fault with the latest firmware - I havent taken the AP's back down - they seem OK from what I can tell - allowing clients to get DHCP leases etc. But I had to take the switch firmware back to 184.108.40.20613
To a degree, I'm happy to help diagnose the issue but when I upgrade, my network dies - I don't have hardware lying around to test with. If Ubiquiti want to give me a hand with an extra switch to be able to help pinpoint this down I'm more than willing to do everything I can to help.
I managed to get back onto the original controller and get the config off that before I reverted down, so I know the config I was using is the same as the config that had the issues - no changes etc. So it 100% is a firmware related issue.
07-09-2018 06:33 AM
There will be no fixes for issues like this.
The "fix" is to replace the Ubnt hardware with that of a different manafacturer if you want bugs fixed.
In my opinion Ubnt is home hardware and not production ready, and should not be used in production environments.
07-30-2018 02:17 PM
I agree. This is unacceptable. Every UBNT firmware upgrade is a chance to totally break your network. How come we the users are the experts on this issue and have identified exactly the last switch firmware without the problem (3.9.6) after which all future firmwares are broken? Why if we've drilled it down like this can't UBNT quickly and easily identify the code that broke the switches? Only a few changes go into each firmware release, and it shouldn't be hard to track down.
Why are we the users the experts on this issue? Why hasn't there been a post by a UBNT employee in ages? This should be an easy fix, but it appears that rolling out new features and new products is more important than actually making the platform bug free.
08-13-2018 12:26 PM
I've banged my head to the wall for months now with this very problem. It took me a long time to dig up enough information from my customers end users and gather log data to see what is the source of this problem. After a long time, I was able to verify that customer's DHCP relateted problems are caused by Dante's multicast audio traffic.
The DHCP problem was tricky. Maybe a few times a week I got reports at completely random times that the network wouldn't connect (wifi not leasing any IPs) and I saw ~100 pcs of 169.254.x.x IP-adresses on the controller. It took me some time to figure out that the problems started exactly at the time when switch interface connected to Dante-multicast streaming device went up. The problem appeared on all networks (VLANs, WLANs), although Dante networks are separate, non-routed VLANs.
I tried literally everything during this fight, but the hints in this thread about downgrading all switches to 3.9.6 were finally the point where the network began to work as it should. Until today I always had the newest firmware in UniFi switches, hoping that the new one would fix these problems since I had a feeling that they were switch related.
The site's primary network's topology is large but pretty simple: ISP switch - USG Pro as firewall/router - USW 16XG as core switch - many USW 24P250W and USW 48P500W and Cisco/HP switches as access switches - UniFi access points (anything from UAP-Pro to UAP-AC-HD). Most of the actual traffic is generated by Wi-Fi clients, but there are also many wired clients. USG does all routing and firewalling between site's 16 subnets. The Dante nets (primary and secondary) are "VLAN only" nets and most other networks are routed "Corporate" networks.
I wrote this message in two moods. First, I'm thankful that I managed to find the problem and was able to create a workaround (downgrade) so service level won't be so poor. Secondly, I'm angry to UBNT for not fixing this bug yet. Hope that it happens soon, since this one customer of mine alone has 70+ UBNT devices in their network and I will have to start recommending Cisco or some other more "reliable" vendor in the future. To be mentioned: I've been very happy with UBNT products and services, until I found this problem.
Please UBNT fix this ASAP.
08-13-2018 02:47 PM
That is a really interesting observation about Dante triggering the issues.
I’ve been following this thread for a while, and gave up hope that Ubiquiti would fix this bug directly. So I’ve implemented the DHCP snooping config workaround mentioned here: https://community.ubnt.com/t5/UniFi-Routing-Switching/Disable-DHCP-Snooping-on-USW/td-p/1510229/page...
But the mention of Dante multicast caught my attention. We don’t use Dante specifically, but we do have a similar setup, with separate network specifically for multicast video.
While I don’t particularly want to test this on our production network, as I’m thinking back, the times when DHCP started failing seem like they coincided with the times our multicast video server was turned on.
Is it possible that certain types of multicast traffic are somehow triggering the improper snooping behavior? And since it seems to span all networks (not just the ones carrying the multicast traffic), could the snooping problem be affecting trunk ports that carry multiple tagged VLANs?
If I can find a good time on our production network, I’ll test whether starting our multicast video server actually triggers the DHCP issues.
08-13-2018 03:19 PM
My network is set up in exactly that way too. Multicast traffic on its on vlan (vlan only), with other vlans set to corporate.
Trouble is, now i've taken the netowkr back down and its working, i really don't want to break it again just to get some log files which won't be enough and therefore I'll have to break it again, and again and again.
Ubnt please take notice and fix this!
08-13-2018 04:12 PM
I will test the snooping workaround some night in the future and tell you how well it works for me.
I'm not yet ready to implement it in "production" since one of the WLANs is fully open guest network and rogue DHCP-server is an actual threat when thinking of availability and security. So I think that DHCP snooping must be enabled. On the other hand, this bug affects availability so badly that the security sacrifice might be possible..
08-16-2018 05:06 AM
Woah! Have I finally found people with the same issue as me?
I started this thread the other day:
What I am finding is with these Just add power video encoders, when they are encoding video (eg. the HDMI is connected), DHCP stops working on the switch... but only that switch.. it appears that DHCP does work on other switches upstream and begins working as soon as I unplug the video decoder.
I can reproduce the problem at 2 different sites... both are on US-16-150W and are both running firmware 220.127.116.1152
08-16-2018 07:10 AM
08-16-2018 07:16 AM - edited 08-16-2018 07:28 AM
Not entirely sure myself yet. Trouble is, upgrading completely kills my network and I don't have any other switches (not even a small 8 or 16) to be able to test it without it wiping out all functionality in my home network. I really must buy a 8-60W to conduct some tests on. Maybe others can chime in? Definitely create a support ticket with ubnt with anything you find out. - have a look at the first page to see what debugging they want to see - I haven't been able to create a support ticket yet as I have no actionable info. But we need to get Ubnt support working on this - even though they've gone quiet - none of us can live on old firmware forever
08-22-2018 11:56 AM
having a similar issue
My issue is with a server
did you try the downgrade with other clients ?
09-07-2018 01:41 AM - edited 09-07-2018 01:47 AM
got the same issues starting with 3.9.15 firmware on UAP AC PRO with MS Windows DHCP Server.
DHCP is not working all the time, had to reboot AP in order to get DHCP working back for wifi clients.
So i cant upgrade my UAP AC PRO APs to more than v18.104.22.16837 (have not tried 3.9.6).
Set up :
* WIFI1 : Untagged VLAN, ipv4, 10.0.0.0/8 network, WPA 2 Enterprise, Radius, DHCP from Windows Server (most issues with this network)
* WIFI2 : Tagged VLAN, ipv4, 192.168.x.0/24 network, WPA 2 Perwonal, DHCP from Ubuntu ich-dhcpd (less errors)
09-13-2018 01:56 PM
I also got DHCP dropouts with 22.214.171.12452 until a couple of weeks ago when I downgraded the switch to 126.96.36.19913 and all has been well since then. Just tried to upgrade again and had dropouts again.
I added the flag to disable DHCP snooping (https://community.ubnt.com/t5/UniFi-Routing-Switching/Disable-DHCP-Snooping-on-USW/td-p/1510229) just in case (and checked that snooping was disabled) but that didn't make any difference (I didn't expect it to as I'm not doing DHCP relay - the DHCP server is on the LAN).
I don't have a USG (2 x AC-PROs and a US-8-150W). DHCP is from a Cisco 5505 ASA. The AC-PROs are both on firmware 188.8.131.5252 (and havn't been changed at all during this time) - I've now downgraded the switch again and everything started working again...
I can't try 184.108.40.20695 because there's no downgrade path to 220.127.116.1113 (which seems like the latest one that actually works). What do I do now Ubiquity?
10-20-2018 08:09 PM
10-20-2018 09:29 PM
If you have ended up here because of DHCP blocking issues - PLEASE READ
Ubiquiti do not care about these sorts of issues. This issue has existed for well over a year, across many firmwares.
DO NOT WASTE YOUR TIME.
Return your product, get a refund. Use this as an example if you need to RMA.
If you manage more than one ubiquiti device - let this serve as your warning. Any money you save on their hardware will be wasted supporting issues like this that they will flat out refuse to resolve.