Reply
New Member
Posts: 27
Registered: ‎12-21-2017
Kudos: 4

3.9.15.8011 blocks DHCP

[ Edited ]

After upgrading 2 x POE-150 and 1 x POE-60 - we now have a totally broken IPv4 network.

It appears the switches are blocking DHCP traffic.

I assume it's a new "feature"

There is no routing issue - IPv6 traffic is still working perfectly on all VLAN's

DHCP is working for all the devices on the network not behind these three switches.

 

Please, how can we roll back the firmware version to something that works?

Emerging Member
Posts: 87
Registered: ‎05-05-2016
Kudos: 9
Solutions: 4

Re: 3.9.15.8011 blocks DHCP

Go into settings, networks and edit the VLAN in question.  Is DHCP guarding enabled?  If it is either add in the IP address of your DHCP server into the trust server field or disable it.

 

If you want to roll back the firmware just go into devices, find the switch in question, config, manage device and enter the URL for the firmware.  I don't know if Ubnt maintain a list of download locations for all the firmware but a quick search of the forum should find a previous version.

New Member
Posts: 15
Registered: ‎05-14-2016
Kudos: 3

Re: 3.9.15.8011 blocks DHCP

[ Edited ]

I'm having the same issue. Guarding is disabled (although I probably should be using it). I took a look at my USG and confirmed the DHCP packets aren't making it to the device. I rolled back to v3.9.3.7537. You can get older firmware at https://www.ubnt.com/download/unifi-switching-routing

Ubiquiti Employee
Posts: 4,987
Registered: ‎08-08-2016
Kudos: 5343
Solutions: 344

Re: 3.9.15.8011 blocks DHCP

What's your DHCP setup like on the network @troykelly2? USG with DHCP server on it, or a DHCP relay involved, or? 

Ubiquiti Employee
Posts: 4,987
Registered: ‎08-08-2016
Kudos: 5343
Solutions: 344

Re: 3.9.15.8011 blocks DHCP


bhourigan wrote:

I'm having the same issue. Guarding is disabled (although I probably should be using it). I took a look at my USG and confirmed the DHCP packets aren't making it to the device. I rolled back to v3.9.3.7537. You can get older firmware at https://www.ubnt.com/download/unifi-switching-routing


On 3.9.x you want to be newer than 3.9.3. 3.9.6 would be much better, fixes provisioning issues with dropping all traffic for up to a minute or so during provisioning. 

https://community.ubnt.com/t5/UniFi-Updates-Blog/FIRMWARE-3-9-6-7613-for-USW-has-been-released/ba-p/...

Ubiquiti Employee
Posts: 8,714
Registered: ‎01-28-2013
Kudos: 14178
Solutions: 594
Contributions: 20

Re: 3.9.15.8011 blocks DHCP

To add to this, if you're willing to do even some brief debugging on 3.9.15, can you try the following? Would be interested to know what it logs. To perform this extra debugging you would connect to one of the switches (on 3.9.15) via shell, and issue:

telnet localhost
en
debug dhcpsnooping packet
exit
exit

After that, stay connected, and try to renew an IP on 1 or more client(s) behind the switch. After that, check:

grep DHCP /var/log/messages

Please share the output from that. You can mask the MACs if you like.. 

 

Cheers,

Mike

New Member
Posts: 27
Registered: ‎12-21-2017
Kudos: 4

Re: 3.9.15.8011 blocks DHCP

I can try it over the Christmas break - right now, we just need a functioning network.

 

I'm rolling everything back to 3.9.6.7613 - as 3.9.14.7960 and 3.9.16.8044 both have the DHCP blocking issue.

 

For clarity - no - DHCP Blocking is not enabled.

Ubiquiti Employee
Posts: 4,987
Registered: ‎08-08-2016
Kudos: 5343
Solutions: 344

Re: 3.9.15.8011 blocks DHCP

@troykelly2 what's your DHCP setup like? Relayed, or server directly on the network(s) in question? 

New Member
Posts: 27
Registered: ‎12-21-2017
Kudos: 4

Re: 3.9.15.8011 blocks DHCP

Microtik RouterOS as the on site router with a DHCP server in each VLAN locally.

                    +-----------+          +-----------+    
+-----------+    ----           |          -           |    
|           \\--/   +-----------+         /+-----------+    
+-----------+|-\       SG300             /   POE-150W       
   CCR1016   \  -\                      /                   
              \   -\+-----------+      /                    
               |    -           |     /     +-----------+   
               \    +-----------+   -/     /-           |   
                \      SG300       /     /- +-----------+   
                 \                /   /--     POE-150W      
                  |              / /--                      
                  \+-----------+//-         +-----------+   
                   |           --------------           |   
                   +-----------+            +-----------+   
                       SG300                  POE-60W       
                                                            
                                                            
                                                            
                                                            
                                                            
                                                            
                                                            
                                                            
New Member
Posts: 27
Registered: ‎12-21-2017
Kudos: 4

Re: 3.9.15.8011 blocks DHCP

For anybody with this issue:

 

ssh user@ip.address
upgrade https://dl.ubnt.com/unifi/firmware/US24P250/3.9.6.7613/US.bcm5334x.v3.9.6.7613.171102.1640.bin

Wait 15 minutes

 

Downgrading to 3.9.6.7613 restores functionality.

New Member
Posts: 15
Registered: ‎05-14-2016
Kudos: 3

Re: 3.9.15.8011 blocks DHCP


UBNT-MikeD wrote:

To add to this, if you're willing to do even some brief debugging on 3.9.15, can you try the following? Would be interested to know what it logs. To perform this extra debugging you would connect to one of the switches (on 3.9.15) via shell, and issue:

telnet localhost
en
debug dhcpsnooping packet
exit
exit

After that, stay connected, and try to renew an IP on 1 or more client(s) behind the switch. After that, check:

grep DHCP /var/log/messages

Please share the output from that. You can mask the MACs if you like.. 

 

Cheers,

Mike


I'm back on 3.9.15.8011 (USG 4.4.12.5032491), and I'll report back when it fails. I should of included this in my initial comment but it's not an immediate failure. I updated last night before I went to bed and all my stuff went offline sometime in the evening. I discovered it this morning. After power cycling my 24150W it worked, temporarily. It's my home network, so I'm happy to break it as much as you need.

New Member
Posts: 15
Registered: ‎05-14-2016
Kudos: 3

Re: 3.9.15.8011 blocks DHCP

[ Edited ]


I'm back on 3.9.15.8011 (USG 4.4.12.5032491), and I'll report back when it fails. I should of included this in my initial comment but it's not an immediate failure. I updated last night before I went to bed and all my stuff went offline sometime in the evening. I discovered it this morning. After power cycling my 24150W it worked, temporarily. It's my home network, so I'm happy to break it as much as you need.


It didn't take long. After I updated the firmware I disconnected/reconnected from my AP and this looks to be normal:

 

Dec 21 15:34:08 Rack daemon.info switch: DHCP_SNP: DHCP_DISCOVER port:1 vlan:1 - client MAC:XX:XX:XX:XX:XX:51
Dec 21 15:34:09 Rack daemon.info switch: DHCP_SNP: DHCP_OFFER port:24 vlan:1 - server IP:172.16.0.1, client MAC:XX:XX:XX:XX:XX:51 IP:172.16.1.78

 

Subsequent adapter resets on my test device fail and sometimes I see DHCP_DISCOVER but most of the time I don't. Here's a case later where my client doesn't see an IP:

 

Dec 21 16:08:35 Rack daemon.info switch: DHCP_SNP: DHCP_DISCOVER port:1 vlan:1 - client MAC:XX:XX:XX:XX:XX:51
Dec 21 16:08:36 Rack daemon.info switch: DHCP_SNP: DHCP_OFFER port:24 vlan:1 - server IP:172.16.0.1, client MAC:XX:XX:XX:XX:XX:51 IP:172.16.1.78

 

Port 24 is my HD AP and I'm using enterprise + USG RADIUS. As I think RADIUS is still beta let's call that an edge case / red herring.

 

What's interesting is I have a 8150W hanging off my 24 and that device power cycles and comes up ok - on the same IP. I don't see a DHCP_DISCOVER request for that so I think the DHCP client is falling back to a cached lease.

 

I power cycled my AV rack earlier which provides PoE to a few Ubnt cameras and other gear. Here's an execrpt of that DHCP_DISCOVER log:

 

 

ec 21 15:43:27 Rack daemon.info switch: DHCP_SNP: DHCP_DISCOVER port:7 vlan:10 - client MAC:C8:2A:14:57:CF:59
Dec 21 15:43:27 Rack daemon.info switch: DHCP_SNP: DHCP_DISCOVER port:4 vlan:10 - client MAC:F0:9F:C2:E6:54:44
Dec 21 15:43:30 Rack daemon.info switch: DHCP_SNP: DHCP_DISCOVER port:4 vlan:10 - client MAC:F0:9F:C2:E6:54:44
Dec 21 15:43:33 Rack daemon.info switch: DHCP_SNP: DHCP_DISCOVER port:4 vlan:10 - client MAC:F0:9F:C2:E6:54:44
Dec 21 15:43:35 Rack daemon.info switch: DHCP_SNP: DHCP_DISCOVER port:7 vlan:10 - client MAC:C8:2A:14:57:CF:59
Dec 21 15:43:43 Rack daemon.info switch: DHCP_SNP: DHCP_DISCOVER port:7 vlan:10 - client MAC:C8:2A:14:57:CF:59

 

To be fair most devices in my AV rack did come back, but I suspect that's due to the DHCP client implementation persisting the lease on disk (as rfc2131 says it should). If I statically assign an IP to any client I can get connectivity back.

 

tl;dr Sometimes I see a DISCOVER, sometimes I don't. When I do see a DISCOVER I don't always see an OFFER or REQUEST. 

 

Let me know how I can help further.

 

 

Ubiquiti Employee
Posts: 189
Registered: ‎04-16-2017
Kudos: 15
Solutions: 4

Re: 3.9.15.8011 blocks DHCP

[ Edited ]

@drapper39

Could you provide the topology and test again on the switch which DHCP server connected?

Please also provide output information using following commands:

US.v3.9.16# telnet 127.0.0.1

Entering character mode
Escape character is '^]'.

Warning!
The changes may break controller settings and only be effective until reboot.

(UBNT) >en

(UBNT) #show ip dhcp snooping server

 

New Member
Posts: 27
Registered: ‎12-21-2017
Kudos: 4

Re: 3.9.15.8011 blocks DHCP

@UBNT-AngusL I know the request was not to me - but I am happy to do some testing on the 60W. Please let me know what you need - you already have topology from a previous post.

New Member
Posts: 15
Registered: ‎05-14-2016
Kudos: 3

Re: 3.9.15.8011 blocks DHCP


UBNT-AngusL wrote:

@drapper39

Could you provide the topology and test again on the switch which DHCP server connected?

Please also provide output information using following commands:

US.v3.9.16# telnet 127.0.0.1

Entering character mode
Escape character is '^]'.

Warning!
The changes may break controller settings and only be effective until reboot.

(UBNT) >en

(UBNT) #show ip dhcp snooping serve

 


Sure thing.

 

I have a USG pro with a 24150W connected, and then a HD AP, legacy (non-ubnt AP), and 8150W connected off of that. 

 

(UBNT) #show ip dhcp snooping server

Total number of bindings: 0

IP Address VLAN Lease (Secs)
--------------- ---- -----------

Mac IP Address VLAN Intf Blocked Lease (Secs)
----------------- --------------- ---- ---- ------- ------------
80:2A:A8:CD:54:A5 172.16.0.1 1 24 0 86400
80:2A:A8:CD:54:A5 172.16.8.1 5 24 0 86400
80:2A:A8:CD:54:A5 172.16.12.1 15 24 0 86400

 

FWIW my RADIUS authenticated SSID seems to give me the most trouble.

Ubiquiti Employee
Posts: 189
Registered: ‎04-16-2017
Kudos: 15
Solutions: 4

Re: 3.9.15.8011 blocks DHCP

@troykelly2

That's great, please test on the 60W and get me the topology come out on controller and export the site (SETTINGS --> Site).

Ubiquiti Employee
Posts: 189
Registered: ‎04-16-2017
Kudos: 15
Solutions: 4

Re: 3.9.15.8011 blocks DHCP

@drapper39

There is no vlan10 on the output information. What's the DHCP server on vlan10?

New Member
Posts: 15
Registered: ‎05-14-2016
Kudos: 3

Re: 3.9.15.8011 blocks DHCP


UBNT-AngusL wrote:

@drapper39

There is no vlan10 on the output information. What's the DHCP server on vlan10?


vlan10 is l2 only - it goes to a nanobeam gen2 that connects to a neighbor's home network in 192.168.1.0/24 and they run dhcp. i am not using dhcp relay.

New Member
Posts: 8
Registered: ‎08-20-2016
Kudos: 1

Re: 3.9.15.8011 blocks DHCP

*** Completely agree this new firmware is blocking DHCP. ***  I just spent the last 6 days troubleshooting why my wifi clients would not get an IP from the dhcp server.  A ridiculous amount of time to be frank.  Read thru many posts in this community but none were successful in solving my problem until I came across this post. I contributed to my mess by introducing too many changes to my network at one time.  I added a Security Gateway Pro 4, a non-POE 24 port Unifi switch, and an 8 port POE Unifi switch.  And I upgraded all firmware.  After having installed all that and upgrading firmware, wifi clients could not get an IP from DHCP.  I eventually removed everything out of the equation and just connected the AP to the Edge Router (DHCP Svr) and clients could then get an IP.  So had to be something with the Switches.  I had verified that the switch configuration was fine.  That just left the firmware as the possible culprit - which is the subject of this thread.  I downgraded to 3.9.6.7613 and - presto - everything with DHCP was working again for the wifi clients. 

 

Ubiquiti needs to review/overhaul the testing procedures that let this loose in the wild before they release future firmware updates. This update seriously affected a critical service. .

New Member
Posts: 27
Registered: ‎12-21-2017
Kudos: 4

Re: 3.9.15.8011 blocks DHCP

It has been frustrating to see how poorly this has been handled. The firmware should have been revoked immediately on confirmation that it was an issue.
It's really not hard to reproduce, so I can't wrap my mind around why there's been no action.
We (and our respective production environments) are being treated as beta testers, and that means Ubiquiti are not providing production ready hardware and software.
Reply