USG Pro, USW48, chaos after power up!!!! (repeatable) - Any clues ? [LONG]

[einsteinagogo]

DHCP is OFF on USG. (on main LAN, and I turned it off on Guest Networks)

I also disconnected USG, and DHCP IP Addresses were still being given out..... from a non range defined anywhere on the LAN!

turned off Unfi Controller, and it fixed it's self....

which is what I've observed three different times now.

[manroweb]

So you say to shut down the UniFi controller you shut down the Windows server it runs on.

Are you sure that server does not have DHCP on?

If shutting that down fixes the problem it sounds like thats where DHCP is coming from as the UniFi SDN cannot do DHCP.

[einsteinagogo]

The server was setup especially for Unfi Controller, we would not add another DHCP role on this server, on the LAN, when we already have a Windows server on the LAN issuing DHCP IP Addresses.

This all worked well, with the APs, until USG was added.

There is no DHCP server running on that server.

The server which issued the DHCP packets and IP Address, is not pingable, and this server does not have that IP Address.

The logic does suggest it's coming from the Server, but then SDN does configure the USG.

Just going back...... this server was NOT powered up, and the DHCP IP Addresses we being issued to things....

which points to the USG Pro?

But going back to the DHCP Rogue server on the network, you would think if I turned off the Windows DHCP Server, my clients WiFi and LAN would get IP Addresses....

they didn't.....

and then also, there were times where clients timed out and did not get any IP Address.....from any DHCP server....

there are so many contradictions here, I certainly cannot follow what is going on....

other than 

1. Power on from COLD everything is fine. (no Windows Server DHCP no UniFi server present at this time).

2. Devices start getting weird IP Addresses....<---- this suggest it must be USG issuing DHCP (there are no servers on, just laptops and workstations, and other devices, printers etc)

3. WiFi goes OFF and re-provisions it self.... <<------ why - no idea -----> AP flashing white (no longer stread blue)

4. USW48/USW8 go bonkers...<---- have videos I can see slow ping times ---> broadcast storm why ?

5. Servers start up UniFi Controller/DHCP Server <----- it's a mess.... cannot get leases from Microsoft DHCP Server ---> could be broadcast storm ---> they don't get 192 or 10 IP Addresses any more

6. No WiFi... SSIDs missing....<---- why is gone ? no idea >---- controlled by Controller ?

7. Turn off Controller (server).

8. Fixed!

9. Wait 30 mins.

10. All fine again and working.

[TCOA]

Use on of the following to gather information like MAC address of the rogue.

https://gallery.technet.microsoft.com/DHCPLOC-Utility-34262d82
http://www.networksecurityhome.com/network_tools/dhcp_discovery/dhcp_discovery.html

[manroweb]

Please show me a screenshot of your LAN setups on UniFi (all of them).

And run wireshark as I suggested earlier to find where DHCP packets are coming from.

Just because something is not pingable does not mean anything Firewall can be set to block ping

[einsteinagogo]

now running..

[einsteinagogo]

other than the Authorised DHCP Server (Microsoft Server), that's the only DHCP packets.

[einsteinagogo]

the DHCP is issuing 192 and 10 IP Addresses, our Corp. network is 121.245.136.0/22

the IP Addresses that are being issues do not belong to any of the above networks.

[manroweb]

Can go into each and screenshot?
Not sure why you are blurring LAN ip/subnet, not like we can get to it.

Your USG is sat behind another router?

[einsteinagogo]

[einsteinagogo]

and you will not that the IP Address screenshot provided on the client is not from above...

it's not possible that my ISP is providing IP addresses ??? and these are being passed to my clients ?

but I cannot reproduce this other than powering everything off....

and if it really is DHCP, should it cause all Ubquiti Kit to go to 100% CPU and Broadcast storm????

[manroweb]

Try settings your LAN side to DHCP relay.

If you can turn on DHCP guarding. See if we can prevent it.

Your WAN looks odd. PPPoA/E show a subnet.

[einsteinagogo]

tried experimenting with DHCP relay before it didn't work, and Support stated it was Beta...

I can turrn on DHCP guarding and specifying the DHCP server on the LAN...

BUT if the DHCP server (Microsoft) was off, you would think I should be able to receive DHCP offers from this rouge DHCP server...

also can you explain the difference between IP Address posted above and networks define on the USG Pro ?

and do you think this is the reason for flatlined 100% CPU on ALL Ubquiti Kit ?

a rouge DHCP server ?

[manroweb]

You definitely have a rogue DHCP server, and that could be causing the strange activity on your network.

It is key you find it. Things will get worse. I still think you have some DHCP running on the Windows server the controller is running on as when you power that off your problems go away.

I see you are in UK. I'd be happy to do a Teamviewer session with you and see what I can find.

[einsteinagogo]

thanks for the offer.

there is no DHCP server.

if there was, if I turned off the Microsoft DHCP server, I would get an IP Address ???

wouldn't I ?

a rouge DHCP server takes out Ubuquti Networking Equipment ?

[einsteinagogo]

well if Ubquiti equipment is any good, I've turned on DHCP guarding....

so it should fine the rouge DHCP server within minutes if it exists ???

[manroweb]

A rogue DHCP server would issue IPs to anything it can communicate with.

The fact devices are getting an IP tells you there is something running DHCP services you don't want to be.

If there were no rogue DHCP server, then if you shut down your known DHCP server your devices would all end up with an APIPA IP.

[einsteinagogo]

Well this gets weirder....(but maybe next time I have a power cut, I'll disconnect these devices!)

I've had an alert from USG, it's found a Rouge DHCP Server on two ports on the USW48, which are NOT connected to anything!

However these MAC Addresses do match items on my WiFi network, and there are:-

A D-Link DCS-935L Network WebCam

A D-Link Motion Detector

(part of the MyDLink Security box)

However, both of the above devices do not have a DHCP server, logged in check all the menus - nothing.

DHCPLOC and DHCP Explorer also running on a WiFi and LAN devices did not find anything.

So not sure what to believe here?

They are set to DHCP themselves.....

[manroweb]

Disable these 2 ports and see if it appears elsewhere.

Still think you have a loop somewhere

[einsteinagogo]

After messing with these D-Link devices, there's something very odd about them.....or normal for most users!

when powered on (if there is no WiFi network available for them to join) they go into Ad-Hoc Mode for Setup issuing DHCP IP Addresses....by design.... e.g. they issue their ESSID for you to connect to...

e.g. you need to do this with them if you receive them new....to setup.

BUT, oddly when the WiFi network comes up, they join the network (still there is no DHCP available at this time servers still booting), so it seems they start issuing DHCP requests across the LAN....... and that screws up Ubquiti, and it seems at this point they do not ask DHCP for a new IP Address.....they try to continue using the IP Address they had originally before the power down. (which may have been alloctaed to another device), and it does not help there are two D-Link devices doing this...

10.x

192.x

It certainly gives me something to do (remember to turn them off at next power cut!)

IF DHCP causes this Broadcast storm...??

We can only wait until next power cut.