Reply
Member
Posts: 114
Registered: ‎01-26-2017
Kudos: 6
Solutions: 3
Accepted Solution

USG Pro, USW48, chaos after power up!!!! (repeatable) - Any clues ? [LONG]

[ Edited ]

Not sure where to go with this issue.....but it's now becoming a real PITA, and can easily be repdoruced as below, so it's not a one off, but it has taken me 3 hours to try and resolve it this morning, so I'm finally posting here, and will escalate to Ubquiti support.

 

I've never had so many network related issues, with other vendors such as HPE, Cisco - but I have no idea what is causing the following issue!

 

All is working "well" usually (SNMP issues, occasional USG disconnection)

 

UniFi Controller - 5.9.29 - Windows 2012R2 - Service Instance
USG PRO 4.4.22.5086057
1 x USW-48 3.9.54.9373

1 x USW-8 3.9.54.9373
2 x UAP-AC-PRO 3.9.54.9373

 

The PoE switch is connected to the USW48 via 4 trunk ports, and the UAP-AC-PRO are connected to the 2 x UAP-AC-PRO. The USG is linked to the USW-48, and the USG is connected upstream with ethernet cable to fibre media converter to ISP.

 

1. All networking equipment is on UPS, so PoE, USG1, USW48, USW8 - last about 1.5 hours before UPS shuts down.

2. All Servers are also on UPS and shutdown the UniFi Controller is a virtual machine hosted on VMware vSphere (ESXi) - it's Windows 2012 R2 server.

 

The scenario is as follows, eventually we've gone DARK (all equipment OFF - servers and networking).

 

A. Power is restored.

B. Networking equipment USW48, USG8, 2xAPs, USG1 come back up quickly. Flashing white, and then all blue.

C. WiFi available e.g. SSID names can be seen and joined.

 

Networking is then working as expected, e.g. can reach the internet, ping times to 8.8.8.8 - 10ms.

 

LAN and WiFi devices are working, at this time Servers are NOT initialised because storage needs to come up first, before ESXi host servers are initialised.

 

BUT then something ODD happenes....

 

AP start flashing white, WiFi disappears, iPads and iPhones cannot join WiFi - states incorrect password.

 

USW48 - switch has gone bonkers and ALL lights are flashing at the same time (I've a video of this!)

 

Laptops on WiFi are disconnected.

 

Fixed LAN devices ping times are very slow....e.g. ping to local IP devices, 8.8.8.8 internet are now over 1000ms.

 

Servers are now UP including Active Directory, DNS, DHCP and UniFi.

 

AP provision again, White Light, after initially coming up correctly after power outage.

 

What is also very odd, is devices start receiving DHCP IP Addresses from something 10 and 192 - the DHCP server that issues these cannot be pinged. (I disconnected the USG and 10 and 192 DHCP addresses are still being issued, I turned off DHCP on these LANs on the Controller and they still appear).

 

New Network Devcies cannot receive IP Addresses from LAN based server on WiFi and LAN.

 

I noticed an AP was disconnected, I unplugged network cables to reset them, and they came back connected with STUN errors, which finally disappeared!

 

I Shutdown UniFi server and all returned to normal!

 

Strange network activity stopped, WiFi devices received IP Addresses, WiFi networks become available.

 

The moment I saw the switch stop flashing I new the issue was over!!!

 

WHY????

 

Uploading videos to YouTube!

 

The screenshot is of a Windows 10 LAN-fixed PC, which gets an IP Address from DHCP ???? is this from the Controller ?

 

it does not make sense, no range on the controller is specificed as this ? and was also disabled ?

 

After 30 minutes of Controller OFF, I powered back on, and no issues returned...

 

other than chasing my tail for 3 hours after power on.

UCK-G2-PLUS v.0.8.12•UniFi Network 5.9.29-11384-1• UniFi Protect 1.6.1
USG 4.4.22.5086045 •USG PRO 4.4.22.5086057
USW-48 4.0.14.9736 • USW-8 4.0.14.9736
UAP-AC-PRO 4.0.14.9736• UAP-AC-PRO 4.0.14.9736
UVC-G3-Flex 4.9.25
IMG_3763.jpg

Accepted Solutions
Member
Posts: 114
Registered: ‎01-26-2017
Kudos: 6
Solutions: 3

Re: USG Pro, USW48, chaos after power up!!!! (repeatable) - Any clues ? [LONG]

Despite uploading all logs, and escalation to Tier 2 Support.

This was the response so other users of Uqui. equipment, can see the response

"I'm afraid this is something which is beyond our scope of support as it is quite evident that it is your LAN hosts which are causing the issue in your network. We offer support for all UBNT devices. We will do our best to ensure your Ubiquiti product is working in accordance with our warranty, but we may not be able to resolve problems or answer questions when third-party devices are involved. However, looking at your findings and the steps that you have performed already, unfortunately, there isn't much we can do from our side."

The issue does not occur on HPE or Cisco switches.

UCK-G2-PLUS v.0.8.12•UniFi Network 5.9.29-11384-1• UniFi Protect 1.6.1
USG 4.4.22.5086045 •USG PRO 4.4.22.5086057
USW-48 4.0.14.9736 • USW-8 4.0.14.9736
UAP-AC-PRO 4.0.14.9736• UAP-AC-PRO 4.0.14.9736
UVC-G3-Flex 4.9.25

View solution in original post

Member
Posts: 114
Registered: ‎01-26-2017
Kudos: 6
Solutions: 3

Re: USG Pro, USW48, chaos after power up!!!! (repeatable) - Any clues ? [LONG]

There was no solution here.

 

Ubiquiti stated its an issue with my LAN hosts.

 

I'm afraid this is something which is beyond our scope of support as it is quite evident that it is your LAN hosts which are causing the issue in your network. We offer support for all UBNT devices. We will do our best to ensure your Ubiquiti product is working in accordance with our warranty, but we may not be able to resolve problems or answer questions when third-party devices are involved. However, looking at your findings and the steps that you have performed already, unfortunately, there isn't much we can do from our side.

I have since removed the LAN hosts from Ubiquiti network equipment and connected back to HPE an Cisco network switches.

 

I've since tested these LAN hosts on other Ubiquiti network equipment (4 different devices) and the same issue occurs with other networks!!!

UCK-G2-PLUS v.0.8.12•UniFi Network 5.9.29-11384-1• UniFi Protect 1.6.1
USG 4.4.22.5086045 •USG PRO 4.4.22.5086057
USW-48 4.0.14.9736 • USW-8 4.0.14.9736
UAP-AC-PRO 4.0.14.9736• UAP-AC-PRO 4.0.14.9736
UVC-G3-Flex 4.9.25

View solution in original post


All Replies
New Member
Posts: 17
Registered: ‎08-22-2018
Kudos: 2

Re: USG Pro, USW48, chaos after power up!!!! (repeatable) - Any clues ? [LONG]

Does your unifi server have DHCP server enabled? sounds like it does.

 

I would install unifi on Ubuntu server (most people/sites state unifi works better for them on Ubuntu - i would guess mostly because you can control updates much easier than on windows, or perhaps its the JRE that they use).

 

You definalty have to find out what server is also issuing DHCP ip addresses.  There are tools out there to help with this if its not the Unifi Windows machine.

Regular Member
Posts: 543
Registered: ‎09-13-2017
Kudos: 96
Solutions: 33

Re: USG Pro, USW48, chaos after power up!!!! (repeatable) - Any clues ? [LONG]

Two things to look at.

 

1) USW-48 is connected to USW-8 via 4 trunk ports. Have you actually setup  port "aggregate" on both switches or are they all just connected and will cause a major network loop.

 

2) you have another DHCP service running on something. You need to locate it.

As an interim setup DHCP guarding on your UniFI LAN(s), but this other DHCP is still going to be there so you need to locate it.

Setup wireshark on a laptop or something, plug into network. Setup a port mirror with the port that links your switch to USG and filter for DHCP. You should see traffic to/from the USG with DHCP requests and responses, but also the rogue DHCP server. Get its IP and shut down the DHCP service.

Member
Posts: 114
Registered: ‎01-26-2017
Kudos: 6
Solutions: 3

Re: USG Pro, USW48, chaos after power up!!!! (repeatable) - Any clues ? [LONG]

@marcusone1

 

"Does your unifi server have DHCP server enabled? sounds like it does."

 

 

 

"I would install unifi on Ubuntu server (most people/sites state unifi works better for them on Ubuntu - i would guess mostly because you can control updates much easier than on windows, or perhaps its the JRE that they use)."

 

Does not solve my issue, in the future due to virtual machines using network storage (which use the Ubq switches), will be either dumping Ubquiti investment or purchasing Cloud Key 2.

 

"You definalty have to find out what server is also issuing DHCP ip addresses.  There are tools out there to help with this if its not the Unifi Windows machine."

 

Turn off Unif Controller problem resolve it self.

 

 

 

 

 

UCK-G2-PLUS v.0.8.12•UniFi Network 5.9.29-11384-1• UniFi Protect 1.6.1
USG 4.4.22.5086045 •USG PRO 4.4.22.5086057
USW-48 4.0.14.9736 • USW-8 4.0.14.9736
UAP-AC-PRO 4.0.14.9736• UAP-AC-PRO 4.0.14.9736
UVC-G3-Flex 4.9.25
Member
Posts: 114
Registered: ‎01-26-2017
Kudos: 6
Solutions: 3

Re: USG Pro, USW48, chaos after power up!!!! (repeatable) - Any clues ? [LONG]

"Two things to look at.



1) USW-48 is connected to USW-8 via 4 trunk ports. Have you actually setup port "aggregate" on both switches or are they all just connected and will cause a major network loop.



2) you have another DHCP service running on something. You need to locate it.

As an interim setup DHCP guarding on your UniFI LAN(s), but this other DHCP is still going to be there so you need to locate it.

Setup wireshark on a laptop or something, plug into network. Setup a port mirror with the port that links your switch to USG and filter for DHCP. You should see traffic to/from the USG with DHCP requests and responses, but also the rogue DHCP server. Get its IP and shut down the DHCP service."

The problem was there before we added the USW-8.

The switch has been added as a 4 trunk, and "aggregated" on both switches.

So I do not believe this is the issue.

There are no other DHCP servers on the LAN, except the DHCP server serving IP Addresses which is a Windows Server. <--- which is the DHCP server for the LAN.

controller was configured for DHCP on 192 and 10, and ALL works.....happily

but this power up after down, is nonsense....

Again, after shutting down the Unfi Controller Server, all returned to normal.

In the example 10.255.255.1 is not in any DHCP server on the LAN.

(even when the controller was configured)

DHCP on controller is - 10.0.30.1/24 and 192.168.10.254/2

DHCP IP Addresses being issued is not those....

UCK-G2-PLUS v.0.8.12•UniFi Network 5.9.29-11384-1• UniFi Protect 1.6.1
USG 4.4.22.5086045 •USG PRO 4.4.22.5086057
USW-48 4.0.14.9736 • USW-8 4.0.14.9736
UAP-AC-PRO 4.0.14.9736• UAP-AC-PRO 4.0.14.9736
UVC-G3-Flex 4.9.25
Member
Posts: 114
Registered: ‎01-26-2017
Kudos: 6
Solutions: 3

Re: USG Pro, USW48, chaos after power up!!!! (repeatable) - Any clues ? [LONG]

[ Edited ]

is DHCP going to cause this....surely not ?????

 

and it's not DHCP

 

UCK-G2-PLUS v.0.8.12•UniFi Network 5.9.29-11384-1• UniFi Protect 1.6.1
USG 4.4.22.5086045 •USG PRO 4.4.22.5086057
USW-48 4.0.14.9736 • USW-8 4.0.14.9736
UAP-AC-PRO 4.0.14.9736• UAP-AC-PRO 4.0.14.9736
UVC-G3-Flex 4.9.25
Regular Member
Posts: 543
Registered: ‎09-13-2017
Kudos: 96
Solutions: 33

Re: USG Pro, USW48, chaos after power up!!!! (repeatable) - Any clues ? [LONG]

[ Edited ]

The windows server is your DHCP server?

What is your LAN setup on your controller which would by default setup DHCP on your USG?

 

The video to me suggests broadcast storm aka network loop.

 

 

Member
Posts: 114
Registered: ‎01-26-2017
Kudos: 6
Solutions: 3

Re: USG Pro, USW48, chaos after power up!!!! (repeatable) - Any clues ? [LONG]

[ Edited ]

all disabled

 

we do not have a need to use USG DHCP on main LAN.

 

But it is like that at initial startup USG/Controller turns on DHCP before DHCP is available on the Main LAN and causes chaos...

 

BUT what the hell is going on with the switch ? this is not correct....

 

high ping times, slow responses, it's like a broasdcast storm affecting all ports on the switch....

 

But if this was true, if we turn off DHCP Server on Windows Server, we should be able to get IP Addresses from 192 and 10 from something and we DO NOT!

 

It's only this Power on Sequence which causes issues.

 

 

 

 

UCK-G2-PLUS v.0.8.12•UniFi Network 5.9.29-11384-1• UniFi Protect 1.6.1
USG 4.4.22.5086045 •USG PRO 4.4.22.5086057
USW-48 4.0.14.9736 • USW-8 4.0.14.9736
UAP-AC-PRO 4.0.14.9736• UAP-AC-PRO 4.0.14.9736
UVC-G3-Flex 4.9.25
Member
Posts: 114
Registered: ‎01-26-2017
Kudos: 6
Solutions: 3

Re: USG Pro, USW48, chaos after power up!!!! (repeatable) - Any clues ? [LONG]


@manroweb wrote:

The windows server is your DHCP server?

What is your LAN setup on your controller which would by default setup DHCP on your USG?

 

The video to me suggests broadcast storm aka network loop.

 

 


Yes, Windows Server is DHCP server.

 

LAN setup on controller matches our LAN, e.g. our Corporate LAN matches our LAN and Windows DHCP server, but this is DISABLED.

 

and the IP Addresses being issued are not these, and do not match anything in Controller!

 

and it was disabled, and still issuing 192 and 10 IP Addresses from something...

 

and stopped when Unfi Controller was powered off.

UCK-G2-PLUS v.0.8.12•UniFi Network 5.9.29-11384-1• UniFi Protect 1.6.1
USG 4.4.22.5086045 •USG PRO 4.4.22.5086057
USW-48 4.0.14.9736 • USW-8 4.0.14.9736
UAP-AC-PRO 4.0.14.9736• UAP-AC-PRO 4.0.14.9736
UVC-G3-Flex 4.9.25
Member
Posts: 114
Registered: ‎01-26-2017
Kudos: 6
Solutions: 3

Re: USG Pro, USW48, chaos after power up!!!! (repeatable) - Any clues ? [LONG]


@manroweb wrote:

 

 

The video to me suggests broadcast storm aka network loop.

 

 


definately agree on that...?????

 

If it was DHCP it should not cause a network loop ???

 

but shutting down controller, fixed it....

 

 

UCK-G2-PLUS v.0.8.12•UniFi Network 5.9.29-11384-1• UniFi Protect 1.6.1
USG 4.4.22.5086045 •USG PRO 4.4.22.5086057
USW-48 4.0.14.9736 • USW-8 4.0.14.9736
UAP-AC-PRO 4.0.14.9736• UAP-AC-PRO 4.0.14.9736
UVC-G3-Flex 4.9.25
New Member
Posts: 17
Registered: ‎08-22-2018
Kudos: 2

Re: USG Pro, USW48, chaos after power up!!!! (repeatable) - Any clues ? [LONG]

very strange that controller off (I assume you mean the controller software) stops the issue.

 

Can you then bring up the controller software again and its fine?

 

Perhaps re-install controller (why I mentioned Ubuntu if you go down that path... a big pain, as i wouldn't use a settings backup, in case that's the issue).

 

Have you contacted support and sent longs to them?

Member
Posts: 114
Registered: ‎01-26-2017
Kudos: 6
Solutions: 3

Re: USG Pro, USW48, chaos after power up!!!! (repeatable) - Any clues ? [LONG]

Yes, I actually power off the Windows Server which controller software is installed on.

30 minutes later I powered up the server, and controller and all is well again.

It's been the same in all versions of the controller software and now on 5.2.9.

I've not contact support, I was waiting for any ideas, but it will esclate this now, as today was the final straw, it's bad enough having a power cut in the business, but today 3 hours trying to fix....

when it all just starts working again...
UCK-G2-PLUS v.0.8.12•UniFi Network 5.9.29-11384-1• UniFi Protect 1.6.1
USG 4.4.22.5086045 •USG PRO 4.4.22.5086057
USW-48 4.0.14.9736 • USW-8 4.0.14.9736
UAP-AC-PRO 4.0.14.9736• UAP-AC-PRO 4.0.14.9736
UVC-G3-Flex 4.9.25
Member
Posts: 114
Registered: ‎01-26-2017
Kudos: 6
Solutions: 3

Re: USG Pro, USW48, chaos after power up!!!! (repeatable) - Any clues ? [LONG]

[ Edited ]

just digging through stats and logs... and found this....

 

you can see when the issue was...

 

this is the USW-48 3.9.54.9373

 

 

UCK-G2-PLUS v.0.8.12•UniFi Network 5.9.29-11384-1• UniFi Protect 1.6.1
USG 4.4.22.5086045 •USG PRO 4.4.22.5086057
USW-48 4.0.14.9736 • USW-8 4.0.14.9736
UAP-AC-PRO 4.0.14.9736• UAP-AC-PRO 4.0.14.9736
UVC-G3-Flex 4.9.25
2018-12-08 19_51_53-usw48.png
Member
Posts: 114
Registered: ‎01-26-2017
Kudos: 6
Solutions: 3

Re: USG Pro, USW48, chaos after power up!!!! (repeatable) - Any clues ? [LONG]

USW-8 3.9.54.9373

 

2018-12-08 19_55_29-UniFi.png

UCK-G2-PLUS v.0.8.12•UniFi Network 5.9.29-11384-1• UniFi Protect 1.6.1
USG 4.4.22.5086045 •USG PRO 4.4.22.5086057
USW-48 4.0.14.9736 • USW-8 4.0.14.9736
UAP-AC-PRO 4.0.14.9736• UAP-AC-PRO 4.0.14.9736
UVC-G3-Flex 4.9.25
New Member
Posts: 17
Registered: ‎08-22-2018
Kudos: 2

Re: USG Pro, USW48, chaos after power up!!!! (repeatable) - Any clues ? [LONG]

Do you run anything else on the windows server?

 

If you don't; 

I would move the controller to a Linux machine (and even try just moving your configuration, there are lots of guides on doing this that are rather easy to follow)... What I did and haven't had a single adoption or other issue since doing so.

 

Or pickup one of the new Gen 2 Keys, they sound awesome Man Happy

 

 

Member
Posts: 114
Registered: ‎01-26-2017
Kudos: 6
Solutions: 3

Re: USG Pro, USW48, chaos after power up!!!! (repeatable) - Any clues ? [LONG]

2018-12-08 19_58_52-UniFi.png2018-12-08 19_58_17-UniFi.png2018-12-08 19_57_39-UniFi.pngit's not my imagination, these are from USG, and APs, so something weird is going on causing CPU to rocket... and logging is lost... and hence my outage so here's proof.

UCK-G2-PLUS v.0.8.12•UniFi Network 5.9.29-11384-1• UniFi Protect 1.6.1
USG 4.4.22.5086045 •USG PRO 4.4.22.5086057
USW-48 4.0.14.9736 • USW-8 4.0.14.9736
UAP-AC-PRO 4.0.14.9736• UAP-AC-PRO 4.0.14.9736
UVC-G3-Flex 4.9.25
Member
Posts: 114
Registered: ‎01-26-2017
Kudos: 6
Solutions: 3

Re: USG Pro, USW48, chaos after power up!!!! (repeatable) - Any clues ? [LONG]

DHCP is causing this ???????

all APs, two USW switches, and USG all to go CPU high ?????
UCK-G2-PLUS v.0.8.12•UniFi Network 5.9.29-11384-1• UniFi Protect 1.6.1
USG 4.4.22.5086045 •USG PRO 4.4.22.5086057
USW-48 4.0.14.9736 • USW-8 4.0.14.9736
UAP-AC-PRO 4.0.14.9736• UAP-AC-PRO 4.0.14.9736
UVC-G3-Flex 4.9.25
Member
Posts: 114
Registered: ‎01-26-2017
Kudos: 6
Solutions: 3

Re: USG Pro, USW48, chaos after power up!!!! (repeatable) - Any clues ? [LONG]


@marcusone1 wrote:

Do you run anything else on the windows server?

 

If you don't; 

I would move the controller to a Linux machine (and even try just moving your configuration, there are lots of guides on doing this that are rather easy to follow)... What I did and haven't had a single adoption or other issue since doing so.

 

Or pickup one of the new Gen 2 Keys, they sound awesome Man Happy

 

 


the server is specifically deployed for years to run Unfi Controller, no other software runs on it.

 

Originally deployed to support 1 x AP, then added another AP - all fine.

 

then added a USG.

 

all started going a bit weird, after adding USW48....which replaced a HPE/Cisco switch (which worked fine). later added USW8 for PoE.

 

later upgraded USG to USG Pro.

UCK-G2-PLUS v.0.8.12•UniFi Network 5.9.29-11384-1• UniFi Protect 1.6.1
USG 4.4.22.5086045 •USG PRO 4.4.22.5086057
USW-48 4.0.14.9736 • USW-8 4.0.14.9736
UAP-AC-PRO 4.0.14.9736• UAP-AC-PRO 4.0.14.9736
UVC-G3-Flex 4.9.25
Member
Posts: 114
Registered: ‎01-26-2017
Kudos: 6
Solutions: 3

Re: USG Pro, USW48, chaos after power up!!!! (repeatable) - Any clues ? [LONG]

no controller logs present.

and any syslogs, would not get to destination, as syslog server could not be reached.
UCK-G2-PLUS v.0.8.12•UniFi Network 5.9.29-11384-1• UniFi Protect 1.6.1
USG 4.4.22.5086045 •USG PRO 4.4.22.5086057
USW-48 4.0.14.9736 • USW-8 4.0.14.9736
UAP-AC-PRO 4.0.14.9736• UAP-AC-PRO 4.0.14.9736
UVC-G3-Flex 4.9.25
Regular Member
Posts: 543
Registered: ‎09-13-2017
Kudos: 96
Solutions: 33

Re: USG Pro, USW48, chaos after power up!!!! (repeatable) - Any clues ? [LONG]

But it is like that at initial startup USG/Controller turns on DHCP before DHCP is available on the Main LAN and causes chaos...

 

So turn of DHCP on the USG. Set it as DHCP relay and point to the Windows Server.

You have 2 DHCP servers running.

Reply