06-13-2018 09:40 AM
Installed a UniFi system about two years ago at a customer location which included a 16 port UniFi PoE switch. All has been working very well with no issues until about a week ago.
Customer calls and says they have no network connectivity, so I look at the controller (on site cloud key) which is offline. Able to ping the modem, but not the gateway.
I go out to the site and see that nothing connected to the switch is working, but the router is blinking strangely on the LAN port. All components are connected to a battery backup with AVR, and no other components not connected to the switch are operating just fine.
I go ahead and disconnect all patch cables from the switch and then unplug and replug the switch into power. Nothing.
I swing in a replacement switch (non PoE UniFi 48) and nothing comes back online (cloud key now plugged into USB) no lights on the cloud key, and the LAN port on the USG just blinks in strange patterns (on off, off for a while, on for a second, etc)
I go ahead and factory reset the USG. Nothing after the reset. Attempt the same thing with the cloud key. Nothing.
I go ahead and install a new USG and Cloud key. Now I can see the network and get online with my laptop. Several ports with things like a surveillance system, a phone system, and two computers aren't doubg anything and thebnetwort connected fire/burglar alarm has been randomly beeping and chirping the siren.
I go ahead and try a new NIC card in the phone system. Works now. Install PCI-E NIC cards in the PCs and now they work. The devices with non-replaceable NICs are done.
To me it seems that the switch failed catastrophically and took out most Ethernet connected devices in the process. Gateway, Cloud Key, PBX, Camera system, two PCs, and a fire/burglar alarm.
No other devices connected to the same battery backup have any issues and the battery backup shows no recent events of surge or otherwise intervening in the utility power.
Anyone have any ideas? Thoughts?
06-13-2018 10:02 AM
I had something similar happen to me, but I was home to see it happen. Lightning hit a tree right outside my front door. The power never went off, and nothing connected to any of the power outlets suffered any damage due to it coming through the power lines. But, I lost all my network switches, and a SiliconDust HDHR Prime tuner. Lucky for me, the rest of the devices survived.
If you look closely, you'll see a dead squirrel on the ground to the left of the tree. We've jokingly said we didn't know if it was the power from the bolt that killed it, or the sheer fear causing a heart attack from the light and noise.
06-13-2018 12:49 PM
I second the lightning diagnosis. I have BTDT before.
Was at an antenna farm just off the Everglades years ago, pre-CAT3 networking. We'd get HIT bi-monthly. Phone system was usual victim, as well as the antenna elements that were burnt off on found on the ground.
Based on that experience, I never run copper between buildings; glass is your friend.
06-13-2018 01:25 PM
Just exploring all the possibilities. It may be possible that something happened somewhere else and took out the switch with it.
In all my years in IT, I've seen some crazy things. I was setting up a network at a central Texas police department in a very old building. I got all the cabling run (this was back when it was common to use 10base2 coax) and was just about to hook up the T to the NIC, had my arm leaning on the computer case and touched the T and got zapped. There were all sorts of wiring issues through the building, so the computers were grounding through the ethernet cable. I had to walk away and tell them I wasn't doing any more until they fixed the issues. They called the boss of the company and threw a fit. I told him I got shocked and that was the end of it. I'm sure had I hooked up the cable to the NIC it would have fried it or something else.