Upcoming Maintenance Alert:

The UBNT Community will be upgraded at 5pm MDT on April 25th. During this time the community forums will be set to read-only status.

Learn more

×
Reply
Highlighted
Emerging Member
Posts: 62
Registered: ‎12-21-2016
Kudos: 13
Solutions: 2

EdgeSwitches have destroyed two new Dell systems, up in smoke! WTF is going on?

[ Edited ]

I now have two brand-new out of the box EdgeSwitches that have been destroyed. The first one I posted about yesterday:

 

https://community.ubnt.com/t5/EdgeSwitch/New-ES-24-250W-up-in-smoke/m-p/1934306#M12777

 

The second one blew up today (smoked), though it's still online and I'm collecting information from it now.

 

It destroyed a brand new Dell server and at least one Dell workstation.

 

It seems like it's turning the PoE power on for some reason when it shouldn't be.

 

Both switches bought from Amazon in the USA. There is nothing unusual about our setup. Just new PCs and servers for a new business location we were setting up, and we decided to give Ubiquiti a try on switches.

 

More information coming soon after I've collected it.

 

 UPDATE: Changed word "servers" to "systems" in title. One server and one PC are verified casualties so far, not two servers.

SuperUser
Posts: 11,501
Registered: ‎12-08-2008
Kudos: 8331
Solutions: 525
Contributions: 1

Re: EdgeSwitches have destroyed two new Dell servers, up in smoke! WTF is going on?

Not saying that this is the issue, but I always turn off PoE on any ports that don't need it.   I have seen a lot of vendor's PoE switches put power on devices that aren't PoE at all -  the underlying design of how PoE switches decide to turn the power on is very simple and can be fooled by any device that starts to pull power, even bad wiring can trigger it.   Better safe than sorry...

JIm

" How can anyone trust Scientists? If new evidence comes along, they change their minds! " Politician's joke (sort of...)

"Humans are allergic to change..They love to say, ‘We’ve always done it this way.’ I try to fight that. "Admiral Grace Hopper, USN, Computer Scientist
Emerging Member
Posts: 62
Registered: ‎12-21-2016
Kudos: 13
Solutions: 2

Re: EdgeSwitches have destroyed two new Dell servers, up in smoke! WTF is going on?

I'm still working on this. We have ordered some new switches (from another vendor) and still have to get our project back on track, so I need to deal with that first.

 

Here's the key evidence. These syslog messages were recorded on the swtich right at the time a Dell workstation was being pluged into it, which caused it to blow smoke. This destroyed the Dell PC. I was logged into the switch remotely at the time and got this over ssh:

 

<14> May 18 14:17:25 switchname-1 BOXSERV[boxs Req]: boxs.c(691) 214 %% Error 17 occurred reading thermal sensor 2 data
<14> May 18 14:17:23 switchname-1 BOXSERV[boxs Req]: boxs.c(691) 213 %% Error 17 occurred reading thermal sensor 7 data
<14> May 18 14:17:23 switchname-1 BOXSERV[boxs Req]: boxs.c(691) 212 %% Error 17 occurred reading thermal sensor 6 data
<14> May 18 14:17:23 switchname-1 BOXSERV[boxs Req]: boxs.c(691) 211 %% Error 17 occurred reading thermal sensor 5 data
<14> May 18 14:17:23 switchname-1 BOXSERV[boxs Req]: boxs.c(691) 210 %% Error 17 occurred reading thermal sensor 4 data
<14> May 18 14:17:23 switchname-1 BOXSERV[boxs Req]: boxs.c(691) 209 %% Error 17 occurred reading thermal sensor 3 data
<14> May 18 14:17:23 switchname-1 BOXSERV[boxs Req]: boxs.c(691) 208 %% Error 17 occurred reading thermal sensor 2 data
<14> May 18 14:17:21 switchname-1 BOXSERV[boxs Req]: boxs.c(691) 207 %% Error 17 occurred reading thermal sensor 7 data
<14> May 18 14:17:21 switchname-1 BOXSERV[boxs Req]: boxs.c(691) 206 %% Error 17 occurred reading thermal sensor 6 data
<14> May 18 14:17:21 switchname-1 BOXSERV[boxs Req]: boxs.c(691) 205 %% Error 17 occurred reading thermal sensor 5 data
<14> May 18 14:17:21 switchname-1 BOXSERV[boxs Req]: boxs.c(691) 204 %% Error 17 occurred reading thermal sensor 4 data
<14> May 18 14:17:21 switchname-1 BOXSERV[boxs Req]: boxs.c(691) 203 %% Error 17 occurred reading thermal sensor 3 data
<14> May 18 14:17:21 switchname-1 BOXSERV[boxs Req]: boxs.c(691) 202 %% Error 17 occurred reading thermal sensor 2 data
<14> May 18 14:17:19 switchname-1 BOXSERV[boxs Req]: boxs.c(691) 201 %% Error 17 occurred reading thermal sensor 7 data
<14> May 18 14:17:19 switchname-1 BOXSERV[boxs Req]: boxs.c(691) 200 %% Error 17 occurred reading thermal sensor 6 data
<14> May 18 14:17:19 switchname-1 BOXSERV[boxs Req]: boxs.c(691) 199 %% Error 17 occurred reading thermal sensor 5 data
<14> May 18 14:17:19 switchname-1 BOXSERV[boxs Req]: boxs.c(691) 198 %% Error 17 occurred reading thermal sensor 4 data
<14> May 18 14:17:19 switchname-1 BOXSERV[boxs Req]: boxs.c(691) 197 %% Error 17 occurred reading thermal sensor 3 data
<14> May 18 14:17:19 switchname-1 BOXSERV[boxs Req]: boxs.c(691) 196 %% Error 17 occurred reading thermal sensor 2 data
<12> May 18 14:17:18 switchname-1 UBNT_POE[interrupt servi]: ubnt_poe_common.c(4735) 195 %% UBNT POE Down!!
<12> May 18 14:17:18 switchname-1 UBNT_POE[interrupt servi]: ubnt_poe_common.c(4735) 194 %% UBNT POE Down!!
Emerging Member
Posts: 62
Registered: ‎12-21-2016
Kudos: 13
Solutions: 2

Re: EdgeSwitches have destroyed two new Dell servers, up in smoke! WTF is going on?

[ Edited ]

So here's the story:


The company I'm working for is setting up a new office location, so we bought some new servers and workstations from Dell, two Ubiquiti EdgeSwitch 24-Port 250W switches, an ERPOE-5 EdgeRouter, and all the other miscellaneous stuff you need for a new business.

 

These were our first two EdgeSwitches (and our last), so they were new to us. We have a couple of EdgeRouters already that we've been using for a few months as we've been trying out more and more Ubiquiti equipment.


I work remote -- I've never touched any of this gear or even set eyes on it. I have an on-site Windows Admin/Server guy who I've been my hands-on to set things physically set up for me. I configured everything from console ports and ssh remotely.


The switches were bought on Amazon in the USA and I had them set up for about two weeks in a lab-style environment. During that time I had one crash on the 1.0.1 version of the OS that required a hard-reboot to fix, but after upgrading they appeared to be working normally as far as hardware goes. Also, we had multiple other devices plugged into these switches without any trouble, including Cisco switches, a Cisco ASA, an ERPOE-5 EdgeRouter, and a few other things.


Yesterday we moved the switches and EdgeRouter into a new room along with all of the Dell computers so that we could set up a staging environment: Configure everything, set up the Windows domain, update the PCs and configure them, et-cetera.


Yesterday the first switch blew up. I'm told it made a huge cloud of smoke, made popping/arcing sounds, and completely died. We figured it was just bad luck. We now know that it destroyed a Dell PowerEdge T330 server in the process.


I reconfigured the second switch today to take over for the first one that died and we started working again when it too blew smoke, though it didn't completely die like the first one. That's probably because my Windows Admin guy pulled the network jack of the Dell PC he had just plugged in when it started smoking. The Dell PC is a OptiPlex 7440 AIO. The PC is still working, but we suspect that the network interface may have been destroyed. We will find out for sure tomorrow when we test it on a new switch.


It's possible that a third Dell PC may have also been damaged but I'm not sure yet. I know it was plugged in at some point. We will fully asses the damage tomorrow when we bring another switch in.


When the second switch blew up today, I was on the phone live with my Windows guy trying to figure out why the PC could not get a DHCP lease on the network. I did some troubleshooting and discovered via "show mac-addr-table" that there no layer 2 addresses were showing up on the port, even though interface status showed the port as up.


I was testing some different configurations involving the "switchport mode access/trunk" commands and that's when it blew up.


At that point I collected some info from the command line and then we just unplugged everything. Both switches will be shipped back to Amazon.


Here's the info on the second switch:


MAC ID 1629G 802AA89F903B
Serial # 802AA89F903B

 

Emerging Member
Posts: 94
Registered: ‎12-09-2014
Kudos: 17
Solutions: 9

Re: EdgeSwitches have destroyed two new Dell servers, up in smoke! WTF is going on?

Just curious..but did both of these switches die when connected to the exact same Dell server?  Because that's how I'm reading it.

Emerging Member
Posts: 62
Registered: ‎12-21-2016
Kudos: 13
Solutions: 2

Re: EdgeSwitches have destroyed two new Dell servers, up in smoke! WTF is going on?

 

I attached a show tech output along with some poe info. Nothing interesting really as far as I can see. I X-ed out some private info but nothing important.

 

I had my on-site guy take a bunch of pictures, but there's nothing really interesting or worth posting.

 

Emerging Member
Posts: 62
Registered: ‎12-21-2016
Kudos: 13
Solutions: 2

Re: EdgeSwitches have destroyed two new Dell servers, up in smoke! WTF is going on?

 

The first switch died when it blew up the first server. It's noteworthy that the server had been plugged in and working for awhile. I'm not exactly clear about what triggered that event. I get the impression that one of the workstations may have been plugged in, triggering the event. I've been told that workstation has been moved to the side and we will test it tomorrow. The server is 100% non-booting dead hardware destroyed.

 

The second switch died when one of the workstation/PCs was being moved around between ports, while I was troubleshooing an issue live with the guy on site. So, it was working for awhile and then stopped working. We were moving it between ports 15 and 16 on the switch while I was testing different port configurations.

 

So, to answer your question directly, no, the same server was not involved. Also, I'm told that it was two different workstations involved, which is why I'm not sure yet how many PCs have been fried.

 

One additional detail that I just remembered: The on-site guy mentioned multiple times that plugging things into the switch made him nervous, because the fans would rev up and down when he did that, and he mentioned a burning smell when that would happen.

 

Now that I think about it, I have that call recoreded on my phone... 

Member
Posts: 198
Registered: ‎02-05-2017
Kudos: 37
Solutions: 3

Re: EdgeSwitches have destroyed two new Dell servers, up in smoke! WTF is going on?

[ Edited ]

jshel wrote:

The switches were bought on Amazon in the USA. . .


. . .Yesterday the first switch blew up. I'm told it made a huge cloud of smoke, made popping/arcing sounds, and completely died. We figured it was just bad luck. We now know that it destroyed a Dell PowerEdge T330 server in the process. . .

 

. . . The on-site guy mentioned multiple times that plugging things into the switch made him nervous, because the fans would rev up and down when he did that, and he mentioned a burning smell when that would happen.


And where are you? Where did the Dell come from? The first thing I would do is break out a multimeter and start testing the electricity supply, polarity, ground continuity, check for voltage on the shield of LAN plugs, the basics.

 

Having one of these products go bad due to manufacturing defects is bad luck. Having two go bad in a row, statistically near impossible. With what you describe, it sounds like you're dealing with a serious electrical issue, not defective equipment. You might want to tell your on site guy to start wearing rubber gloves, lest he go up in smoke too.

SuperUser
Posts: 3,541
Registered: ‎10-12-2010
Kudos: 1939
Solutions: 28

Re: EdgeSwitches have destroyed two new Dell servers, up in smoke! WTF is going on?

What are the date codes on the ES's?  Are you using shielded patch cables by chance? 

UBRSS, UBWA, UEWA - Ubiquiti Certified Trainer
Emerging Member
Posts: 62
Registered: ‎12-21-2016
Kudos: 13
Solutions: 2

Re: EdgeSwitches have destroyed two new Dell servers, up in smoke! WTF is going on?

[ Edited ]

PatrickShaw wrote:

And where are you? Where did the Dell come from? The first thing I would do is break out a multimeter and start testing the electricity supply, polarity, ground continuity, check for voltage on the shield of LAN plugs, the basics.

 

Having one of these products go bad due to manufacturing defects is bad luck. Having two go bad in a row, statistically near impossible. With what you describe, it sounds like you're dealing with a serious electrical issue, not defective equipment. You might want to tell your on site guy to start wearing rubber gloves, lest he go up in smoke too.


This all happened in the USA. I just included that because I know there are a lot of international people here on the forum too, but I can see how you might have thought I was implying this incident happened somewhere else.

 

No, I'm not going to do air and water samples or break out a shovel to do a geological survey of the area to satisfy your need to white-knight for Skub, your favorite brand product.

 

I already posted proof that PoE engaged on the switch when it should not have and there is no evidence that I'm aware of to make me suspect there is any kind of issue with the site power or other envrionmental problem.

 

This sounds exactly like a manufacturing defect: where multiple products in a row come off the assembly line all broken in exactly the same way. It does happen from time to time. That's why I posted this. If there is a bad batch of switches out there (and we already know that has happened before, multiple times), other people probably want to know about it.

 

New Member
Posts: 2
Registered: ‎02-06-2017

Re: EdgeSwitches have destroyed two new Dell servers, up in smoke! WTF is going on?

If ya don't mind sharing, who was the seller of these switches on Amazon?
I ask because I just ordered and received 2 of the exact same switch from Amazon this week. The switches were shipped by Amazon, but sold by a 3rd party. I'd like to know if it's the same seller as your switches.

You've got me a bit nervous cause I've go one of them up and running on my test/config bench at the office right now.

Fingers crossed, knock on wood... please don't burn down the office!!
Emerging Member
Posts: 62
Registered: ‎12-21-2016
Kudos: 13
Solutions: 2

Re: EdgeSwitches have destroyed two new Dell servers, up in smoke! WTF is going on?


Magician wrote:

What are the date codes on the ES's?  Are you using shielded patch cables by chance? 


Good question on the date codes. Will those be on the bottom maybe? I'll have my on-site guy take a picture. Maybe he missed that. The original packing boxes unfortunately were already discarded.

 

The patch cables were all unshielded cat5e pre-mades bought from CDW. Nothing special or weird. I did have my guy visually inspect the cable, take a picture of it, and it's now plugged into something else working fine.

Member
Posts: 198
Registered: ‎02-05-2017
Kudos: 37
Solutions: 3

Re: EdgeSwitches have destroyed two new Dell servers, up in smoke! WTF is going on?

[ Edited ]

 

No, I'm not going to do air and water samples or break out a shovel to do a geological survey of the area to satisfy your need white-knight Skub, your favorite brand product.

 


Well excuse me but that's not what I said. I understand your frustration, I just suggested you eliminate the obvious before writing off the brand out of hand, but If you're not willing to do basic site competency checks before anything else (really, it just involves sticking a probe in a socket) and you're going to be a jerk on the forums with genuinely helpful posters, then I suggest you and your on-site guy's competency can legitimately be questioned too.

Emerging Member
Posts: 62
Registered: ‎12-21-2016
Kudos: 13
Solutions: 2

Re: EdgeSwitches have destroyed two new Dell servers, up in smoke! WTF is going on?


BHKIT wrote:
If ya don't mind sharing, who was the seller of these switches on Amazon?
I ask because I just ordered and received 2 of the exact same switch from Amazon this week. The switches were shipped by Amazon, but sold by a 3rd party. I'd like to know if it's the same seller as your switches.

You've got me a bit nervous cause I've go one of them up and running on my test/config bench at the office right now.

Fingers crossed, knock on wood... please don't burn down the office!!

Sold by: Unifying Solution

 

I will try and get the date codes tomorrow as requested by that other guy.

 

Here are the MACs and serial #s:

 

MAC ID 1629G 802AA89F8DD7
Serial # 802AA89F8DD7

 

MAC ID 1629G 802AA89F903B
Serial # 802AA89F903B

 

Hope you have more luck than we did! I had high hopes for Ubi switches. Maybe I'll try them again in a few years.

 

New Member
Posts: 2
Registered: ‎02-06-2017

Re: EdgeSwitches have destroyed two new Dell systems, up in smoke! WTF is going on?

Well ours are from a different seller, but I'm not gonna lie... I'm still a bit nervous.

New Member
Posts: 21
Registered: ‎04-06-2011
Kudos: 1

Re: EdgeSwitches have destroyed two new Dell systems, up in smoke! WTF is going on?

There must be something else going on here. Ethernet ports typically coupled with small transformers (google ethernet magnetics), so any DC voltage really shouldn't blow the whole server. I guess you could pop the transformer if enough current flow, causing your ethernet to die, but i wouldn't think so.

 

Looking up some ethernet transformer specs, i found the following.

 

Isolation is necessary to meet IEEE 802.3 AC and DC isolation specifications for cabled configurations.

IEEE 802.3 isolation requirements are described in section 14.3.1.1 of the specification, and include the

ability to sustain cable faults to 1500V 50 or 60 Hz or 2250Vdc voltage levels for 60 seconds.

 

I'm not saying that the switches were not at fault, but there would have to be a serious issue whereby you end up with some AC voltage on the ethernet line. DC voltage doesn't transfer through transformers (Apart from a short spike when connected). I've personally  connected a canopy POE injector the wrong way (accidently) and presented 24V DC directly to my laptop on at least a couple of occasions and had no issue.

 

I'd really be checking the power.

 

Were all these devices on the same circuit? Perhaps theres a serious ground difference between them? I have seen ethernet links where the differences in ground potential would eventually pop the switch on more than one occasion.  The ethernet magnetics really should make this a mute point, but again... I've seen it happen. Replace with fiber.... Never failed again.

 

I'd be really interested to see what actually caused this. I really think that further investigation is required. If its a switch fault, im sure we'll see more in the near future.

 

 

Ubiquiti Employee
Posts: 2,082
Registered: ‎10-05-2015
Kudos: 631
Solutions: 186

Re: EdgeSwitches have destroyed two new Dell systems, up in smoke! WTF is going on?

@jshel

@shansted is right. I am not aware of any way PoE voltage on an Ethernet cable can cause a PC/Server to not boot as you described. There are some isolated cases where a range of switches outputs 24V power on boot for a short amount of time. If the end device has the proper Ethernet surge protection (I would imagine a Dell PC/Server would) this would not affect the device, even the Ethernet port. This has already been addressed and fixed with a hardware change on our end.

 

If interested, I would like to get those switches you have back for analysis. You can submit an RMA, PM me the RMA#, and I will make sure these switches are replaced and sent back. Sorry for the inconvenience. 

 

I have heard of reports of Amazon sellers selling used, non-Ubiquiti refurbished devices. I hope this is not the case. You can always purchased from one of our distributors or resellers listed here, buy direct from our store, or from Amazon if it is shipped and sold by Amazon. This way you can be sure you are getting fresh, new stock.

Member
Posts: 198
Registered: ‎02-05-2017
Kudos: 37
Solutions: 3

Re: EdgeSwitches have destroyed two new Dell systems, up in smoke! WTF is going on?

[ Edited ]

shansted wrote:

There must be something else going on here. . .

 

I'm not saying that the switches were not at fault, but there would have to be a serious issue whereby you end up with some AC voltage on the ethernet line. . .

 

Were all these devices on the same circuit? Perhaps theres a serious ground difference between them? I have seen ethernet links where the differences in ground potential would eventually pop the switch on more than one occasion.

 


This is exactly what I thought when reading. Frying an ethernet/LAN port is one thing, completely destroying a server is a different level. If this is NOT a more serious electrical fault, then the OP has a talent for hyperbole.

Emerging Member
Posts: 62
Registered: ‎12-21-2016
Kudos: 13
Solutions: 2

Re: EdgeSwitches have destroyed two new Dell systems, up in smoke! WTF is going on?


shansted wrote:

 

I'm not saying that the switches were not at fault, but there would have to be a serious issue whereby you end up with some AC voltage on the ethernet line. DC voltage doesn't transfer through transformers (Apart from a short spike when connected). I've personally  connected a canopy POE injector the wrong way (accidently) and presented 24V DC directly to my laptop on at least a couple of occasions and had no issue.

 

I'd really be checking the power.

  


 

Yea, I wish I knew what caused this too. All I know is what you know, unfortunately. The switches were sub $1000 and are going back to Amazon for a refund. We had Dell replace the motherboard in the Server. It's just not worth our time to dig into this too deeply. Hopefully it just doesn't happen to anyone else. If it does, maybe they will find this thread.

 

The building is a oncology treatment center with lots of high-value equipment. There is no reason to suspect any power or external related issues. Moreover, the staging room has seen lots of other equipment in it that have never had a problem before. We plugged in some temporary Cisco SB 300 and Netgear switches today to take over the job until the new permanant switches come in and there have been no new problems.

 

Also you got your electrical filtering components backwords; capacitors block DC, inducers/coils/xformers block AC, not the other way around. Yes I have a background in electronics.

Emerging Member
Posts: 62
Registered: ‎12-21-2016
Kudos: 13
Solutions: 2

Re: EdgeSwitches have destroyed two new Dell systems, up in smoke! WTF is going on?


BHKIT wrote:
Well ours are from a different seller, but I'm not gonna lie... I'm still a bit nervous.


I wouldn't worry about it. Hopefully we just got a bad batch or something. You won't know until you find out!

Reply