Highlighted
SuperUser
Posts: 14,507
Registered: ‎12-08-2008
Kudos: 11286
Solutions: 693
Contributions: 1

Prepare for the worst...

[ Edited ]

Thought I would put this out here as a warning to those who think the large Data Center or building you are in will keep everything running for you...

 

1 week ago about this time in the morning the power at our main facility (skyscraper in Downtown) went out.   No big problem - we have hours of battery backup there.   But then some non-critical things started going off line, and after contacting the building we went down there to find no AC power on the entire 36th floor of the building because of this

1670PanelSm.jpg

This is (was)  the switch cabinet for the entire 36th floor - they were working on it and something went wrong.   You can't see the inside, but virtually everything inside that open panel that's still in the "On" position is gone - literally vaporized all the components inside - the only things left were the 600A fuses for the floor circuit  that welded themselves in place so they couldn't even get them out.

1670RearLowerSm.jpg

This is the lower half of the rear of the cabinet - those are the main bus-bars for power for one of the three power risers in the building there, and they are not supposed to be that purple-grey color - everything there should be copper, shiny nickel or painted steel.   What you see (including coating the insulation on the cables) is the vaporized remains of the interior guts which deposited itself everywhere.  The white stuff you can see behind the buss bars is melted plastic shielding that flowed down from the switches above the blown out panel,    This panel is still hot with 480VAC at the time this was taken BTW...

1670PanelRearUpperSm.jpg

This is the upper half of the cabinet - you can see more melted plastic insulation and shield material here too.   The cabinet itself is 8 feet high - and you can see where the fire went out the front grate - that's exactly why they put those grates in there - to keep the pressure from exploding the cabinet like a bomb...

 

Fortunately the electrician working on it at the time wasn't badly hurt, but it knocked out all but a handful of power outlets on another circuit on the entire floor (and fortunately the lights were on yet another circuit) but we had to scramble to first power all the battery chargers and someAC only servers (for moniitoring, VoIP softswitch and some other things) on the remaining outlets with power, but that wasn't adequate to run some of our legacy Hosting system.   So we had to bring down 3 generators to run all that - fortunately the building elevators (all 24 of them) were on a different circuit too.   Once we got things stabilized, we found out that they would need to completely power down the building to remove the damaged switchgear and set up a temporary feed for the 36th floor the following night.

 

And this didn't just affect us -  on our floor there is microwave equipment for 3 of the 4 local Network TV stations, plus several large 2-way radio systems, another large wireless ISP, and the City's backup E-911 system.   Plus the building houses the Western Data Center for one of the largest Insurance Carriers in the US, so a lot of people were impacted.

 

The next evening we moved everything over to the generators, and the Utility cut the building power at 9:00 PM.   By a little after midnight they had things cleared away enough and a new temp panel in place for our floor and re-energized the building (which is always tricky with something of this size) and we could switch back over to main power.   The electricians had a crew of 20+ there just to get it all done as fast as possible, and they did a great job.   So that solved the immediate problem, but the permanent fix will take around a month of planning and getting the new switchgear in place, and we'll have to do the generator-dance again because it will take down the whole building again for hours while the put the new parts in place.

 

Moral of this story?   Just because you are in a large facility or Data Center it doesn't mean things can't go very wrong and impact you severely.   You need to plan for the wort case, and then think hard about what could go even worse.   And be ready for it.   Obviously there are things that are so catastrophic that you can't really prepare for them, but there were ISPs in New York after Hurricaine Sandy who's buildings were flooded and they literally had to hand-carry Diesel fuel up flights of stairs for days to keep things running, but they did it.   Just be prepared as best you can for when things like this happen.

Jim

 

" How can anyone trust Scientists? If new evidence comes along, they change their minds! " Politician's joke (sort of...)
"Humans are allergic to change..They love to say, ‘We’ve always done it this way.’ I try to fight that. "Admiral Grace Hopper, USN, Computer Scientist
"It's not Rocket Science! - Oh wait, Actually it is... "NASA bumper sticker
"Just because you can do something doesn't mean you should."my mantra in the Programming classes I used to teach once upon a time...
Veteran Member
Posts: 5,965
Registered: ‎07-03-2008
Kudos: 1888
Solutions: 140

Re: Prepare for the worst...

Familiar territory -- I've been through a couple of those.  One big transformer meltdown (shot hot oil nine stories in the air which got sucked into the genset).  The other was an inverter failure that pulled so much short-term current that it welded two 50,000 AIC fuses into their holders -- at which point the inverter immolated itself.  Re-charging the Inergen system cost more than replacing the inverter.

 

Perhaps someone upgraded a transformer and didn't bother to notify all the connected clients about the change in available fault current?  That worker really should be counting his blessings.

Regular Member
Posts: 758
Registered: ‎01-27-2015
Kudos: 292
Solutions: 6

Re: Prepare for the worst...

Wow! Glad everyone is OK. 

 

There must be something in the air...

 

One of our main towers sites is being converted from a wharehouse to apartments. They just finished the demolition phase.

 

A few weeks ago a crew was removing old conduit and pipes. They were told that one conduit was dead and OK to clear...

 

A gasoline powered saw cut right the the 240 VAC lines running from the 1st to the 10th floor. Sparked liked craszy, knocked the guy on his ass, but no other inury.

 

It blew power in the entire building, sadly including the elevators.

 

We ran off batteries and generators until temp power was restored. Then had my electrician come in and run a new line between 1st and 10th  floors. 

 

Fortunately our customers never noticed, but I had to walk up & down those stairs way too many times. 

 

 

 

Established Member
Posts: 1,109
Registered: ‎11-16-2010
Kudos: 758
Solutions: 15

Re: Prepare for the worst...

Incidents like these are unfortunately pretty common. We had one very similar happen to one of our datacenters: electrician doing a maintenance dropped his ratchet inside one of those power distribution cabinets (pretty much the same type as the pics you posted). The whole thing blew up, and he was lucky to get away with no major injuries. This cut off the power feeding the datacenter, and the generators started.... and ran for 2 days as the electricians waited for the right part to be flown in.

 

Expensive mistakes Leaving

"No Guts, No Glory!"