Fear of Friday the 13th. I’ve seen the movies, heard the stories, and even spread some of the rumors. But that’s all it’s ever been for me is movies, stories, and rumors. I’ve never really bought into the whole Friday the 13th thing……..until Friday July 13th 2012.
It was a dark dreary night with fog laying heavy on the pavement. Ok, not really, it was a regular work day here at Fogo with a few usual Georgia pop-up thunderstorms making their way through town. At 6:06 pm lightning struck near the Clem Substation, about 5 miles south of Carrollton, with a magnitude of -91 kA. With the average bolt of negative lightning carrying 30 kA, this was a big’un. Because we have two utility feeds coming into the facility, in most cases we would not have experienced an outage because the utility would have switched over to the backup feed, but according to the local provider there was a “mis-coordination of [the] automated switch. Ultimately the normally closed switch saw operations and opened as it was supposed to; however, the normally open switch (backup feed) saw the same operations and would not close due to the loss of voltage.”
So in summary lightning struck and we lost power to the facility. There were several of us here at the office when it happened, but the beauty of it was that we didn’t need to be. Here is how the facility handled it:
6:06:35 pm – Lightning strike, lights flicker in the office, utility power is lost
– Building Management System sends alert of the power loss
– Flywheel UPS (A leg) ATS Command Contact Closes and discharges power to Data Center
– Battery UPS (B leg) senses loss of voltage and goes “On Battery”
6:06:37 pm – Both generators kick on
6:06:42 pm – Generator 1 takes load and powers facility. Generator 2 is on standby.
6:06:52 pm – Battery UPS rectifier ON system Normal, “Off Battery”
6:07:08 pm – Flywheel UPS ATS Command Contact Opens, discharge complete
On Generator power…
7:26:14 pm – Utility is restored and Generators stay on for an additional 25 min stabilization phase and 5 min cool down
The facility, critical and non-critical, was on generator power for 1 hour and 19 min. The data center never lost power. We had a fuel truck on-call and ready in the event that power was not restored in several hours or days.
On a side note, that same night a similar storm hit Turner Field, the lights went out, and the Braves game was delayed for 16 min. The lights stayed on at Fogo…..as they always do.
Tags: best data center
, data center power outage
, Redundant Infrastructure
, Redundant Power