Power Outages: Human Response Can Be Key

Expect the unexpected. Certainly these are words we live by in the data center world, but what about other industries? When power outages happen and systems don’t operate as planned, how prepared is the staff for the emergency response?

I ran into a situation like this skiing in Colorado over spring break, of all places. The weather was awful. It was cold, snow was blowing furiously and there was a wind chill around 0F. I was on a quad express chairlift with my son, my brother-in-law and my niece when the chairlift suddenly stopped. O.K., somebody crashed at a blazing 1MPH when getting off the chair…it happens (I guess!).

A minute later my son’s girlfriend called and said the complex below lost power. All right then, a power outage.  Certainly they’ve got a contingency plan for this, as it takes utility companies between 30-60 minutes to switch over to a backup circuit.  Hypothermia would certainly begin setting in by then!

Given that we were 5 chairs from the top, a few minutes later we see a snowmobile cruising full speed to the Chair Lift Tower, then, a puff of black smoke and the sound of a diesel generator.  “Good, they DO have a generator” I say.  The generator continues to run, then shuts off.  The chairlift has not budged.  “Uh-oh.”  Soon there were more snowmobiles, more generator activity, still no movement.  The four of us can no longer control our shivering…jumping off the lift would be a last resort resulting in broken limbs.

Finally, after about 30 minutes, just as we’re starting to lose consciousness, the generator starts and the lift begins to move, We’re saved!! (Confession: weather was for drama only…it was sunny and 40F!)

Because I’m an electrical engineer by trade, I had to know what happened.  I stop my snowboard as close to the Chairlift Control Tower as possible and try to get one of the technicians to talk. I throw out a couple of basic electrical terms and soon he starts spilling his guts.  He said, “we lost Control Voltage and couldn’t start the chairlift” and then “we had no idea, but luckily one of the skiers that worked for the switchboard company understood this, and once we got 24v controls back, we could start the lift.”

Without Control Voltage, you have no “control” of the electrical system – this can mean breakers, PLCs, etc.  The generator can generate the 480v, but that’s not enough.  They likely could not operate the PLC’s to control the Variable Frequency Drives to operate the lift itself.

Human life was in play, although risk was minimal. Technician response: fail.

Lessons learned: Emergency situations create reactive training opportunities. In Data Centers Operations we have to create proactive training opportunities by understanding the systems and system response (training), creating written procedures, and continually drilling the staff so the unexpected doesn’t create a bigger emergency!