For many technology managers, late October’s Hurricane Sandy was a wake-up call. AV professionals in the Northeast Corridor unaccustomed to dealing with natural disasters found themselves scrambling to keep systems running during the calamity. Many managers learned valuable lessons that they want to share with the industry.
Anthony Verbanac, IT director for NewBay Media, publisher of AV Technology magazine, manages the company’s data center on East 28th Street in Manhattan. The main office lost power during Super Storm Sandy but, fortunately, no data went missing.
“Thankfully, our data center located in the basement of our HQ building never missed a beat and was on generator power the entire time,” said Verbanac. However, NewBay Media did lose its primary data/voice circuit and backup voice circuits; cutting the company off from the rest of the world.
“The issue is that our primary circuit is coming out of a flooded ‘telco hotel’ at 75 Broad Street in lower Manhattan,” Verbanac said. “Once the power was restored to our building late Friday, we were able to get email and other critical services up and running over two backup Time Warner Cable connections.” At press time, NewBay Media was trying to restore voice services over the backup line.
Lesson learned: AV tech managers need to vet the physical location of their telecom data circuits, to find out if their carrier’s telco hotels are vulnerable to flooding and other weather-related events. “If this [Hurricane Sandy’s flooding] is the new norm in potential storms, as some suggest, then the current location of some Manhattan data centers is no longer suitable,” Anthony Verbanac noted.
Just a subway ride away from NewBay Media is Michael Andrews, a full service technical-production company in midtown Manhattan. The company’s senior resource manager is Kristine Penny. Michael Andrews’ office sustained no damage but it was closed for three days. “We lost a number of jobs that were cancelled due to the storm,” Penny said.
When staff were able to return to work, they discovered that their telephone and email communications were down: “We had to use our cellphones to communicate,” said Penny. “But a lot of people’s cellular service wasn’t working properly, so we found ourselves relying on text messages.”
Even when Michael Andrews’ communications issues were resolved, the chaotic state of New York’s mass transit system made it extremely difficult for staff to come to work, and to get to jobs off-site. “I had a tech today who was three hours late because he could not get on the train,” Penny said. “It was that packed.”
Lesson learned: Next time a storm is expected, Kristine Penny will draw up a contingency plan for staffing. Mindful of NYC’s lingering transit issues, “We will draw up car pools ahead of time,” she said.
Another tech manager in Sandy's path was Matthew Wilk, associate director of digital classroom services at Rutgers in New Jersey. “The university was closed down for two days by the storm, which is unheard of in its history,” he said.
Fortunately, Wilk and his fellow technicians disconnected and protected classroom AV equipment before Sandy, so their systems didn’t suffer any damage from on-campus flooding. Nevertheless, Wilk learned that “better interdepartmental coordination” is needed in the run-up to weather emergencies.
Another lesson that applies in any emergency: Connect sensitive electronic equipment to AC voltage regulators, to prevent electronics damage from AC voltage spikes during power issues.
“When AC voltages drop to very low levels in a storm and cause brownouts, they tend to come back with a huge spike,” said Garth Powell, Furman’s senior product designer. “High-quality surge suppression and over-voltage protection will be needed to make it through without valuable equipment sustaining damage. A typical AC strip will not survive this sustained surge.”
Meanwhile, vital equipment should have its own backup power source. “With the Furman F1500-UPS and Battery-Extension package, critical control devices can continue functioning perfectly for hours at a time while others are ‘in the dark,'” Powell said.
David Pogue, regular columnist for the New York Times, wrote recently: “On Twitter, a number of people have suggested that it might be interesting to hear how a tech columnist muddles through a 10-day stretch without electricity and Internet (not to mention heat or hot water). The short answer is: Pretty much like the other seven million people whose electricity blew out with the storm. You manage.” He offers concrete tips for surviving without power on his New York Times blog.
One final Sandy lesson comes from Nicolas Scott, Crestron's solutions marketing manager, control systems, user interfaces, software. “Keep two 12-Volt car batteries wired in series so you can keep all of your Crestron gear up and running if the power goes out,” Scott advised. “Or you can just go with a CEN-UPS1250 uninterrupted power supply."
Do you have advice for system redundancy or emergency staffing? Perhaps you have your own lesson learned from the storm? Write into our LinkedIn page today and share your experience. Our thoughts are still with the thousands of people in the Tri-State area still without power and or housing. Technology managers and system integrators can help the recovery effort today by sharing tech expertise—#nytechresponds and www.nytechresponds.org.
