A Fine Night For Stargazing

August 14th provided New Yorkers with a rare chance to pick out constellations in the night sky, thanks to a severe failure in the electric power transmission grid that plunged a wide swath of the U.S. and parts of Canada into a dusk and nightfall without electricity.

But just because the lights went out in several cities doesn't mean that commerce stopped elsewhere. Business continuity plans, designed to last through Y2K and battle-tested on September 11th, came to the fore during the power grid failure, thus limiting the initial impact to people in areas lacking electricity.

"Every bank in the country has, and is required to have, a disaster recovery plan and multiple backup systems," says Charlotte Birch, media spokeswoman at the American Bankers Association, Washington, D.C.

The result was a reported smooth transition for back-office processing throughout the financial system. "Banks really didn't miss a beat in terms of transitioning over to backup systems," says Birch. "The way the banks have responded and continued operations during the blackout has shown the resiliency and the preparedness of the banking system."

MAKING IT TO PAYDAY
For one thing, most members of the U.S. workforce received their mid-month direct deposits as usual. "Employees who use Direct Deposit will have the money in their accounts, even if they've been told not to report to work today," says William B. Nelson, Executive Vice President of NACHA, a Herndon, Va.-based payments association. "However, people in the Northeast that still get paid by check may experience difficulties in collecting and depositing a paycheck if today is payday." "The clearinghouses and the payment processors have been operational and online all night," adds Michael Herd, director of public relations for NACHA.

Also, the major credit card networks remained online. "From an operational perspective on our global network and all of the endpoints on the network, everything was fine," says Artie Ehrens, senior vice president, computer and network services, speaking from the MasterCard data center in St. Louis, Mo. "If you had gone into the network command center and looked at the network and all of the charts, you would not have known we had an interruption of power in the Northeast."

The power grid failure gave MasterCard the chance to implement its well-rehearsed business continuity plans. "We went through a very structured approach at assessing, evaluating, and of course reporting - not only to our employees, but also to our customers and our management," says Randy Till, business continuity management, MasterCard.

"We were on a teleconference call probably within 15 minutes after the event had occurred," adds Till. "We probably had 40 to 50 people within our organization on that call."

Visa was similarly unfazed by the grid failure. "Visa's systems are operating and have not been halted by the Eastern power outage," says Camille Lepre, spokeswoman for Visa USA, San Francisco. "We have backup power to all our data centers."

"If we need to route transactions, we can do so to other data centers that we have," adds Lepre. Visa operates data centers at several undisclosed locations.

For consumers located outside of the failed grid, transactions went through as usual, even when dealing with banks located in the affected area. "The banks and the financial institutions need to be available, because you may be one of their cardholders and live in Seattle - they want to get their transactions approved," says MasterCard's Ehrens.

But even if the banks and the payments networks stand ready, it proved difficult to find merchants willing to stay open during a power failure, let alone those possessing and willing to use the old "zip-zap" machines to take a card imprint.

Along with the lights, ATM screens also went dark. But some ATMs are designed to shut down more gracefully than others. If a power outage occurred in the middle of a transaction, some machines would simply go blank and swallow the customer's card. While "dip readers" and "swipe readers" don't have that problem, motorized readers may have left some customers stranded without their cards or their money. For its part, NCR happens to offer an card-return feature that automatically triggers during power failures. "Not all ATMs have it, and you actually have to want the feature, but if the power goes down, the machine will get the card back to you," says Andrew Orent, vice president, self-service for the Americas, NCR Corporation, Dayton, Oh.

Another point of exposure during a power failure is that security cameras and alarm systems may not work during a power outage, although no such robberies exploiting that vulnerability were reported.

A GALAXY FAR FAR AWAY
Yet some online customers did encounter difficulties. For example, customers of about 250 credit unions that use Galaxy Plus Credit Union Systems, a Fiserv subsidiary based in Troy, Mich., were unable to access their accounts through August 15. "The power went out, the generator came on, everything ran along great, and then a part in the diesel generator gave way," says Leslie Muma, CEO of Fiserv, Brookfield, Wis. "We just tested that generator within the past seven days."

Chalk it up to Murphy's Law - or to the karma of those people that use credit unions instead of commercial banks. "The only thing that was wrong was that the part broke," said Muma on Friday, August 15. "We had to order the part -- the part is in, the generator is up, the lights are back on, we're waiting for the computer room to cool down, we'll have the systems back up and our credit unions will be posting Saturday morning to catch up."

Just in case the new part hadn't worked, Galaxy Plus was also preparing to transfer processing to a contingency site. "We would have had our customers back up by early Saturday morning in either case."

"Our folks crossed every 'T' and dotted every 'I', other than to have a backup generator for the backup generator," says Muma. "Some people do that, but there are very few, and it's enormously expensive."

"We've got seventy sites around the country, all with generator backup, and if we had to double the generators, we'd have to take the credit union rates up," adds Muma. Indeed, each time an event of this magnitude occurs, emergency planners learn more about how to refine and fine-tune backup systems for future occurrences. "At 9/11 we learned what kind of disaster recovery the banking industry in the New York metro area had," says NCR's Orent. "Some were good, and some were pretty bad."

"Sept. 11 was a major test of the system, and some improvements were made afterwards," says the ABA's Birch. "In fact, a lot of the preparations for a massive power outage were made when everyone was wondering what would happen when the clock struck midnight and entered the Y2K year."