A Sense Of Urgency

Banks race to overhaul outmoded disaster recovery and business continuity plans

The banking industry has prided itself on its ability to weather a range of actual and potential disasters, a result of strict government regulation and the cost of system downtime. Yet no firm was prepared for the disaster that struck New York on Sept. 11, with its terrible loss of life and property.

"No one went unscathed. Regardless of the extent of their plans, everyone was affected," said Todd Gordon, vice president and general manager for business continuity and recovery services at IBM.

The events of Sept. 11 have permanently altered the lens through which people-including disaster recovery experts-view the world around them.

"People from the board of directors down will take a broader view of disaster recovery," said Gordon. "It is a question of recognizing that the world has changed; it is incorrect to believe you can do without a good contingency plan."

Nor is contingency planning likely to be relegated to the scrap heap of trendy management ideas. "It was such a horrific event that I think people will remain vigilant," said Martin Goulbourn, senior vice president of field operations at Comdisco, a Rosemont, Ill.-based contingency services provider.

Over a dozen banks, many of them international, had offices located in the World Trade Center (see sidebar) while many more were housed in nearby buildings that were either destroyed or heavily damaged in the attack. Several firms with offices near the site had to evacuate large numbers of employees, including Deutsche Bank (5,000), Merrill Lynch (9,000) and Shearson Lehman (6,000).

Still, most systems were operational soon after the attack. Deutsche Bank's money transfer systems were operational within two hours. A spokesperson for Bank of New York said, "Our plans worked well, our systems came back up; we were well prepared."

Good preparations helped keep the loss of information, or virtual assets, to a minimum. "There wasn't a lot of data loss," said Gordon. The fact that the attacks occurred before the markets opened mitigated the loss of data, as did the general preparedness of most IT departments.

"If anything, the events of September 11 proved the functionality of newer high tech solutions such as vaulting," said Ken Smith, president of planning solutions at SunGard Recovery Services, Wayne, Pa. High bandwidth data mirroring will become more common for backing up mission critical information, he added.

SWATH OF DESTRUCTION

Banks spend from 7%-8% of their data center budgets on disaster recovery (e.g., backing up mission critical information from mainframe and midrange servers, as well as databases) compared to 2.5% -3% by the average enterprise, according to Donna Scott, a consultant with GartnerGroup.

Still, none had planned for a disaster on the scale of Sept. 11, in which thousands of people died, and millions of square feet of office space and thousands of workstations were destroyed.

"Most companies assumed in their disaster recovery plans that they would eventually be able to return to their offices," said Scott.

In fact, in the days immediately after the attack, the task of locating office space and workstations proved burdensome for most companies.

"There just was not enough alternate office space or workstations," said IBM's Gordon. "Some companies had alternate workstations, but they were located outside the region. Another unique aspect of this incident was that you couldn't get on a plane."

Comdisco, which maintains approximately 2,000 available work-area seats in the New York region, was operating at over 100% capacity the week after the attacks. "We were able to build out our service capacity and meet the increased demand by taking one of our own office complexes and using it for clients," said Goulbourn. Comdisco has since sold its business continuity business to SunGard.

The physical destruction also meant the loss of departmental servers, desktop and laptop workstations and data that were not centrally backed up. "In the past, most network administrators were very concerned about keeping departmental server loads down," said SunGard's Smith. "Backup and recovery systems for the desktop and departmental server will now become more of a priority."

The most important technology-related failure of Sept. 11-and the one with the most far-reaching implications for disaster recovery planning-was the loss of telecommunications service to lower Manhattan. "Telecom links were down as a result of third-party failures beyond our control," said the Bank of New York spokesperson.

Two major Verizon POPs (points-of-presence) were located in the World Trade Center complex, and damage was also sustained by a nearby switching station. "There was too much concentration of traffic over networks at one Verizon site," said Gordon. Companies will require greater redundancy in telecommunications and networking in the future, he added.

Although an estimated 10 cell towers were destroyed in the attack, wireless communications played an important role in recovery efforts, as did alternative forms of communication such as e-mail.

LESSONS LEARNED

One of the most important lessons learned from Sept. 11 has been the need to simplify, test and update business contingency plans.

"In some cases, the plans were too big and ignored detailed issues-where to meet, how to contact people, having a disaster hotline that works when all phone systems are down," said Millard Banker, a consultant at Strohl Systems, a King of Prussia, Pa.-based recovery software and services firm. In the future, recovery plans will become "more lean and mean," he said.

SunGard's Smith concurs. "Even if you have 2,000 employees, with 50 to 100 backup work area seats available, you could have a 24-hour jump in recovery. The cost is minimal and the impact is huge."

"We found that our clients were for the most part undersubscribed in terms of their need for contingency work areas and networks and terminals," said Comdisco's Goulbourn. "Plans need to be updated every six months."

Guaranteeing redundancy and mitigating risk in technology systems, business processes, and physical assets, make up the other important lesson from Sept. 11.

Examples include having multiple copies of contingency plans available-hard copies, on networks, even on multiple hard drives-and making sure that employees are trained in multiple roles. At one firm, the only copy of the contingency plan was located in the World Trade Center offices. At another, said Strohl Systems' Banker, "they had copies of the recovery plan on the network in New York and London and Tokyo, but they couldn't get to any of them."

Some companies had to cope with the loss of entire departments of experts. In one instance, an entire IT department was destroyed. In another, a company lost all but one member of its HR staff. "Some of our customers have a good history of cycling people through a range of functions and departments. Organizations need to look at critical staff and clone the knowledge of those people," said Goulbourn.

Ensuring that those in leadership roles are psychologically prepared for an emergency is mandatory. "One company needed to move people and get back to business but some of the designated leaders couldn't handle it; they needed counseling, which is a factor that no one was well prepared for," said Banker.

TO CENTRALIZE OR NOT

The mitigation of future risk was a factor in Morgan Stanley's decision to sell its midtown office space to Shearson Lehman, according to SunGard's Smith. "By relocating their downtown offices to the new midtown building, Morgan Stanley would have had a dangerously high concentration of employees in one area."

Companies are reassessing the wisdom of centralization. "Companies want to centralize in order to save money but they are also realizing that they increase their risk exposure," said Gartner's Scott.

In the areas of telecommunications and networking, IBM's Gordon recommends investigating whether third-parties can deliver diversity and redundancy. "Companies need to ask secondary and tertiary providers where they get their lines from. We found that in some cases the backup carrier was leasing lines from the primary carrier and that is why there was a service failure."

Diverse circuit routing with different central offices is one solution to creating more redundancy in the telecommunications and networking infrastructure. Another is simply contracting with different regional carriers.

---

Banks With Offices Located in WTC

- Atlantic Bank of New York

- Banco LatinoAmericano de Exportaciones

- Bank of America

- Bank of Taiwan

- The Chugoku Bank

- Farmers Bank

- First Commercial Bank

- Fuji Bank

- Hua Nan Commercial Bank

- Overseas Union Bank

- San-In-Godo Bank Ltd.

- Shizuka Bank, Ltd.

- Taipei Bank

- Union Bank of California

- Xcel Federal Credit Union

Source: Washington Post Online