Planning for Continuity
Following the terrorist attacks of Sept. 11, 2001, the banking industry realized that no business is immune from catastrophic events. The severity of the 2004 and 2005 hurricane seasons, however, taught financial institutions that disaster recovery programs alone cannot protect their businesses, forcing banks to reevaluate the strength of their backup plans.
With the destruction wrought by Hurricane Katrina still largely visible, banks have renewed their focus on preparedness as they rethink their risk management strategies and bolster their business continuity plans.
"These events forced banks industrywide to take a long, hard look at how effective their disaster recovery plans might have been in these instances," says Michael Croy, director of business continuity for Forsythe Solutions Group, a Skokie, Ill.-based provider of technology infrastructure solutions, of the severe hurricanes that hit the United States in 2004 and 2005. And financial institutions that were directly impacted by these brutal storms learned firsthand how unprepared they really were, he adds. "Damage [from Katrina] was so severe in certain regions that some banks couldn't bring up applications until January [2006]."
Even more sobering are predictions that some organizations may never recover. Several industry sources cite research that predicts that 90 percent of unprepared companies that suffer 10 days of data center downtime for any reason will be out of business within a year.
"Every minute of downtime—planned or unplanned—costs the institution more than just millions of dollars," explains Mark Vanston, senior adviser for business continuity and enterprise risk strategies for Palo Alto, Calif.-based Hewlett-Packard (HP). "It can also cost them the confidence of those who depend on that institution. This includes customers, suppliers and business partners."
Thus, the banking industry now strives toward the lofty—though critical—goal of business continuity. Rather than rely on disaster recovery plans that help them get back to work after a catastrophic event, banks are focusing on business continuity plans, frameworks that are established to keep them operational in the first place. While these plans clearly highlight the importance of hot sites, banks also have learned that Web-based communications networks and imaging solutions could be the tools they need to survive a catastrophe.
The Best-Laid Plans
Even the best-laid plans can encounter unexpected challenges. So experts suggest that banks update their business continuity plans on a quarterly basis and regularly test their continuity frameworks.
"Business continuity exercises are a must to ensure [their] effectiveness," says Romir Bosu, CEO for CompuShare (South Coast Metro, Calif.), a provider of information technology consulting and solutions for the financial services industry. "A battery of potential scenarios guides staffs through mock recovery plans. This ensures their effectiveness."
Thus, management teams are strategically restructuring their plans to account for every possibility. "Every scenario, every possibility needs to be accounted for," says Kim Waller, managing director for Scarborough, a division of Financial and Professional Risk Solutions, the Chicago-based holding company that is owned by Aon Corp. "Drills need to be exercised and plans need to be put into action. This will ensure that banks can get back to business and take care of their customer commitments."
Galveston, Texas-based Moody National Bank ($860 million in total assets) clearly understands the importance of business continuity planning and testing. After escaping the 2004 hurricane season unscathed, Moody executives didn't want their luck to run out. "We hadn't touched our plan in awhile, so we began updating it at the end of 2004," recalls Betty Bisesi, the bank's assistant vice president, IT operations manager. "By February 2005, we had a [new] plan in place," she continues. "Using the plan as a guide, we began running exercises with upper management and acting out mock scenarios within each department."
Moody's efforts paid off when Hurricane Rita threatened Galveston in mid-September 2005. When the Texas island was evacuated, Moody sprung its continuity plan into action with the help of Jack Henry & Associates' (Monett, Mo.) Centurion Disaster Recovery division, which maintains 10 business continuity hot sites across the United States. These pseudo-branches are outfitted with telephone lines, Internet connections and hardware and software that enable a small staff to service customers. Before Rita hit in the predawn hours on Saturday, Sept. 24, Moody set up shop at a Jack Henry/Centurion hot site located in Lenexa, Kansas.
"We normally do our annual testing at the hot site located in Allen, Texas. However, we could not use it in September because there were Jack Henry New Orleans customers already at the hot site," Bisesi explains. "We were able to open some branches for business on Sunday."
Hot to Trot
Jack Henry isn't the only technology provider that offers banks the use of hot sites. Taking a proactive approach toward business continuity, HP has invested $100 million to build what it calls business continuity and recovery centers. Designed to get customers back up and running in minutes, these secure facilities help banks proactively maintain, recover or resume their critical business processes following virus attacks, natural disasters and other unforeseen events, according to HP's Vanston. Currently, 76 facilities operate in regions across the Americas, Africa, the Middle East, Asia Pacific and Europe, he notes.
"These new centers offer an adaptive infrastructure that helps our customers to manage change, while reducing complexity and cost," asserts Vanston. "Our centers address our increasing customer demand and help our customers build a resilient IT environment that can easily accommodate tomorrow's needs," he adds. "HP's mobility solutions also allow employees to connect to systems and data remotely in case of a disaster."
Of course, in the event of a disaster, banks' internal business continuity plans can be only as effective as the plans of their third-party partners. To avoid service disruptions as the result of hurricanes or other catastrophes, Houston-based electronic funds transfer network PULSE EFT Association created a dual processing environment in 2003. With one platform residing in Houston and the other in Dallas, banking and third-party data partners seamlessly connect to the platforms via custom solutions developed by PULSE. "In the event of a hurricane—or any catastrophic event—we can switch platforms within minutes, and we can run 100 percent of our volume through either platform," says Warren Coles, PULSE's EVP of operations.
In the wake of Hurricane Rita, PULSE switched all of its processing volume to its Dallas location. "It was prudent to move operations to Dallas 48 hours before landfall," Coles explains. "Dallas ran 100 percent of our operations for four days. We never experienced a moment of downtime, and our users seamlessly used the network."
A Backup Plan
Still, banks cannot run hot sites without access to updated core processing and customer data. The recent hurricane seasons proved that this mission-critical data must be protected outside of the bank's four walls.
"Banks cannot keep tapes locked up in a vault in the building or an IT center across the street," says Douglas J. Barton, manager, Centurion Disaster Recovery Services. "Banks need adequate data backups that are stored a safe distance away," he adds. "This ensures that they can retrieve information when they need it."
Mount Olive, N.C.-based Southern Bank and Trust ($1.1 billion in total assets) operates 50 branches in coastal areas, but the institution was grateful to be untouched by last year's tumultuous hurricane season. "We were very fortunate that it was a mild season for us," says Drew Covert, the bank's SVP, corporate administration. "But we would be foolish to disregard what could happen in the future."
To prepare for that uncertain future, Southern Bank and Trust dutifully backs up its core processing operations, which are handled by First Citizens Bank and Trust Co. (Raleigh, N.C.). "They do disaster recovery testing regularly, and those results are relayed to us in timely reports," Covert says of First Citizens Bank and Trust.
To ensure access to data during business disruptions, many banks are leveraging Web-based technology, such as Westlake, Ohio-based Hyland Software's application service provider solution. Hyland's solution features the vendor's OnBase content management repositories, Centera replication hardware from EMC (Hopkinton, Mass.) and operating platforms in Hyland's Washington, D.C., and Cleveland data centers.
"All loan documents and images are housed in our OnBase database, and they are retrieved through a Web-based interface," says Jason King, Hyland's director of financial services. "As long as they had the proper procedures in place, banks affected by the hurricanes set up their hot site and accessed a Web interface to successfully retrieve their information."
People's Bank (Biloxi, Miss.; $845 million in total assets) also is diligent about its contingency plans. However, Chevis Swetman, the bank's president and CEO, feels there always is room for improvement. "Here in the Gulf Coast, hurricanes are not unusual occurrences," he says. "While we have a detailed contingency plan, we are already considering systems to prepare for next year."
Reach Out and Touch Someone
While access to data is critical, communicating with employees is Swetman's top concern. When Hurricane Katrina forced down landlines and bell towers across the Gulf Coast, Swetman relates, he had no way to communicate with his staff. "Our personnel could not call into the office, and we could not call out. Worse, the day after the storm, only 18 people from our 230-person workforce reported to work. We were worried and had no way to learn of their well-being."
Fortunately, no one was hurt. But not all of People's Bank's locations escaped unscathed—seven of the bank's 17 branches were damaged by the storm. As a result of the experience, "We want a better method to communicate with staff, as well as transmit and retrieve data to our operating center," Swetman says. "We will be looking at different data and voice links, though we are unsure what the answer will be."
Telecommunications also challenged Birmingham-based Regions Bank ($84.6 billion in total assets). But, while most cell phones and e-mail exchanges were useless during Hurricane Katrina, bank executives could communicate through text messaging. "Surprisingly, several people experienced much better reception with their BlackBerrys—as long as they could keep the batteries charged," says Pat Martin, vice president, corporate communications for the bank.
While satellite telephones also offer a potential solution for communicating with staff during a disaster, voice over IP (VoIP) is garnering much attention recently. The Internet protocol-based data networking technology is gaining popularity in terms of business continuity planning because it can be set up over the Internet or corporate intranets. Moody National, for example, added VoIP to all of its 29 branches in September 2005.
Beyond voice communications, to ensure access to data, some banks plan to support their VoIP efforts with virtual private networks (VPNs)—private data networks that use the public telecommunications infrastructure. These connections maintain privacy through tunneling protocols and security procedures, but offer a low-cost option because they tap a shared infrastructure.
"Historically, banks were leery of VPNs as their primary connectivity method," says Joseph D. Lockwood, CTO, COCC, an Avon, Conn.-based processing solutions company. "However, in a disaster, Internet-based VPNs could provide the best option to restore communications."
Sharper Imaging
In addition to exposing communications problems, the recent hurricane seasons also revealed just how dependent the industry still is on paper-based documents. People's Bank felt the problem firsthand when its New Orleans-based check processing facility was washed away. "Most of our paper items are sent to New Orleans—then everything we were accustomed to was no more," the bank's Swetman explains. "We had to begin dealing with a center in Alabama," he adds. "The center was more than 160 miles away—we needed to rely on couriers and if they had access to fuel to get there."
As a result of the damaging storms of 2004 and 2005, imaging technology has received a much-needed push. "Imaging technology would have helped a lot in this situation," Moody's Bisesi says. "As image architectures move across the industry, banks in all regions will gain easier access to information necessary to do business."
While Moody already has check imaging in its Galveston- and Pasadena-based data centers, the bank is installing branch capture devices across all of its branch locations. "We won't have to rely on couriers to deliver items for proof processing and also deliver items to our clearing bank," Bisesi says. The project, whichlaunched in February, should be completed by July, she adds.
For its part, Southern Bank is working with Hyland Software to improve the bank's imaging capabilities. "This supports a permanency of records, more so than the industry has had in the past," Southern Bank's Covert says. "It is the next logical step to keep documents secure."
Similarly, People's Bank will explore its own check presentment options this year. "We need to more effectively manage paper," says the bank's Swetman. "We might have waited until 2007 to explore these solutions. However, now we plan to make this a priority for the second or third quarter of this year," he adds. "[The hurricanes] definitely accelerated the process."