Miss. Credit Union Continues to Improve On Its Katrina-Inspired DR Overhaul
Hurricane Katrina turned Larry Mayo, vice president of information technologies at Keesler Federal Credit Union, a $1.9 billon, 17-branch credit union headquartered in Biloxi, Miss., into an expert on disaster recovery.
When Hurricane Katrina hit, "One of our branches completely washed away and all our branches took water in some form," Mayo recalled in an interview with BS&T today. "Fortunately our data center wasn't flooded, our issue then was we had no communications. After that, the board came to me and asked what could have happened. I told them we could have been dead in water. They said, what can we do about that? I said, give me a month or so and I'll give you an outline of what we need to do. They saw the need for better disaster recovery and gave me the money to build it."
The credit union set up a redundant data center 350 miles away and set up IBM storage area networks with encrypted replication between the main data center in Biloxi and the new location, as well as a third-level disaster recovery site run by Centurion. Mayo says that should there be a failure in the credit union's data center, primary systems could be up and running again within 30 minutes. Every year, he tests this theory end to end, by failing systems over to the redundant site. In October, Mayo plans to conduct a "big bang" in which he'll fail all systems over at once and operate completely off the backup site.
To accommodate all this data movement, Mayo is planning a major network overhaul in which he will install WAN accelerators and compression technology between the backup data center and the primary data center in the hopes of reducing the bandwidth needed to replicate all that data (the data centers are connected with a 100 MB network).
At the time of Katrina, some financial institutions had completely based their disaster recovery plan on backup tape restoration, Mayo notes. "For many companies, about 24% of their tapes were unusable when they got to their disaster recovery center. Tapes are not the most reliable resource and they're a dying breed. As cloud computing evolves, there are many other alternatives to tape technology."
That said, Mayo does have backup tape in place as a protection in case all else fails, and after Katrina hit he installed a CryptoStor Tape appliance from Thales Information Systems Security to encrypt the data coming from the credit union's Jack Henry/Symitar Episys core banking platform. CryptoStor is deployed at both the main data center and the redundant center.
Data encryption is both a regulatory requirement and common sense. "I don't want my name in a newspaper story saying, 180,000 accounts were compromised because we didn't encrypt the files or tapes," Mayo says. He tested the encryption keys earlier this year. "Encrypted tapes are pretty much useless if you don't have the encryption keys," he observes.
Fortunately, the credit union has never had to resort to tapes. "I hope I never have to use them, but the technology is there, we've tested it and are comfortable with it," Mayo says. "My staff is trained on what needs to be done to make it all work. We have no doubts that if push comes to shove, we'll be able to use those tapes."
The current disaster in the Mississippi region is, as everyone knows, the massive oil spill in the Gulf. Although Keesler Federal Credit Union has not been directly affected by the mess, "We're right on the coast, we do have customers who are shrimpers and fisherman and suffering like the shrimpers and fisherman over in Louisiana. The economy is in disarray." Help to such customers is worked out on a case-by-case basis. While Hurricane Katrina affected all 80,000 members of the credit union, the oil spill has affected 1,000, if that, Mayo says.
But Mayo is skeptical about the talk today that the surface oil is almost all gone. "You know that magically that stuff didn't disappear overnight, we're talking 180 million gallons of oil and it's sitting somewhere," he says. "We're going to be suffering from the spill globally for decades."
Tweet This