One Wild Week

What a week it was. Lloyd Hession, chief information security officer with financial-network provider Radianz, surely would agree. "It's been one helluva incredible 48 hours," he told an InformationWeek reporter, who reached him by phone Friday. "Can I call you back?" Click.

The scramble began with the emergence on Monday of the Blaster worm, which exploited a known vulnerability in Microsoft Windows to severely hamper corporate networks around the world, hitting some 12,000 to 30,000 systems per hour at its peak. The week ended with the largest power outage to hit North America, taking down businesses from Toledo to New York City. Ironically, the power outage may have slowed the spread of Blaster--but it won't stop the more destructive variations that are expected to begin appearing any day now (see story, "'Children Of Blaster'").

Feeling vulnerable again? You're not alone. Even companies with state-of-the-art security and business-continuity plans are wondering what's next.

"There may be some weaknesses in the nation's power system, and that's a great threat," says Randy Till, VP of business continuity at MasterCard International. MasterCard's business offices in Detroit and Toronto lost power last week, but its most-important systems were unaffected by the outage. Diesel generators back up its Missouri data-processing centers with enough fuel to last 23 days. And its Purchase, N.Y., facility has generators that can run for 12 hours. Now it plans to go a step further and investigate how power-outage cascades happen. "Are there vulnerabilities that can impact one or more locations at a time, and what have we put in place as far as control and prevention?" Till asks.

That's long-term planning beyond the reach of most small and midsize companies. They can afford to spend only so much on business continuity, and they likely suffered the most from the outage, says Joel Rakow, a contract IT executive with executive-placement firm TatumCIO Partners, who is preparing a disaster-recovery plan for a Los Angeles bottled-water company. "There are probably hundreds of companies in the $100 million to $500 million revenue range that don't have something like a diesel generator," Rakow says. At least one MasterCard partner--a small financial institution in Canada--didn't have operating generators when it lost power.

With so many threats, sometimes companies just aren't guarding against the right ones. That happened to DFS Group Ltd., a San Francisco company that runs duty-free retail shops in 400 locations worldwide. Chief technology officer Rick Hamilton was visiting his network-monitoring vendor in Austin, Texas, when Blaster smacked his company. Fortunately, fewer than 100 of 5,500 seats were affected, but the attack served as a wake-up call. Microsoft had warned DFS Group about the virus several weeks ago, but the company had applied the patch for it sporadically. As a result, it spent a few days cleaning up infected machines. "We had the opportunity to be more prepared and not so reactive," Hamilton says. "This past week has been a good reminder of the need to be prepared." On Friday, Hamilton's team was readying for a possible new round of attacks over the weekend.

Despite this incident, Hamilton expects he'll still have to wrangle with business execs over funding for business-continuity efforts. "Retailers don't want to spend the time or money on disaster recovery, and that's a matter of fact," he says. It's not just a problem in that industry: A Forrester Research survey released last month found that few business executives understand the vulnerability of their business-critical data. DFS Group is strengthening alliances with some vendors, such as network-monitoring company NetSolve Inc. and IT consulting and outsourcing company Cognizant Technology Solutions Corp., for insight into how to be better prepared.

But the power outage that stretched over seven states and portions of southern Canada points to one little tic in a traditional piece of business-continuity preparedness advice: Back up data to redundant servers that are off site. Companies typically locate these servers no more than a few hundred miles away, but last week that distance didn't necessarily help matters. SunGard Data Systems Inc., which provides business-continuity services, got calls from New York City clients that built redundant data centers in New Jersey after Sept. 11, 2001. "Guess what? It's out of power," says Jim Simmons, group CEO of SunGard Availability Services. By Friday, 62 customers had declared disasters and another 100 had put SunGard on notice that they might need its services.

"Despite spending millions, can you really count on your continuous-operation plans to kick in when you need them?" Hession asked when he called back. "You'd be amazed at how many stories there are of leaked tanks, generators out of gas, or a room that got too hot so they had to shut down." The American Stock Exchange in Manhattan didn't start taking trade orders until about 3:45 p.m. Friday, because of a substation outage that zapped its air conditioning.

So it should come as no surprise that after a week like this past one, many business-technology pros are rethinking continuity and security procedures. Execs at Pathmark Stores Inc., which lost power at 70 of the chain's 143 grocery stores across the Northeast, are among them. On Thursday night, senior VP and CIO Bob Schoening's team was up past midnight dealing with the crisis, reworking automated-delivery schedules and making adjustments to the online-inventory system to help stores that lacked sufficient backup refrigeration power quickly restock spoiled food. By this week, the company will start its day-after debriefing, which Schoening sums up as "what we've done to prepare for something like this and to look at what we could've done better."

-- with George V. Hulme

Additional coverage can be found at InformationWeek.