Serving Up Consolidation

Wachovia has drastically reduced its UNIX support costs by merging its servers

UNIX support at Wachovia Corp. was spiraling out of control. As servers were added to support ever-growingtransaction volume, more and more people had to be hired to maintain them. And with an announcement in August of 1,800 layoffs, the last thing the bank wanted was to continue increasing headcount to support a growing number of UNIX servers.

"With every so many servers, we had to hire additional staff to manage those servers," said Richard Mattox, senior vice president and manager of technical services in Wachovia's information services department. "We were seeing this as a never-ending proposition as our UNIX application environment was growing."

In addition to staffing, the IT infrastructure had started to pose other problems. Software maintenance, hardware and software upgrades were all snowballing, and availability was becoming an issue as well. "We were finding our service levels being compromised by the complexity of our heterogeneous environment," Mattox recalled. "It was another pointer towards the need to consolidate our systems."

So Mattox's team started the search for a way to manage a large-200 servers and growing-environment with the staff on hand, while at the same time improving systems availability. They didn't have to search far. As it turned out, Hewlett-Packard was at the time the bank's preferred UNIX vendor, although it is now one of several UNIX service providers.

HP was willing to allow Wachovia to view its server road map, including new architecture and design, thereby easing the task of selling the consolidation project to top management, which might have balked at the high initial costs.

"It required a strong justification. There is a large upfront investment before the payback begins," Mattox said, noting that the old environment still needs to be maintained while the new one is being broken in.

And individual business units needed to be persuaded that server consolidation wouldn't affect system performance. "It's a real education process for your line of business customers," said Mattox. "They have to understand that they will not be negatively affected by being in a high-availability cluster with other applications."

CLUSTERS AND NODES

Armed with information about HP's current and future plans, the $70.8 billion bank chose the vendor's Enterprise Parallel Server (EPS) data center cluster. Encompassing a complete suite of HP UX-based central system software, EPS is a centrally managed, high-capacity enterprise platform that can easily accommodate very large server consolidation and mission-critical high availability. HP's consulting arm assisted in implementing EPS along with four HP 9000 K570 server nodes and a D-Class server for systems administration and management.

Systems administration has been centralized through the use of the D-Class server, running HP OpenView Network Node Manager and IT Operations. An EMC direct access storage device and fiber channel server connections round out the configuration.

The majority of the implementation was handled in-house, with HP offering on-site assistance and education. Wachovia loaded and configured all of the operating system components, as well as the database products and some other utility products, like back-up and recovery software. HP loaded and configured its MC/ServiceGuard package and ClusterView, a component used to manage this environment.

Deployment was handled in two phases, over a period of about six months in late 1998. Wachovia first deployed its internal IT X-server support onto the HP 9000 K-Class server to ensure the stability of the system before adding a line of business.

The 9000 server now handles all X-server functions at the Winston-Salem, N.C.-based data center, as well as the master UNIX scheduling package and a treasury services line-of-business reporting application.

This project has taken the number of X-servers from six down to one. The bank was also on the verge of purchasing a new server for the UNIX scheduling package, which was instead loaded onto the cluster.

Although the consolidation has produced staff savings, Mattox said that there's no connection between it and the layoffs at Wachovia that were announced in August. "With our ability to consolidate onto the EPS platform, we've been able to avoid hiring any additional staff. What we've really done is taken the existing staff and freed up their time to handle other functions," he said. "It was more of a cost avoidance-we were able to defer hiring because we were able to stem the proliferation of UNIX servers."

That's not to say IT has been untouched by layoffs. According to Frank Robb, executive vice president at Wachovia, positions were lost across the total information services area, but with no connection to efficiencies brought by technology.

"This was a result of many things that are happening in the financial services community," Robb said. "Many banks are finding that in order to deal with revenue shortfalls, they have to accommodate them on the expense side. This is really a reduction of the expense needs of the total company rather than automated technology improvement."

Wachovia isn't alone in this. According to Annie Early, research director at GartnerGroup, "Wachovia is trying to look for a higher rate for their stakeholders, and I think that's kind of the name of the game right now-look where the efficiencies are so that you can keep producing that same return on the stakeholders' investments and keep the stock price at a lucrative position."

ROOM FOR MORE

To increase the value of the investment in the HP technology, Mattox's group is looking for more candidates to roll into the cluster. "We advertise it as a high-availability platform with a very high potential for deferring additional expense in the way of upgrades and additional boxes," he explained.

However, it's not easy to locate good possibilities. "UNIX consolidation is a very challenging environment, in that the applications, utilities and subsystems that you try to consolidate onto a single platform all have to align very tightly," he explained.

"At Wachovia, we want to experience true consolidation, and our definition of that is stacking multiple applications on a single server. In order to do that, the application has to run on the same version of the operating system, and then the various utility products have to align or at least not use the same UNIX function so as not to overlay each other and cause problems. We have found that particularly challenging; it's hard to find applications that line up."

But even if converting to true stacking applications isn't feasible, high availability can be created by defining and installing applications on one node and then having another node on the system designated as a failover node, to take over in the event of an outage on the primary node.

That's worked well for Wachovia. Over the past year and a half, each server has had something go wrong at some point that caused a failover to occur, Mattox said. And every time, the failover went unnoticed by end users.