VisaNet’s Semi-Annual Upgrade Is the Largest Ever

Nine months in the planning, a massive system upgrade of Visa USA's payment network -- the largest in the world -- went off without a hitch earlier this month. The upgrade involved some 120,000 code changes, 150,000 worker-hours and 27 major enhancements and was carried out at four data centers around the world. Visa, which processes 100 million transactions a day, completed the upgrade at its data centers in Tokyo and London and at two locations on opposite coasts in the United States over a 24-hour period beginning at 1 a.m. on Oct. 2, Tokyo time. Each center was down for about two hours, beginning with Tokyo. During the time each center was down, local transactions were routed to the other three. The upgrades were timed to occur during the early-morning hours, when volume is at its lowest.

Visa upgrades its VisaNet system semiannually, in April and October. Each semiannual upgrade involves the months-long planning process, during which Visa's internal IT team receives requests from Visa member banks for new services and begins building the corresponding programming code. Visa makes a test bed available so the IT staffs of member banks can try out their own internal system changes against the VisaNet changes. A quality-assurance phase, lasting six to eight weeks, allows for fine-tuning before the upgrade goes into production.

This upgrade, the largest ever, included enhancements to speed the credit-card dispute-resolution process, simplify currency conversions and update Visa's loyalty-points program.

Some 300 engineers in Visa's development centers, as well as operations staff in the data centers, participate in each upgrade. Although most of the upgrades take place automatically, the process is managed and monitored by a central command center in California. There are also local command centers in each of the data centers. Each center has systems to communicate the center's status and the progress of the upgrade.

The upgrades are designed so that no more than one center is taken down at a time. As each one is brought back online, it's synchronized to work with the others that haven't yet been upgraded. The upgrade is a delicate balancing act because two versions of VisaNet--the old and the new--are running simultaneously throughout the 24-hour period. For example, while London is down, Tokyo is running the new version and the two U.S. centers are still running the older version. The network is designed to process transactions originating from either the older or newer systems.

Although other real-time, high-availability systems are brought down for regularly scheduled maintenance, Visa's is unique in that the network can never be brought down entirely, says Scott Thompson, executive VP of technology solutions at Inovant, Visa's processing subsidiary. "In any other real-time environment, all four system images would be taken down at the same time, the new system would be installed, then the centers brought back one at a time," he says.

In addition to regularly scheduled maintenance, VisaNet undergoes occasional unplanned outages, on the average of once a month per center, Thompson says. The outages seldom exceed a few seconds, and the network experiences not so much as a blip in throughput. Each center has enough capacity to run the entire worldwide transaction load. So-called heartbeat messages, indicating that a center is ready to receive transactions from another center, are constantly being generated and transmitted throughout the network.

Earlier this year, during its annual weeklong stress test, Visa ran the system at a processing load of 6,200 transaction messages a second, a 20 percent increase over last year's peak on Dec. 24 of 5,200 transaction messages a second. Each card purchase requires about two transaction messages.

Article courtesy of InformationWeek, October 18, 2004.