Bracing For The Big One

When hurricane Floyd spiraled up the East Coast this year, TV news programs were quick to send camera crews into the storm's maw, capturing memorable footage of evacuations, wind-battered coastlines, and flooded streets. While the human drama was compelling, businesses bore the brunt of the storm. Even companies with sophisticated disaster recovery plans, such as AT&T Corp., were powerless to stop the destruction. At a New Jersey facility shared by AT&T and Bell Atlantic Corp., flood waters burst the seals on cable vaults and poured into a basement full of electrical equipment. That forced Bell Atlantic to power down, which left more than one million people without service for several days. "We were pumping it out as fast as we could," says an AT&T spokesperson, "but the flood waters were shooting up through the toilets and off the ceiling. We were ready to start building boats and collecting animals." AT&T also suffered damage to a nearby building that serves cellular customers, leaving tens of thousands of people without service. And this despite the fact that the company has a crack team of experts who can respond to network problems almost instantly, aided by a fleet of emergency trucks that will bring equipment to wherever it's needed. In this case AT&T helped Bell Atlantic restore service by hoisting reels of cable up to its fourth-floor location and then running them through the halls and down the stairs to Bell Atlantic's ground-floor switching equipment. It was just one example of the type of storm-related activity that doesn't make the news: businesses struggling behind the scenes to fend off economic disaster. Earthquakes. Fires. Hurricanes. Explosions. Businesses face tremendous risks when disasters strike, yet few are well prepared to cope with them. There are a host of things that companies can do to protect themselves, yet experts estimate that only about 25% have solid strategies for responding to serious trouble. A company's exposure to Y2K problems, or to any form of disaster, isn't limited to its information technology, of course, but almost invariably the IT department is the focus of any contingency plan. "We have no formal program or plan beyond the IT department," says Chip Cyr, CIO at Aearo Co., Indianapolis, a $300 million manufacturer of industrial safety equipment. But its contingency plan for IT is comprehensive and aimed at providing 24/7 service to operations in the U.S. and several European sites. That level of preparedness puts Aearo in the minority, but perhaps not for long. There is a growing awareness of the need to prevent computer downtime and to respond instantly to any interruptions in availability. Vendors of disaster-recovery products and services are seeing business grow 25% to 40% a year. One reason is Y2K. This potential disaster has been on everyone's radar screen for years and has received so much attention that there will be no excuse to be caught off guard. But other changes in the business environment are -- or should be -- prompting companies to devise detailed responses to any form of trouble. The rise of e-commerce, the growing dependence of manufacturers on large, integrated software systems such as enterprise-resource planning (ERP), and the need to support global operations all combine to make the specter of downtime untenable. Thus, there is a real need for contingency planning, sometimes dubbed business-continuity planning or disaster-recovery planning. Vital but often overlooked, a disaster-recovery plan easily can mean the difference between smooth operations and complete collapse. "It's entirely possible that a company could go out of business if it doesn't protect its information assets," says George Ferguson, marketing manager for Hewlett-Packard Co.'s (HP) business-recovery services. "In 1989 the [Los Angeles] fires disrupted First Interstate Bank's operations," he notes, "and their survival was directly attributable to having a solid contingency plan they could invoke immediately." Manufacturing is not as information-intensive as financial services, but it's certainly heading that way. Ferguson notes that more than one-third of his company's clients have contracted with HP because they need to protect their ERP systems. "As ERP becomes the linchpin of the organization," Ferguson says, "companies begin to see that any interruption will have a huge impact." Contingency planning takes many forms, and contracting with a service provider such as HP is just part of what likely will be a multifaceted response to trouble. A range of companies including HP, IBM Corp., Comdisco Inc., SunGard Data Systems Inc., and others provide many disaster recovery services. One option is shifting all data processing to a "hot site," a fully equipped data center that is geographically distant from the site in peril. In this age of distributed computing, of course, companies sometimes can shift operations to one of their own facilities. Blumenthal Mills Inc., a manufacturer of upholstery fabric and mattress ticking in Marion, S.C., did just that this year. The company is used to hurricanes; in fact, it tests its disaster-response strategy every year before hurricane season starts. "We began to track Hurricane Floyd when it was still just a tropical storm way out in the Atlantic," says Ed Griffin, IT manager at the 600-employee company. When Floyd brushed against Florida and began heading up the coast, Blumenthal Mills was ready. The company shifted all its IT operations to its headquarters in New Orleans in an instant. "It's as easy as flicking a switch," Griffin says. "We have software that replicates all transactions in both locations, so if one goes down we can conduct operations at the other site." But even a system as simple as flicking a switch is not without its problems. Last year when Hurricane Bonnie hit, power outages were so frequent that the backup power supply burned out and the company couldn't boot up its computers. "Now we just shut everything down and run IT operations from New Orleans," Griffin says. "We don't worry about having to replace hardware -- we can do that. But the data is something we're very concerned about." Many companies share that concern, but a surprising number don't act on it. One reason, of course, is cost. Monthly fees for the sorts of recovery services offered by Comdisco, HP, and others can run from hundreds of dollars to several hundred thousand dollars. Some financial services companies spend more than $1 million a year. Just paying a consulting firm to help craft a contingency plan can cost from tens of thousands of dollars up to a staggering $25 million. But a more common reason is neglect. "After the '94 earthquake in Los Angeles our phone rang off the hook," says HP's Ferguson. "But within two weeks it was back to normal. People have short memories, and they think 'I survived this one, I'll survive the next one.'" He estimates that fewer than one company in four has a viable updated plan that can be put in place at once. It's a rare company that has done absolutely nothing to prepare for disaster, however. Far more common is a situation in which a plan has been created and then allowed to languish, often for years. That can be tantamount to having no plan at all. "When they need it," says John Jackson, president of Comdisco Continuity Services, "they find that it has nothing to say about hardware, software, or networks they've added since the plan was created. And it may require key employees to man certain battle stations, but those employees have left and their replacements have never been instructed what to do or expect." Contingency-planning experts say that plans should be updated yearly and whenever business processes change. And plans should be communicated better than they typically are -- to senior managers to alert them that a plan is in place and then throughout the IT and other departments that actually have to implement them. That is a chore that historically has fallen far down the priority list for most companies, but Y2K may change all that. In a seeming bit of irony, the one party that companies won't be able to turn to in the event of Y2K disasters are their disaster-recovery vendors. That's because Y2K does not qualify as "unanticipated or unexpected" and hence is not a "declarable" disaster. That doesn't mean the vendors will be enjoying New Year's Eve celebrations while their clients' computer systems crash and burn. "We'll be on an extremely heightened sense of alert," says Comdisco vice president Allan Graham. "If our customers have problems with the telecom or power systems, those situations qualify." What won't qualify are the failures that have been getting most of the press: glitches in companies' own software. "And that makes sense," says HP's Ferguson, "because if the software doesn't run on their systems, it won't run on ours, so moving to a hot site accomplishes nothing." At this late hour companies appear remarkably sanguine about their internal preparedness for Y2K, and most contingency plans turn on what to do if a business partner encounters trouble. Those scenarios tend not to have technological solutions, but hinge instead on old-fashioned manual work-arounds. "We've detailed a whole series of paper-based processes," says Blumenthal Mills' Griffin. "If we lose electronic connections we'll immediately shift to printed POs, take orders over the phone, and pay over the counter. We're really not too concerned right now." One contingency decision is whether to build in some extra inventory in the event that a supplier is knocked out for any reason, but companies are reluctant to do this. "We have tens of thousands of SKUs," says Charles Snyder, vice president of applications delivery and Y2K project director at Levi Strauss & Co., San Francisco, "so it would cost us more to build inventory than to fix any problems that might come up." Snyder notes that Y2K contingencies in the manufacturing sector differ widely depending on the line of business. "If you have a nested supply chain, like they do in automotive, or if you're very information-intensive, then you have a lot of things to consider, but apparel is more labor-intensive, so we have limited exposure." Snyder says that for Levi Strauss the big issue will be process control: What happens on the shop floor if an embedded system fails as the calendar turns over? "The good news is that if that happens, it's a one-time fix," Snyder says. "It won't amount to more than a reboot or the equivalent if we have to replace something." If a disaster does disrupt business, companies have an array of options at their disposal. Many contingency plans are built around the services of an outside vendor, who can run operations from its facilities (the aforementioned hot sites) until the client's site is restored. There also is an arsenal of hardware and software products that provide redundancy, so that if one site is knocked out operations can be instantly shifted to another location. Uninterruptible power supplies, huge batteries that can run even mainframe computers for hours, are available and often provide a useful bridge until emergency generators can be turned on. There is even software available for creating and maintaining the contingency plans themselves, so that a company has a dynamic document at its disposal. But most companies stress that the heart and soul of a good contingency plan is determining who does what, and why. "You have to start by prioritizing your business processes," says Dan Dec, a partner with the operational-risk-services division of PricewaterhouseCoopers in New York. "What makes your organization tick, and what runs those processes?" If Y2K has galvanized senior managers and made them aware that contingency planning is vital, the timing is fortunate. The business environment is changing, to the point where hot sites and similar forms of backup may not be enough. The truth is that any loss of computer availability, even to address routine maintenance and backup, now qualifies as a disaster. Companies that haven't developed responses to disruptions run a greater risk than ever before and may find that the next millennium begins with a very disheartening bang.