PROVENANCE: Volume 1, Number 1 - Nov. 4, 1995

DISASTER PLANNING FOR DATA DEPARTMENTS: Working with clients to prepare for the worst

- by Guy Robertson, MLS

The author is: a widely published information security and related emergency planning consultant.

This morning your favourite client discovers a flood in his data centre. His hardware has been under water for several hours. He doesn't realize the full extent of the damage. He doesn't know how much data he has lost. He has no idea how he can recover from this disaster. What can you do for him?

How can you help a company that loses everything in a fire? What advice can you give to an information manager who fears the worst from a megathrust earthquake or a bomb? What about the increasingly urgent issues of operator error and sabotage?

Until recently resellers, systems integrators and their suppliers have contributed little to clients' disaster plans. But a new attitude towards computer security and data recovery is providing opportunities for those vendors who can reduce their clients' exposure to risk.

Events such as the Oklahoma Bombing and the Kobe Earthquake have forced many senior managers to consider the possibility of serious disruptions to their operations. The reseller who can help a client mitigate risks will be welcome, and well rewarded.

"Two factors are expanding the interest in disaster planning for data processing functions," says Brad Duholke, a regional manager for Comdisco Canada Ltd.

- "First, senior managers are increasingly aware of the effect a catastrophe can have on their business.
- Second, EDP audit societies across Canada are stressing the need for data recovery programs and effective business continuity planning."

Clients want more than enhanced data security, "says Duholke. "They're looking for the ability to make informed decisions under adverse circumstances. With the explosive growth on the desktop, they realize how much they could lose in even a minor emergency. They want back-up and breathing space as part of their routine operations."

For many organizations, disaster planning begins with some kind of emergency preparedness program (EPP), which comprises a risk analysis and a mitigation plan.

A risk analysis determines what natural and human-caused risks threaten a particular company. Natural risks vary from region to region and from site to site.

In Eastern Canada and the Maritimes, major risks include fire and flooding, severe weather and toxic spills.

The Prairies are notorious for floods and winter storms.

According to seismologists, Southwestern B.C. is overdue for "the Big One", an earthquake larger than any that have occurred lately around the Pacific Rim.

In most cities electrical spikes, surges, and brownouts can damage network components.

Depending on their severity, these risks can cause anything from minor inconvenience to permanent downtime.

Human-caused risks include various forms of sabotage, from simple-minded arson to sophisticated viruses. Operator--what computer security experts call the Whoops Factor--causes enormous data losses every day.

"Every computer site is exposed to risk from the personnel who run the system," says Ken Nagel, president of Proact DataStor Corp in Vancouver. "People drop equipment and trip over cables. An inexperienced technician can wreak havoc on any system. And there will always be programming errors caused by simple carelessness."

Some risks arise from both natural and human causes. For example, a flood can originate from a overflowing body of water, a broken dyke or dam, a burst pipe, or a hole in the roof. A rain storm is a frequent culprit, along with warm weather that melts snow too quickly for a drainage system to handle.

A disgruntled employee can take revenge on his employer--or former employer--by knocking a hole in a pipe, leaving a washroom tap running, or opening a skylight during a downpour. Last year in Toronto, a recently laid-off programmer submerged his erstwhile work space by shoving a garden hose through a window at closing time on a Friday evening. The resulting flood was not discovered by janitorial staff until the following Sunday morning. Insurance adjusters estimated over $700,000 in equipment damage; the cost to restore the data was even higher.

"Because so many data departments are located in basements, flood risk to computer systems is much greater," says Alan Reyno, a regional manager with Munters Moisture Control, which manages flood response teams across North America. "even a small amount of water can corrode the guts of a system within a couple of days. In many cases, the only solution is to replace the hardware as quickly as possible."

Reyno also notes that fires and floods often occur simultaneously.

"Let's say a fire breaks out in a building. The sprinklers go off. The firefighters arrive and turn on their hoses. It's not their job to worry about data preservation; they'll soak anything that smoulders. That means that their good work will douse the flames and protect lives, but it will also damage the hardware beyond repair. In this case, the well-intentioned flood causes far more damage than the fire. We see situations like this all the time."

A risk analysis should lead to a mitigation plan to reduce the prevailing risks. Frequently mitigation is inexpensive and commonsensical: plumbing and fire control devices should be regularly inspected; an uninterruptible power supply (UPS) should be installed and maintained; and all employees should be encouraged to learn first aid and CPR. Some mitigation measures, however are more complicated and require service from outside sources. Here a supplier can play an essential role in protecting a client.

During a sales call or informal lunch, the supplier can assure the client that while there are certain risks that prevail in the region, the supplier knows the client's needs and will be around to offer support and service should all Hell break loose. If the client is working on a disaster plan, the supplier can offer to become a part of the EPP team. Client and supplier can form a strategic alliance, which involves an agreement by the supplier to offer immediate service if a disaster occurs. Thus a hardware vendor will be obliged to replace without delay any equipment that is beyond repair, and a systems integrator must be on site as soon as possible to provide any necessary consulting.

There is always a cost attached to a strategic alliance, but only if a client needs help to recover from a disaster. Through the strategic alliance included in the EPP, the client has a greater chance of business recovery after a disaster, while the supplier has a stronger bond with the client.

"There's no question that supplier and client must work together before and after a disaster," says John Wilson, emergency preparedness manager at the Insurance Corporation of British Columbia. "I think that suppliers should be more assertive when it comes to getting clients ready to face a disaster. To many companies of all sizes will not be able to recover from a large data loss, simply because no paid much attention to the back-up routine. That defies common sense, and causes a lot of grief."

Wilson's associate manager, Lorne Chomos, was attending a conference in San Jose, California during the Loma Prieta earthquake of 1989. "I saw that the organizations most likely to survive were those that had planned cooperatively with a wide variety of sources and suppliers," says Chomos. "But that's California, where they're constantly preparing for serious contingencies. I only wish that the same sort of preparedness would become the norm in Canada."

When an organization has completed the EPP, usually it concentrates on a disaster response plan (DRP). The aim of any DRP is to protect life. The protection of property is secondary, since life is irreplaceable and there is always a reseller, systems integrator, or vendor available to supply necessary equipment. The message to the client during this phase of disaster planning is simple:

- Look after yourself first, and we the suppliers will look after your equipment needs when the fire is out or the floodwaters recede.

At this point, resellers might consider a DRP for themselves. Do you know the safest route out of your building? Do you know how to use the fire extinguishers on your site? If you work in an earthquake zone, have you identified a safe place to duck and cover until the shaking stops?

Remember that you will not be able to offer emergency service to your clients unless you and your business are safe and functional.

The final phase of disaster planning is the business recovery plan (BRP), which is also called the continuitiy plan. The purpose of the BRP is to ensure that an organization will survive after any disaster listed in the EPP. Ideally the organization will be able to conduct basic operations within a specified time. For example, the BRP for a bank will outline the most effective ways for branches to open and offer service to customers within 72 hours of any disaster. If a branch is destroyed, the BRP will indicate alternate facilities. If the disaster has damaged the bank's systems, hot or cold site activation will be recommended.

"The range of back-up and recovery measures is becoming broader and more sophisticated," says Ken Nagel, whose firm specializes in the post-disaster recovery of computer systems. "But no matter what device or program a company implements to recover its system, the basic formula for data security still applies. CIA is always a concern, and always will be."

CIA stands for confidentiality, integrity, and availability: three essential conditions for the preservation of data. Lacking any one of these conditions, a body of data and the system in which it resides are seriously jeopardized. A BRP for a data processing department will give directions for the maintenance of CIA following any emergency. In the event of a catastrophe such as the destruction of an entire facility, a BRP will advise systems personnel to activate a remote site, which can be a cold or hot site.

A cold site is a secure space where a systems department can set up its equipment and resume operations. A hot site contains a redundant system that allows the department to get back in business with a minimum of downtime. Both kinds of remote sites offer suppliers an opportunity for productive long-term relations with clientele.

"Hot and cold sites are the best insurance policy a company has," says Michael Weston, president of Data Base File Tech (DBFT) in Victoria, B.C. "You can back up data as much as you like, but unless that data is available after a disaster, your company will not be able to recover. In fact, severe data loss is a major cause of post-disaster bankruptcy. The earthquakes in California prove this again and again, as do the bombings of the Alfred P. Murrah Federal Building in Oklahoma and the World Trade Center in New York City."

Weston insists that strategic alliances between suppliers and their clients are crucial at every phase of the disaster planning process. "Multiple partnerships are increasingly common for the establishment and maintenance of hot sites, especially those reserved for large clients. Every supplier should be just as familiar with the BRP as the corporation's staff."

Weston's DBFT facility has become a major remote site centre owing to its basic specifications. The building will stand up to an earthquake over 8.5 on the Richter scale, and its security system is the most sophisticated in Canada. It serves a variety of large organizations in the finance and government sectors, as well as smaller companies from across the continent. DBFT can store any medium, and has vaults with controlled environments for tapes, disks, and microfilms. Also available at DBFT are large data silos that contain vital electronic information for regular back-up and fast post-disaster recovery. As DBFT's remote site space fills up, more suppliers are becoming aware of the spin-off opportunities.

Another provider of remote site space is FACS Records, which has facilities in Ottawa and various Western Canadian cities.

"There's a growing interest in maintaining servers in secure off-site locations," says Mike McAuley, FACS director of client services. "Our original business was tape storage, and that's still a cornerstone of our operations. We also store an enormous amount of paper records, but we see our business gravitating more and more toward advanced data back-up and support. I think that we'll be working with a lot more suppliers in the near future. Business recovery is a major issue, especially at a time when every news broadcast covers another earthquake or hurricane."

A BRP should also contain a comprehensive inventory of:

In an age of increasing risk to information systems, disaster planning has become a crucial task for all systems departments. To make sure that plans are truly effective, suppliers must bet involved in the planning process from beginning to end. For a supplier, a strategic alliance is both practical and morally responsible. It is also potentially profitable. Now what can you do for your favourite client, who is still looking for a bucket and mop?


Copyright © 1995 - Guy Robertson

| PROVENANCE HOME PAGE | Net Sites Sampler | Top Of Current Page |

Copyright © 1995 - Guy Robertson - NetPac Communications Ltd.