It was supposed to be a jingle-bell rockin’ sort of Christmas party. The date was Saturday, Dec. 9, 2001, and a small group of Rosenau Transport employees were putting on the finishing decorative touches at a nearby restaurant. But at about 3 p.m., as she was ready to go home and change for the party, Pat Young got a call on her cell phone indicating, quite frankly, more than just chestnuts would be roasting on this winter holiday evening.

“I got the call from one of our drivers,” the Rosenau controller says from the company’s office in Edmonton. “He says to me, ‘Do you know what’s going on?’ I thought he was talking about the Christmas party. He says, ‘Our building is on fire.’ The first words out of my mouth were ‘B.S.'”

But it was true. Young called company president Carl Rosenau, whose response was no more reserved than the one she gave the driver with the initial message. Young’s next call went to IT officer Terry Rhode. “I said to him, ‘You know that disaster management program we’ve been talking about? Well, you’ve got your chance to test it,'” says Young.

By the time Young pulled up to the company’s South Edmonton office, the air on 98th Street was a mix of smoke and the clamorous ring of sirens. The fire had torn through most of the building’s south wing, incinerating much of the company’s paper documents, office furniture, computers, and the network server. By early evening, the flames had been extinguished and a backhoe had torn down the smoldering remains of the walls. Blackened or seared office supplies lay scattered across the parking lot. Days later, fire investigators would report that a power bar shorted, causing a small fire which took hold of the objects around it, eventually spreading throughout much of the office. If there was any good news to be had, it was that, mercifully, no one was hurt, and that the company’s warehouse was not involved.

So instead of sitting back and toasting the company’s annual accomplishments over a cup of holiday cheer, senior Rosenau staff broke up into teams and began literally digging the company out from the ashes. Carl Rosenau talked to key customers and dealt with the media. A makeshift operations centre, with industrial block heaters, was set up on the dock, while administrative staff began outfitting a neighbour’s warehouse as a remote office, using cell phones to communicate with suppliers and other Rosenau terminals. An office-equipment supplier was able to re-route another customer’s order and delivered desks and computers within a couple of hours.

Rhode, the IT guy, was able to build a network on the fly, connecting the computers together. Meanwhile, Young was busy sifting through the charred mess in the parking lot, keeping an eye out for any salvageable tech equipment.

After locating the server, Young and Rhode contacted their computer supplier, Banks-Hill Systems. It was quickly determined that the system was beyond repair. A total loss.

Any contingency plans Rosenau Transport had for dealing with a catastrophic loss of computers and communications were little more than loose suggestions verbally tossed back and forth between Young and Rhode in the months prior to the fire. They had talked primarily about recovering computer data: off-site backup and storage, a ready list of emergency contacts, and a goal to strengthen relationships with disaster-ready technology suppliers.

“Most people as a rule, when signing contracts with suppliers, do it on a price-point,” says Young. “You don’t usually look at other issues. But in a situation of this magnitude you want somebody who is capable of putting together a big team at the other end and work to recover your data as quickly as possible.”

Within hours of the fire, the company’s IT equipment suppliers network communications providers had arranged to have a replacement server and parts flown in from IBM in the United States. Upon arrival, Banks-Hill Systems was able to update most of the lost data from backup disks Rhode kept at home.

By the early hours of Sunday morning, Rhode and Young were in the process of locating 1,700 pieces of freight to be delivered to still-unknown destinations. Even with the new server up and running, there was no way to access the information online. To rebuild the company’s hardwired DSL connection would take days, if not a week.

Luckily, Rhode had a secondary Internet provider (something, he says, all IT professionals should consider) that would install a wireless satellite link on the warehouse roof, allowing workers to remotely access data from the new server.

By 3 a.m. Sunday, the company had an Internet connection established, allowing managers to dispatch trucks and freight all that day. “Our trucks showed up Monday morning with a good portion of the freight on time,” says Rhode.

Customers couldn’t believe it.

“I didn’t go sign a contract with these suppliers because of price,” says Rhode. “I went with them because they proved they can do exactly what they said they can do, and they did. Yes, it cost a little more, but in such an event they provided the type of support we needed.”
Rhode tells businesses that are in the process of choosing IT suppliers to make sure these companies have disaster recovery plans of their own, or at least partners with reputable recovery services.

“If your IT suppliers don’t have recovery plans, how are they going to help you? Once you see theirs, you can ask them to help you build your own and support it in the contract,” he says, adding that a “disaster” for a company doesn’t necessarily mean fire or flood.
Losing your Internet service for a couple days when the construction crews outside your building accidentally chop your line can be harmful, too.

The golden rule of contingency planning and disaster recovery is to develop a system that shortens the outage time and allows a company to continue servicing its clients throughout the disaster. Such programs have received much interest mostly from large corporations since Y2K and Sept. 11, but small to medium businesses, with an “It’s-never-going-to-happen-to-me” attitude, have been harder to convince. Graeme Jannaway, president of the Toronto-based Disaster Recovery Institute Canada, as well as his own continuity consulting business, says people never consider the long-term effects a calamity can have on their company, beyond what property they may lose. He says everything you need for a contingency plan can be housed in three binders:

Binder 1: Project the consequences to your business in the event of various disasters — their financial impact, as well as the costs of re-establishing fundamental resources needed to continue operations and reduce outage time.

How would your business be affected if you lost your phone service? “There are things nobody will pick up on if you don’t do them for a couple days. But if you don’t answer your phone, people notice pretty darn quickly,” Jannaway says.

You can reduce the impact of an outage by securing a remote tele-equipped location, retaining two separate providers, or simply storing a box of pay-as-you-go cell phones off-site. “I’ve had clients tell me if they had 10 people with telephones, they could limp along for a couple of days,” Jannaway says.

What about your data and computers? Storing back-up discs off-site is a good start. Other companies “mirror” data with two separate storage units. Another, more expensive, option is to rent a “hot site,” a back-up location wired to offer computing resources.

Binder 2: “This is what you drag around with you in case you get the call that says, ‘Listen, we’ve got a big problem,'” Jannaway notes. It should include important company contacts, including those on the disaster management team; contact information for your employees; key suppliers, including your bank and insurance company; emergency services (police, fire, hazmat, etc.); and perhaps the location of a staff gathering point like a hotel or even the president’s basement.

Binder 3: Include all the items you want to keep off-site, such as data back-up disks, critical paperwork, an extra chequebook and ledger, and documentation related to your tried and tested continuity plan. Treat this like gold: “I hear it all the time,” says Jannaway. “You get a call in the middle of the night and realize the binder is in the office. Well, sorry, your office has just burned down.”

Pat Young and Terry Rhode stress that any continuity strategy has to evolve as your company grows. For example, many companies don’t review their insurance policies frequently, and may not be covered for all the technology assets added throughout the years. Rhode also suggests taking “extra-expense” insurance over downtime or outage insurance.

Downtime insurance pays for lost revenues during a business interruption. “If you’re down for a week, you get compensated, but by that time your competitor may have had a crack at your best customer,” Rhode says. “Extra-expense insurance means you can spend whatever it takes to get back up and running as quickly as possible and the bill is taken care of.”

Young adds that, although tedious, you should make a written and photographic record of every piece of equipment and furniture you have and attach a value. “The insurance company expects a full list of everything from the computers on the desks to the contents in your drawers within a week,” she says. “But does everyone know all the contents in the office?” That also goes for every department within your gates, like the shop, for example.

If your business is out of commission, the first person to get the shakes, besides you and your customers, will be your banker. You don’t want your credit limit cut off by a panicky rep.

“The bank wants to know the loss,” Young says. “How long before you’re up and running? Are you going to be able to generate revenue? If so, how much?” Give your banker a copy of your recovery plan and a statement showing the program is continually tested. “If you can show that you have a plan to get back on your feet quickly, they’ll be in more of a position to give you what you need if that day happens,” adds Rhode.

The last thing you need when you’re ankle deep in the ashes of your newly decorated corner office are investigators poking their noses around. But they too have a job to do, so assign a co-operative and informed individual to deal with them. “Part of the group you might want to include on your recovery team is the police or fire department,” says Jannaway. “Building those bridges before may make the investigation a little smoother.”

And since you’re handing out seats to your recovery council, include representatives from your best customers too. “It makes them feel involved,” he says. “That feeling of helping to rebuild something can only strengthen your business relationship down the road.”

About 50 per cent of businesses without some sort of continuity plan never reopen after a disaster, and another 30 per cent close up shop after two years. You don’t have to reside on a fault line in San Francisco or in a trailer park in Oklahoma to be at risk. As the braintrust at Rosenau Transport can tell you, there’s no prevention for either the wrath of Mother Nature or the imperfections of a $5 power bar. All you can do is get up, wipe yourself off, and figure out a way to keep rolling.

With an informal recovery plan, good teamwork, and a dash of luck, Rosenau hardly missed a gear.Graeme Jannaway, president of Disaster Recovery Institute Canada, says any extended interruption will send customers into the arms of a competitor. Jannaway (416/694-3274; jannaway@utoronto.ca) offers eight steps to take the minute you get “the call.”

1 Your first priority should be human life. Equip and train your staff to deal with emergencies — how to evacuate the building, provide first aid, etc.

2 Contain the calamity. Call the appropriate authorities and make sure the cause of the problem has been extinguished or controlled.

3 Communicate the right message. “You’ve suddenly become a news story,” he says. “Make sure the message that gets out is the message you want to get out.”

4 Contact your most reliable employees and disaster management team.

5 Contact your best and most important customers and reassure them that a business continuity program is in effect.

6 Get the management team together at a designated meeting point to evaluate the situation.

7 Start implementing the strategies you’ve been drafting, testing, and training for.

8 Mop up. When it’s all over, debrief and even reward your staff for putting your business back together again.