Document and Maintain Your Disaster Recovery Plan

Document and Maintain Your Disaster Recovery Plan

Put your DRP on a diet – keep it fit, trim, and ready for action.

ANALYST PERSPECTIVE

The traditional disaster recovery plan (DRP) “red binder” is dead. It takes too long to create, it’s too hard to maintain, and it’s not usable in a crisis.

“This blueprint outlines the following key tactics to streamline your documentation effort and produce a better result:

• Write for an IT audience and focus on how to recover. You don’t need 30 pages of fluff describing the purpose of the document.
• Use flowcharts, checklists, and diagrams over traditional manuals. This drives documentation that is more concise, easier to maintain, and effective in a crisis.
• Create your DRP in layers to get tangible results faster, starting with a recovery workflow that outlines your DR strategy, and then build out the specific documentation needed to support recovery.”

This project is about DRP documentation after you have clarified your DR strategy; create these necessary inputs first

These artifacts are the cornerstone for any disaster recovery plan.

• Business Impact Analysis
• DR Roles and Responsibilities
• Recovery Workflow

Missing a component? Start here. ➔ Create a Right-Sized Disaster Recovery Plan

This blueprint walks you through building these inputs.
Our approach saves clients on average US$16,825.22. (Clients self-reported an average saving of US$16,869.21 while completing the Create a Right-Sized Disaster Recovery Plan blueprint through advisory calls, guided implementations, or workshops (Info-Tech Research Group, 2017, N=129).)

How this blueprint will help you document your DRP

This Research is Designed For:

• IT managers in charge of disaster recovery planning (DRP) and execution.
• Organizations seeking to optimize their DRP using best-practice methodology.
• Business continuity professionals that are involved with disaster recovery.

This Research Will Help You:

• Divide the process of creating DR documentation into manageable chunks, providing a defined scope for you to work in.
• Identify an appropriate DRP document management and distribution strategy.
• Ensure that DR documentation is up to date and accessible.

This Research Will Also Assist:

• IT managers preparing for a DR audit.
• IT managers looking to incorporate components of DR into an IT operations document.

This Research Will Help Them:

• Follow a structured approach in building DR documentation using best practices.
• Integrate DR into day-to-day IT operations.

Executive summary

Situation

• DR documentation is often driven by audit or compliance requirements, rather than aimed at the team that would need to execute recovery.
• Traditional DRPs are text-heavy, 300+ page manuals that are simply not usable in a crisis.
• Compounding the problem, DR documentation is rarely updated, so it’s just shelf-ware.

Complication

• DRP is often given lower priority as day-to-day IT projects displace DR documentation efforts.
• Inefficient publishing strategies result in your DRP not being accessible during disasters or key staff not knowing where to find the latest version.
• Organizations that create traditional DRPs end up with massive manuals that are difficult to maintain, so they quickly become unreliable.

Resolution

• Create visual and concise DR documentation that strips out unnecessary content and is written for an IT audience – the team that would actually be executing the recovery. Your business leaders can take the same approach to create separate business response plans – don’t mix the two into an all-in-one plan that is not effective for either audience.
• Determine a documentation distribution strategy that supports ease of maintenance and accessibility during a disaster.
• Incorporate DRP maintenance into change management and project intake procedures to systematically update and refine the DR documentation. Don’t save up changes for a year-end blitz, which turns document maintenance into an onerous project.

Info-Tech Insight

1. DR documentation fails when organizations try to boil the ocean with an all-in-one plan aimed at auditors, business leaders, and IT. It’s too long, too hard to maintain, and ends up being little more than shelf-ware.
2. Using flowcharts, checklists, and diagrams aimed at an IT audience is more concise and effective in a disaster, quicker to create, and easier to maintain.
3. Create your DRP in layers to keep the work manageable. Start with a recovery workflow to ensure a coordinated response, and build out supporting documentation over time.

An effective DRP that mitigates a wide range of potential outages is critical to minimizing the impact of downtime

The criticality of having an effective DRP is underestimated.

Cost of Downtime for the Fortune 1000

• Cost of unplanned apps downtime per year: $1.25B to $2.5B
• Cost of critical apps failure per hour: $500,000 to $1M
• Cost of infrastructure failure per hour: $100,000
• 35% reported to have recovered within 12 hours.
• 17% of infrastructure failures took more than 24 hours to recover.
• 13% of application failures took more than 24 hours to recover.

Size of Impact Increasing Across Industries

• The cost of downtime is rising across the board and not just for organizations that traditionally depend on IT (e.g. e-commerce).
• Downtime cost increase since 2010:
  • Hospitality: 129% increase
  • Transportation: 108% increase
  • Media organizations: 104% increase

Potential Lost Revenue

The impact of downtime increases significantly over time, not just in terms of lost revenue (as illustrated here) but also goodwill/reputation and health/safety. An effective DR solution and overall resiliency that mitigate a wide range of potential outages are critical to minimizing the impact of downtime.

Without an effective DRP, your organization is gambling on being able to define and implement a recovery strategy during a time of crisis. At the very least, this means extended downtime – potentially weeks – and substantial impact.

Only 38% of those with a full or mostly complete DRP believe their DRPs would be effective in a real crisis

Organizations continue to struggle with creating DRPs, let alone making them actionable.

Why are so many living with either an incomplete or ineffective DRP? For the same reasons that IT documentation in general continues to be a pain point:

• It is an outdated model of what documentation should be – the traditional manual with detailed (lengthy) descriptions and procedures.
• Despite the importance of DR, low priority is placed on creating a DRP and the day-to-day SOPs required to support a recovery.
• There is a lack of effective processes for ensuring documentation stays up to date.

(Source: Info-Tech Research Group, N=165)

(Source: Info-Tech Research Group, N=69 (includes only those who indicated DRP is mostly completed or completed))

Improve usability and effectiveness with visual-based and more-concise documentation

Choose flowcharts over process guides, checklists over lengthy procedures, and diagrams over descriptions.

If you need a three-inch binder to hold your DRP, imagine having to flip through it to determine next steps during a crisis.

DR documentation needs to be concise, scannable, and quickly understood to be effective. Visual-based documentation meets these requirements, so it’s no surprise that it also leads to higher DR success.

DR success scores are based on:

• Meeting recovery time objectives (RTOs).
• Meeting recovery point objectives (RPOs).
• IT staff’s confidence in their ability to meet RTOs/RPOs.

“Without question, 300-page DRPs are not effective. I mean, auditors love them because of the detail, but give me a 10-page DRP with contact lists, process flows, diagrams, and recovery checklists that are easy to follow.” (Bernard Jones, MBCI, CBCP, CORP, Manager Disaster Recovery/BCP, ActiveHealth Management)

Maintainability is another argument for visual-based, concise documentation

There are two end goals for your DR documentation: effectiveness and maintainability. Without either, you will not have success during a disaster.

Organizations using a visual-based approach were 30% more likely to find that DR documentation is easy to maintain.	➔	“Easy to maintain” leads to a 46% higher rate of DR success.

Not only are visual-based disaster recovery plans more effective, but they are also easier to maintain.

Overcome documentation inertia with a tiered model that allows you to eat the elephant one bite at a time

Start with a recovery workflow to at least ensure a coordinated response. Then use that workflow to determine required supporting documentation.

Recovery Workflow: Starting the project with overly detailed documentation can slow down the entire process. Overcome planning inertia by starting with high-level incident response plans in a flowchart format. For examples and additional information, see XMPL Medical’s Recovery Workflows.

Recovery Procedures (Systems Recovery Playbook): For each step in the high-level flowchart, create recovery procedures where necessary using additional flowcharts, checklists, and diagrams as appropriate. Leverage Info-Tech’s Systems Recovery Playbook example as a starting point.

Additional Reference Documentation: Reference existing IT documentation, such as network diagrams and configuration documents, as well as more detailed step-by-step procedures where necessary (e.g. vendor documentation), particularly where needed to support alternate recovery staff who may not be as well versed as the primary system owners.

Info-Tech Insight

Organizations that use flowcharts, checklist, and diagrams over traditional, dense DRP manuals are far more likely to meet their RTOs/RPOs because their documentation is more usable and easier to maintain.

Use a DRP summary document to satisfy executives, auditors, and clients

Stakeholders don’t have time to sift through a pile of paper. Summarize your overall continuity capabilities in one, easy-to-read place.

DRP Summary Document

• Summarize BIA results
• Summarize DR strategy (including DR sites)
• Summarize backup strategy
• Summarize testing and maintenance plans

Follow Info-Tech’s methodology to make DRP documentation efficient and effective

Follow XMPL Medical’s journey through DR documentation

CASE STUDY

Industry Healthcare
Source Created by amalgamating data from Info-Tech’s client base

Streamline your documentation and maintenance process by following the approach outlined in XMPL Medical’s journey to an end-to-end DRP.

Outline of the Disaster Recovery Plan

XMPL’s disaster recovery plan includes its business impact analysis and a subset of tier 1 and tier 2 patient care applications.

Its DRP includes incident response flowcharts, system recovery checklists, and a communication plan. Its DRP also references IT operations documentation (e.g. asset management documents, system specs, and system configuration docs), but this material is not published with the example documentation.

Resulting Disaster Recovery Plan

XMPL’s DRP includes actionable documents in the form of high-level disaster response plan flowcharts and system recovery checklists. During an incident, the DR team is able to clearly see the items for which they are responsible.

Disaster Recovery Plan

• Recovery Workflow
• Business Impact Analysis
• DRP Summary
• System Recovery Checklists
• Communication, Assessment, and Disaster Declaration Plan

Info-Tech Best Practice

XMPL Medical’s disaster recovery plan illustrates an effective DRP. Model your end-to-end disaster recovery plan after XMPL’s completed templates. The specific data points will differ from organization to organization, but the structure of each document will be similar.

Model your disaster recovery documentation off of our example

CASE STUDY

Industry Healthcare
Source Created by amalgamating data from Info-Tech’s client base

Recovery Workflow:

• Recovery Workflows (PDF, VSDX)

Recovery Procedures (Systems Recovery Playbook):

• DR Notification, Assessment, and Disaster Declaration Plan
• Systems Recovery Playbook
• Network Topology Diagrams

Additional Reference Documentation:

• DRP Workbook
• Business Impact Analysis
• DRP Summary Document

Use Info-Tech’s DRP Maturity Scorecard to evaluate your progress

Info-Tech offers various levels of support to best suit your needs

Diagnostics and consistent frameworks used throughout all four options

Document and Maintain Your Disaster Recovery Plan – Project Overview

	1. Streamline DRP Documentation	2. Select the Optimal DRP Publishing Strategy	3. Keep Your DRP Relevant
Best-Practice Toolkit	1.1 Start with a recovery workflow 1.2 Create supporting DRP documentation 1.3 Write the DRP summary	2.1 Create Committee Profiles	3.1 Build Governance Structure Map 3.2 Create Committee Profiles
Guided Implementations	Review Info-Tech’s approach to DRP documentation. Create a high-level recovery workflow. Create supporting DRP documentation. Write the DRP summary.	Identify criteria for selecting a DRP publishing strategy. Select a DRP publishing strategy. Optional: Select requirements for a BCM tool and issue an RFP. Optional: Review responses to RFP.	Learn best practices for integrating DRP maintenance into day-to-day IT processes. Learn best practices for DRP-focused reviews.
Onsite Workshop	Module 1: Streamline DRP documentation	Module 2: Select the optimal DRP publishing strategy	Module 3: Learn best practices for keeping your DRP relevant
	Phase 1 Outcome: A complete end-to-end DRP	Phase 2 Outcome: Selection of a publishing and management tool for your DRP documentation	Phase 3 Outcome: Strategy for maintaining your DRP documentation

Workshop Overview

Contact your account representative or email Workshops@InfoTech.com for more information.

Workshop Goal: Learn how to document and maintain your DRP.

Use these icons to help direct you as you navigate this research

Use these icons to help guide you through each step of the blueprint and direct you to content related to the recommended activities.

This icon denotes a slide where a supporting Info-Tech tool or template will help you perform the activity or step associated with the slide. Refer to the supporting tool or template to get the best results and proceed to the next step of the project.

This icon denotes a slide with an associated activity. The activity can be performed either as part of your project or with the support of Info-Tech team members, who will come onsite to facilitate a workshop for your organization.

Phase 1: Streamline DRP Documentation

Step 1.1: Start with a recovery workflow

This step will walk you through the following activities:

• Review a model DRP.
• Review your recovery workflow.
• Identify documentation required to support the recovery workflow.

This step involves the following participants:

• DRP Owner
• System SMEs
• Alternate DR Personnel

Outcomes of this step

• Understanding the visual-based, concise approach to DR documentation.
• Creating a recovery workflow that provides a roadmap for coordinating incident response and identifying required supporting documentation.

Info-Tech Insights

A DRP is a collection of procedures and supporting documents that allow an organization to recover its IT services to minimize system downtime for the business.

1.1 — Start with a recovery workflow to ensure a coordinated response and identify required supporting documentation

The recovery workflow clarifies your DR strategy and ensures the DR team is on the same page.

“We use flowcharts for our declaration procedures. Flowcharts are more effective when you have to explain status and next steps to upper management.” (Assistant Director-IT Operations, Healthcare Industry)

Review business impact analysis (BIA) results to plan your recovery workflow

The BIA defines system criticality from the business’s perspective. Use it to guide system recovery order.

Specifically, review the following from your BIA:

• The list of tier 1, 2, and 3 applications. This will dictate the recovery order in your recovery workflow.
• Application dependencies. This will outline what needs to be included as part of an application recovery workflow.
• The recovery time objective (RTO) and recovery point objective (RPO) for each application. This will also guide the recovery, and enable you to identify gaps where the recovery workflow does not meet RTOs and RPOs.

CASE STUDY: The XMPL DRP documentation is based on this Business Impact Analysis Tool.

Haven’t conducted a BIA? Use Info-Tech’s streamlined approach.

Info-Tech’s publication Create a Right-Sized Disaster Recovery Plan takes a very practical approach to BIA work. Our process gives IT leaders a mechanism to quickly get agreement on system recovery order and DR investment priorities.

Conduct a tabletop planning exercise to determine your recovery workflow

1.1.1 Tabletop Planning Exercise

1. Define a scenario to drive the tabletop planning exercise:
   • Use a scenario that forces a full failover to your DR environment, so you can capture an end-to-end recovery workflow.
   • Avoid scenarios that impact health and safety such as tornados or a fire. You want to focus on IT recovery.
   • Example scenarios: Burst water pipe that causes data-center-wide damage or a gas leak that forces evacuation and power to be shut down for at least two days.

Note: You may have already completed this exercise as part of Create a Right-Sized Disaster Recovery Plan.

Info-Tech Insight

Use scenarios to provide context for DR planning, and to test your plans, but don’t create a separate plan for every possibility.

The high-level recovery plan will be the same whether the incident is a fire, flood, or tornado. While there might be some variances and outliers, these scenarios can be addressed by adding decision points and/or separate, supplementary instructions.

Walk through the scenario and capture the recovery workflow

2. Capture the following information for tier 1, tier 2, and tier 3 systems:
   1. On white cue cards, record the steps and track start and end times for each step (where 00:00 is when the incident occurred).
   2. On yellow cue cards, document gaps in people, process, and technology requirements to complete the step.
   3. On red cue cards, indicate risks (e.g. no backup person for a key staff member).

Note:

• Ensure the language is sufficiently genericized (e.g. refer to events, not specifically a burst water pipe).
• Review isolated failures (e.g. hardware, software). Typically, the recovery procedure documented for individual systems covers the essence of the recovery workflow whether it’s just the one system that failed or it’s part of a site-wide recovery.

Note: You may have already completed this exercise as part of Create a Right-Sized Disaster Recovery Plan.

Document your current-state recovery workflow based on the results of the tabletop planning

After you finish the tabletop planning exercise, the steps on the set of cue cards define your recovery workflow. Capture this in a flowchart format.

Use the sample DRP to guide your own flowchart. Some notes on the example are:

• XMPL’s Incident Management to DR flowchart shows the connection between its standard Service Desk processes and DR processes.
• XMPL’s high-level workflows outline its recovery of tier 1, 2, and 3 systems.
• Where more detail is required, include links to supporting documentation. In this example, XMPL Medical includes links to its Systems Recovery Playbook.

This sample flowchart is included in XMPL Recovery Workflows.

Step 1.2: Create Supporting DRP Documentation

This step will walk you through the following activities:

• Create checklists for your playbook.
• Document more complex procedures with flowcharts.
• Gather and/or write network topology diagrams.
• Compile a contact list.
• Ensure there is enough material for backup personnel.

This step involves the following participants:

• DRP Owner
• System SMEs
• Backup DR Personnel

Outcomes of this step

• Actionable supporting documentation for your disaster recovery plan.
• Contact list for IT personnel, business personnel, and vendor support.

1.2 — Create supporting documentation for your disaster recovery plan

Now that you have a high-level incident response plan, collect the information you need for executing that plan.

Info-Tech Insight

Write for your audience. The playbook is for IT; include only the information they need to execute the plan. DRP summaries are for executives and auditors; do not include information intended for IT. Similarly, your disaster recovery plan is not for business units; keep BCP content out of your DRP.

Use checklists to streamline step-by-step procedures

Checklists are ideal when staff just need a reminder of what to do, not how to do it.

XMPL Medical used its high-level flowcharts as a roadmap for creating its Systems Recovery Playbook.

• Since its Playbook is intended for experienced IT staff, the writing style in the checklists is concise. XMPL includes links to reference material to support recovery, especially for alternate staff who might need additional instruction.
• XMPL includes key parameters (e.g. IP addresses) rather than assume those details would be memorized, especially in a stressful DR scenario.
• Similarly, include links to other useful resources such as VM templates.

Included in the XMPL Systems Recovery Playbook are checklists for recovering XMPL’s virtual desktop infrastructure, mission-critical applications, and core infrastructure components.

Use flowcharts to document processes with concurrent tasks not easily captured in a checklist

Recovery procedures can consist of flowcharts, checklists, or both, as well as diagrams. The main goal is to be clear and concise.

• XMPL Medical created a flowchart to capture its phone services recovery procedure to capture concurrent tasks.
• Additional instructions, where required, could still be captured in a Playbook checklist or other supporting documentation.
• The flowchart could have also included key settings or other details as appropriate, particularly if the DR team chose to maintain this recovery procedure just in a flowchart format.

Included in the XMPL DR documentation is an example flowchart for recovering phone systems. This flowchart is in Recovery Workflows.

Reference this blueprint for more SOP flowchart examples: Create Visual SOP Documents that Drive Process Optimization, Not Just Peace of Mind

Use topology diagrams to capture network layout, integrations, and system information

Topology diagrams, key checklists, and configuration settings are often enough for experienced networking staff to carry out their DR tasks.

• XMPL Medical includes these diagrams with its DRP. Instead of recreating these diagrams, the XMPL Medical DR Manager asked their network team for these diagrams:
  • Primary data center diagram
  • DR site diagram
  • High-level network diagrams
• Often, organizations already have network topology diagrams for reference purposes.

“Our network engineers came to me and said our standard SOP template didn't work for them. They're now using a lot of diagrams and flowcharts, and that has worked out better for them.” (Assistant Director-IT Operations, Healthcare Industry)

You can download a PDF and a VSD version of these Data Center and Network Diagrams from Info-Tech’s website.

Create a list of organizational, IT, and vendor contacts that may be required to assist with recovery

If there is something strange happening to your IT infrastructure, who you gonna call?

Many DR managers have their team on speed dial. However, having the contact info of alternate staff, BCP leads, and vendors can be very helpful during a disaster. XMPL Medical lists the following information in its DRP Workbook:

• The DR Teams, SMEs critical to disaster recovery, their backups, and key contacts (e.g. BC Management team leads, vendor contacts) that would be involved in:
  • Declaring a disaster.
  • Coordinating a response at an organizational level.
  • Executing recovery.
• The people that have authority to declare a disaster.
• Each person’s spending authority.
• The rules for delegating authority.
• Primary and alternate staff for each role.

Confirm with your DR team that you have all of the documentation that you need to recover during a disaster

DISCUSS: Is there enough information in your DRP for both primary and backup DR personnel?

• Is it clear who is responsible for each DR task, including notification steps?
• Have alternate staff for each role been identified?
• Does the recovery workflow capture all of the high-level steps?
• Is there enough documentation for alternate staff (e.g. network specs)?

Step 1.3: Write the DRP Summary

This step will walk you through the following activities:

• Write a DRP summary document.

This step involves the following participants:

• DRP Owner

Outcomes of this step

• High-level outline of your DRP capabilities for stakeholders such as executives, auditors, and clients.

Summarize your DR capabilities using a DRP summary document

1.3.1 DRP Summary Document

The sample included on Info-Tech’s website is customized for the XMPL Medical Case Study – use the download as a starting point for your own summary document.

• Business Impact Analysis
• DRP Recovery Workflows
• Supporting Documentation
• Outputs from Reduce Costly Downtime Through DR Testing

DRP Summary Document

XMPL’s DRP Summary is organized into the following categories:

• DR requirements: This includes a summary of scope, business impact analysis (BIA), risk assessment, and high-level RTOs and achievable RTOs.
• DR strategy: This includes a summary of XMPL’s recovery procedures, DR site, and backup strategy.
• Testing and maintenance: This includes a summary of XMPL’s DRP testing and maintenance strategy.

Be transparent about existing business risks in your DRP summary

The DRP summary document is business facing. Include information of which business leaders (and other stakeholders) need to be aware.

• Discrepancies between desired and achievable RTOs? Organizational leadership needs to know this information. Only then can they assign the resources and budget that IT needs to achieve the desired DR capabilities.
• What is the DRP’s scope? XMPL Medical lists the IT components that will be recovered during a disaster, and components which will not. For instance, XMPL’s DRP does not recover medical equipment, and XMPL has separate plans for business continuity and emergency response coordination.

The above table to is a snippet from the XMPL DR Summary Document (section 2.1.3.2).

In the example, the DR team is unable to recover tier 1, 2, and 3 systems within the desired RTO. As such, they clearly communicate this information in the DRP summary, and include action items to address these gaps.

Phase 2: Select the Optimal DRP Publishing Strategy

Step 2.1: Select a DRP Publishing Strategy

This step will walk you through the following activities:

• Select criteria for assessing DRP tools.
• Evaluate categories for DRP tools.
• Optional: Write an RFP for a BCM tool.

This step involves the following participants:

• DRP Owner

Outcomes of this step

• Identified strategies for publishing your DRP (i.e. making it available to your DR team).

Info-Tech Insights

Diversify your publishing strategy to ensure you can access your DRP in a disaster. For example, if you are using a BCM tool or SharePoint Online as your primary documentation repository, also push the DRP to your DR team’s smartphones as a backup in case the disaster affects internet access.

2.1 — Select a DR publishing and document management strategy that fits your organization

Publishing and document management considerations:

Portability/External Access: Assume your primary site is down and inaccessible. Can you still access your documentation? As shown in this chart, traditional strategies of either keeping a copy at another location (e.g. at the failover site) or with staff (e.g. on a USB drive) still dominate, but these aren’t necessarily the best options.	➔	Note: Percentages total more than 100% due to respondents using more than one portability strategy. (Source: Info-Tech Research Group, N=118)
Maintainability/Usability: How easy is it to create, update, and use the documentation? Is it easy to link to other documents as shown in the flowchart and checklist examples? Is there version control? Lack of version control can create a maintenance nightmare as well as issues in a crisis if staff are questioning whether they have the right version.
Cost/Effort: Is the cost and effort appropriate? For example, a large enterprise may need a formal solution (e.g. DRP tools or SharePoint), but the cost might be hard to justify for a smaller company.

Pros and cons of potential strategies

This section will review the following strategies, their pros and cons, and how they meet publishing and document management requirements:

• DRP tools (e.g. eBRP, Recovery Planner, LDRPS)
• In-house solutions combining SharePoint and MS Office (or equivalent)
• Wiki site
• “Manual” approaches such as storing documents on a USB drive

Avoid 42 hours of downtime due to a non-diversified publishing strategy

CASE STUDY

Industry Municipality
Source Interview

Situation

• A municipal government has recently completed an end-to-end disaster recovery plan.
• The team is feeling good about the fact that they were able to identify:
  • Relative criticality of applications.
  • Dependencies for each application.
  • Incident response plans for the current state and desired state.
  • System recovery procedures.

Challenge

• While the DR plan itself was comprehensive, the team only published the DR onto the government’s network drives.
• A power generation issue caused power to be shut down, which in turn cascaded into downtime for the network.
• Once the network was down, their DRP was inaccessible.

Insights

• Each piece of documentation that was created could have contributed to recovery efforts. However, because they were inaccessible, there was a delayed response to the incident. The result was 42 hours of downtime for end users.
• Having redundant publishing strategies is just like having redundant IT infrastructure. In the event of downtime, not only do you need to have DR documentation, but you also need to make sure that it is accessible.

Decide on a DR publishing strategy by looking at portability, maintainability, cost, and required effort

Use the information included in Step 2.1 to guide your analysis of DRP publishing solutions.

The tool enables you to compare two possible solutions based on these key considerations discussed in this section:

• Portability/external access
• Maintainability/usability
• Cost
• Effort

The right choice will depend on factors such as current in-house tools, maturity around document management, the size of your IT department, and so on.

For example, a small shop may do very well with the USB drive strategy, whereas a multi-national company will need a more formal strategy to manage consistent DRP distribution.

The DRP Publishing and Management Solution Evaluation Tool helps you to evaluate the tools included in this section.

Don’t think of a business continuity management (BCM) tool as a silver bullet; know what you’re getting out of it

Portability/External Access:

• Pros: Typically a SaaS option provides built-in external access with appropriate security and user administration to vary access rights.
• Cons: Degree of external access is often dependent on the vendor.

Maintainability/Usability:

• Pros: Built-in templates encourage consistency and guide initial content development by indicating what details need to be captured.
• Pros: Built-in document management (e.g. version control, metadata support), centralized access/navigation to required documents, and some automation (e.g. update contacts throughout the system).
• Cons: Not a silver bullet. You still have to do the work to define and capture your processes.
• Cons: Requires end-user and administrator training.

Cost/Effort:

• Pros: For large enterprises, the convenience of built-in document management and templates can outweigh the cost.
• Cons: Expect leading DRP tools to cost $20K or more per year.

About this approach:
BCM tools are solutions that provide templates, tools, and document management to create BC and DR documentation.

Info-Tech Insight

The business case for a BCM tool is built by answering the following questions:

• Will the BCM tool solve an unmet need?
• Will the tool be more effective and efficient than an in-house solution?
• Will the solution provide enhanced capabilities that an in-house solution cannot provide?

If you cannot get a satisfactory answer to each of these questions, then opt for an in-house solution.

“We explored a DRP tool, and it was something we might have used, but it was tens of thousands of pounds per year, so it didn’t stack up financially for us at all.” (Rik Toms, Head of Strategy – IP and IT, Cable and Wireless Communications)

For in-house solutions, leverage tools such as SharePoint to provide document management capabilities

Portability/External Access:

• Pros: SharePoint is commonly web-enabled and supports external access with appropriate security and user administration.
• Cons: Must be installed at redundant sites or be cloud-based to be effective in a crisis that takes down your primary data center.

Maintainability/Usability:

• Pros: Built-in document management (e.g. version control, metadata support) as well as centralized access/navigation to required documents.
• Pros: No tool learning curve – SharePoint and MS Office would be existing solutions already used on a daily basis.
• Cons: No built-in automation (e.g. automated updates to contacts throughout the system).
• Cons: Consistency depends on creating templates and implementing processes for document updates, review, and approval.

Cost/Effort:

• Pros: Using existing tools, so this is a sunk cost in terms of capex.
• Cons: Additional effort required to create templates and manage the documentation library.

About this approach:
DRPs and SOPs most often start as MS Office documents, even if there is a DRP tool available. For organizations that elect to bypass a formal DRP tool, and most do, the biggest gap they have to overcome is document management.

Many organizations are turning to SharePoint to meet this need. For those that already have SharePoint in place, it makes sense to further leverage SharePoint for DR documentation and day-to-day SOPs.

For SharePoint to be a practical solution, the documentation must still be accessible if the primary data center is down, e.g. by having redundant SharePoint instances at multiple in-house locations, or using a cloud-based SharePoint solution.

“Just about everything that a DR planning tool does, you can do yourself using homegrown solutions or tools that you're already familiar with such as Word, Excel, and SharePoint.” (Allen Zuk, President and CEO, Sierra Management Consulting)

A healthcare company uses SharePoint as its DRP and SOP documentation management solution

CASE STUDY Healthcare

• This organization is responsible for 50 medical facilities across three states.
• It explored DRP tools, but didn’t find the right fit, so it has developed an in-house solution based in SharePoint. While DRP tools have improved, the organization no longer needs that type of solution. Its in-house solution is meeting its needs.
• It has SharePoint instances at multiple locations to ensure availability if one site is down.

Documentation Strategy

• Created an IT operations library in SharePoint for DR and SOPs, from basic support to bare-metal restore procedures.
• SOPs are linked from SharePoint to the virtual help desk for greater accessibility.
• Where practical, diagrams and flowcharts are used, e.g. DR process flowcharts and network services SOPs dominated by diagrams and flowcharts.

Management Strategy

• Directors and the CIO have made finishing off SOPs their performance improvement objective for the year. The result is staff have made time to get this work done.
• Status updates are posted monthly, and documentation is a regular agenda item in leadership meetings.
• Regular tabletop testing validates documentation and ensures familiarity with procedures, including where to find required information.

Results

• Dependency on a few key individuals has been reduced. All relevant staff know what they need to do and where to access required documentation.
• SOPs are enabling DR training as well as day-to-day operations training for new staff.
• The organization has a high confidence in its ability to recovery from a disaster within established timelines.

Explore using a wiki site as an inexpensive alternative to SharePoint and other content management solutions

Portability/External Access:

• Pros: Wiki sites can support external access as with any web solution.
• Cons: Must be installed at redundant sites, hosted, or cloud-based to be effective in a crisis that takes down your primary data center.

Maintainability/Usability:

• Pros: Built-in document management (version control, metadata support, etc.) as well as centralized access/navigation to required information.
• Pros: Authorized users can make updates dynamically, depending on how much restriction you have on the site.
• Cons: No built-in automation (e.g. automated updates to contacts throughout the system).
• Cons: Consistency depends on creating templates and implementing processes for document updates, review, and approval.

Cost/Effort:

• Pros: An inexpensive option compared to traditional content management solutions such as SharePoint.
• Cons: Learning curve if wikis are new to your organization.

About this approach:
Wiki sites are websites where users collaborate to create and edit the content. Wikipedia is an example.

While wiki sites are typically used for collaboration and dynamic content development, the traditional collaborative authoring model can be restricted to provide structure and an approval process.

Several tools are available to create and manage wiki sites (and other collaboration solutions), as outlined in the following research:

• Modernize Communications and Collaboration infrastructure
• Vendor Landscape: Collaboration Platforms

Info-Tech Insight

If your organization is not already using wiki sites, this technology can introduce a culture shock. Start slow by using a wiki site within a specific department or for a particular project. Then evaluate how well your staff adapt to this technology as well as its potential effectiveness in your organization. Refer to our collaboration strategy research for additional guidance.

For small IT shops, distributing documentation to key staff (e.g. via a USB drive) can still be effective

Portability/External Access:

• Pros: Appropriate staff have the documentation with them; there is no need to log into a remote site or access a tool to get at the information.
• Cons: Relies on staff to be diligent about ensuring they have the latest documentation and keep it with them (not leave it in their desk drawer).

Maintainability/Usability:

• Pros: With this strategy, MS Office (or equivalent) is used to create and maintain the documentation, so there is no learning curve.
• Pros: Simple, straightforward methodology – keep the master on a network drive, and download a copy to your USB drive.
• Cons: No built-in automation (e.g. automated updates to contact information) or document management (e.g. version control).
• Cons: Consistency depends on creating templates and implementing rigid processes for document updates, review, and approval.

Cost/Effort:

• Pros: Little to no cost and no tool management required.
• Cons: “Manual” document management requires strict attention to process for version control, updates, approvals, and distribution.

About this approach:
With this strategy, your ERT and key IT staff keep a copy of your DRP and relevant documentation with them (e.g. on a USB drive). If the primary site experiences a major event, they have ready access to the documentation.

Fifty percent of respondents in our recent survey use this strategy. A common scenario is to use a shared network drive or a solution such as SharePoint as the master centralized repository, but distribute a copy to key staff.

Info-Tech Insight

This approach can have similar disadvantages as using hard copies. Ensuring the USB drives are up to date, and that all staff who might need access have a copy, can become a burdensome process. More often, USB drives are updated periodically, so there is the risk that the information will be out of date or incomplete.

Avoid extensive use of paper copies of DR documentation

DR documents need to be easy to update, accessible from anywhere, and searchable. Paper doesn’t meet these needs.

Portability/External Access:

• Pros: Does not rely on technology or power.
• Cons: Requires all staff who might be involved in a DR to have a copy, and to have it with them at all times, to truly have access at any time from anywhere.

Maintainability/Usability:

• Pros: In terms of usability, again there is no dependence on technology.
• Cons: Updates need to be printed and distributed to all relevant staff every time there is a change to ensure staff have access to the latest, most accurate documentation if a disaster occurred. You can’t schedule disasters, so information needs to be current all the time.
• Cons: Navigation to other information is manual – flipping through pages, etc. No searching or hyperlinks.

Cost/Effort:

• Pros: No technology system to maintain, aside from what you use for printing.
• Cons: Printing expenses are actually among the highest incurred by organizations, and this adds to it.
• Cons: Labor intensive due to need to print and physically distribute documentation updates.

About this approach:
Traditionally DRPs are printed and distributed to managers and/or kept in a central location at both the primary site and a secondary site. In addition, wallet cards are distributed that contain key information such as contact numbers.

A wallet card or even a few printed copies of your high-level DRP for general reference can be helpful, but paper is not a practical solution for your overall DR documentation library, particularly when you include SOPs for recovery procedures.

One argument in favor of paper is there is no dependency on power during a crisis. However, in a power outage, staff can use smartphones and potentially laptops (with battery power) to access electronically stored documentation to get through first response steps. In addition, your DR site should have backup power to be an appropriate recovery site.

Optional: Partial list of BCM tool vendors

The list is only a partial list of BCM tool vendors. The order in which vendors are presented, and inclusion in this list, does not represent an endorsement.

Optional: Use our list of requirements as a foundation for selecting and reviewing BCM tools

If a BCM tool is the best option for your environment, expedite the evaluation process with our BCM Tool – RFP Selection Criteria.

Through advisory services, workshops, and consulting engagements, we have created this BCM Tool Requirements List. The featured requirements includes the following categories:

1. Integrations
2. Planning and Monitoring
3. Administration
4. Architecture
5. Security
6. Support and Training

This BCM Tool – RFP Selection Criteria can be appended to an RFP. You can leverage Info-Tech’s RFP Template if your organization does not have one.

Info-Tech can write full RFPs

As part of a consulting engagement, Info-Tech can write RFPs for BCM tools and provide a customized scoring tool based on your environment’s unique requirements.

Phase 3: Keep Your DRP Relevant Through Maintenance Best Practices

Step 3.1: Integrate DRP maintenance into core IT processes

This step will walk you through the following activities:

• Integrate DRP maintenance with Project Management.
• Integrate DRP considerations into Change Management.
• Integrate with Performance Management.

This step involves the following participants:

• DRP Owner
• Head of Project Management Office
• Head of Change Advisory Board
• CIO

Outcomes of this step

• Updated project intake form.
• Updated change management practice.
• Updated performance appraisals.

3.1 — Incorporate DRP maintenance into core IT processes

Focusing on these three processes will help ensure that your plan stays current, accurate, and usable.

Info-Tech Best Practice

Prioritize quick wins that will have large benefits. The advice presented in this section offers easy ways to help keep your DRP up to date. These simple solutions can save a lot of time and effort for your DRP team as opposed to more intricate changes to the processes above.

Assess how new projects impact service criticality and DR requirements upfront during project intake

Understand the RTO/RPO requirements and IT impacts for new or enhanced services to ensure appropriate provisioning and overall DRP updates.

• Have submitters include service continuity requirements. This information can be inserted into your business impact analysis. Use similar language that you use in your own BIA.
  • The submitter should know how critical the resulting project will be. Any items that the submitter doesn’t know, the Project Steering Committee should investigate.
• Have IT assess the impact on the DRP. The submitter will not know how the DRP will be impacted directly. Ask the project committee to consider how DRP documentation and the DR environment will need to be changed due to the project under consideration.

Note: The goal is not to make DR a roadblock, but rather to ensure project requirements will be met – including availability and DR requirements.

This Project Intake Form asks the submitter to fill out the availability and criticality requirements for the project.

Leverage your change management process to identify required DRP updates as they occur

Avoid the year-end rush to update your DRP. Keeping it up to date as changes occur saves time in the long run and ensures your plan is accurate when you need it.

• As part of your change management process, identify potential updates to:
  • System documentation (e.g. configuration settings).
  • Recovery procedures (e.g. if a system has been virtualized, that changes the recovery procedure).
  • Your DR environment (e.g. system configuration updates for standby systems).
• Keep track of how often a system has changed. Relevant DRP documentation might be due for a deeper review:
  • After a system has been changed ten times (even from routine changes), notify your DRP Manager to flag the relevant DRP documentation for review.
  • As part of formal DRP reviews, pay closer attention to DRP documentation for the flagged systems.

This template asks the submitter to fill out the availability and criticality requirements for the project.

For change management best practices beyond DRP considerations, please see Optimize Change Management.

Integrate documentation into performance measurement and performance management

Documentation is a necessary evil – few like to create it and more immediate tasks take priority. If it isn’t scheduled and prioritized, it won’t happen.

“Our directors and our CIO have tied SOP work to performance evaluations, and SOP status is reviewed during management meetings. People have now found time to get this work done.” (Assistant Director – IT Operations, Healthcare Industry)

Step 3.2: Conduct an Annual Focused Review

This step will walk you through the following activities:

1. Identify components of your DRP to refresh.
2. Identify organizational changes requiring further focus.
3. Test your DRP and identify problems.
4. Correct problems identified with DRP.

This step involves the following participants:

• DRP Owner
• System SMEs
• Backup DR Personnel

Outcomes of this step

• An actionable, up-to-date DRP.

Info-Tech Insight

Testing is a waste of time and resources if you do not fix what’s broken. Tabletop testing is effective at uncovering gaps in your DR processes, but if you don’t address those gaps, then your DRP will still be unusable in a disaster.

Set up a safety net to capture changes that slipped through the cracks with a focused review process

Evaluate documentation supporting high-priority systems, as well as documentation supporting IT systems that have been significantly changed.

• Ideally you’re maintaining documentation as you go along. But you need to have an annual review to catch items that may have slipped through.
• Don’t review everything. Instead, review:
  • IT systems that have had 10+ changes: small changes and updates can add up over time. Ensure:
    • The plans for these systems are updated for changes (e.g. configuration changes).
    • SMEs and backup personnel are familiar with the changes.
  • Tier 1 / Gold Systems: Ensure that you can still recover tier 1 systems with your existing DRP documentation.
• Track documentation issues that you discovered with your ticketing system or service desk tool to ensure necessary documentation changes are made.

1. Annual Focused Review
2. Tier 1 Systems
3. Significantly Changed Systems
4. Organizational Changes

Identify larger changes, both organizational and within IT, that necessitate DRP updates

During your focused review, consider how organizational changes have impacted your DRP.

The COBIT 5 Enablers provide a foundation for this analysis. Consider:

• Changes in regulatory requirements: Are there new requirements for IT that are not reflected in your DRP? Is the organization required to comply with any additional regulations?
• Changes to organizational structures, business processes, and how employees work: Can employees still be productive once tier 1 services are restored or have RTOs changed? Has organizational turnover impacted your DRP?
• SMEs leaving or changing roles: Can IT still execute your DRP? Are there still people for all the key roles?
• Changes to IT infrastructure and applications: Can the business still access the information they need during a disaster? Is your BIA still accurate? Do new services need to be considered tier 1?

Info-Tech Best Practice

COBIT 5 Enablers
What changes need to be reflected in your DRP?

Create a plan during your annual focused review to test your DRP throughout the year

Regardless of your documentation approach, training and familiarity with relevant procedures is critical.

• Start with tabletop exercises and progress to technology-based testing (simulation, parallel, and full-scale testing).
• Ask staff to reference documentation while testing, even if they do not need to. This practice helps to confirm documentation accuracy and accessibility.
• Incorporate cross-training in DR testing. This gives important experience to backup personnel and will further validate that documents are complete and accurate.
• Track any discovered documentation issues with your ticketing system or project tracking tools to ensure necessary documentation changes are made.

Example Test Schedule:

1. Q1: Tabletop testing shadowed by backup personnel
2. Q2: Tabletop testing led by backup personnel
3. Q3: Technology-based testing
4. Annual Focused Review: Review Results

Reference this blueprint for guidance on DRP testing plans: Reduce Costly Downtime Through DR Testing

Appendix A: XMPL Case Study

Follow XMPL Medical’s journey through DR documentation

CASE STUDY

Industry Healthcare
Source Created by amalgamating data from Info-Tech’s client base

Streamline your documentation and maintenance process by following the approach outlined in XMPL Medical’s journey to an end-to-end DRP.

Outline of the Disaster Recovery Plan

XMPL’s disaster recovery plan includes its business impact analysis and a subset of tier 1 and tier 2 patient care applications.

Its DRP includes incident response flowcharts, system recovery checklists, and a communication plan. Its DRP also references IT operations documentation (e.g. asset management documents, system specs, and system configuration docs), but this material is not published with the example documentation.

Resulting Disaster Recovery Plan

XMPL’s DRP includes actionable documents in the form of high-level disaster response plan flowcharts and system recovery checklists. During an incident, the DR team is able to clearly see the items for which they are responsible.

Disaster Recovery Plan

• Recovery Workflow
• Business Impact Analysis
• DRP Summary
• System Recovery Checklists
• Communication, Assessment, and Disaster Declaration Plan

Info-Tech Best Practice

XMPL Medical’s disaster recovery plan illustrates an effective DRP. Model your end-to-end disaster recovery plan after XMPL’s completed templates. The specific data points will differ from organization to organization, but the structure of each document will be similar.

Model your disaster recovery documentation off of our example

CASE STUDY

Industry Healthcare
Source Created by amalgamating data from Info-Tech’s client base

Recovery Workflow:

• Recovery Workflows (PDF, VSDX)

Recovery Procedures (Systems Recovery Playbook):

• DR Notification, Assessment, and Disaster Declaration Plan
• Systems Recovery Playbook
• Network Topology Diagrams

Additional Reference Documentation:

• DRP Workbook
• Business Impact Analysis
• DRP Summary Document

Use our structure to create your practical disaster recovery plan.

Appendix B: Summary, Next Steps, and Bibliography

Insight breakdown

Use visual-based documentation instead of a traditional DRP manual.

• Flowcharts, checklists, and diagrams are more concise, easier to maintain, and more effective in a crisis.
• Write for an IT audience and focus on how to recover. You don’t need 30 pages of fluff describing the purpose of the document.

Create your DRP in layers to keep the work manageable.

• Start with a recovery workflow to ensure a coordinated response, and build out supporting documentation over time.

Prioritize quick wins to make DRP maintenance easier and more likely to happen.

• Incorporate DRP maintenance into change management and project intake procedures to systematically update and refine the DR documentation. Don’t save up changes for a year-end blitz, which turns document maintenance into an onerous project.

Summary of accomplishment

Knowledge Gained

• How to create visual-based DRP documentation
• How to integrate DRP maintenance into core IT processes

Processes Optimized

• DRP documentation creation
• DRP publishing tool selection
• DRP documentation maintenance

Deliverables Completed

• DRP documentation
• Strategy for publishing your DRP
• Modified project-intake form
• Change management checklist for DR considerations

Project step summary

Client Project: Document and Maintain Your Disaster Recovery Plan

• Create a recovery workflow.
• Create supporting DRP documentation.
• Write a summary for your DRP.
• Decide on a publishing strategy.
• Incorporate DRP maintenance into core IT processes.
• Conduct an annual focused review.

Info-Tech Insight

This project has the ability to fit the following formats:

• Onsite workshop by Info-Tech Research Group consulting analysts.
• Do-it-yourself with your team.
• Remote delivery (Info-Tech Guided Implementation).

Related Info-Tech research

Create a Right-Sized Disaster Recovery Plan
Close the gap between your DR capabilities and service continuity requirements.

Reduce Costly Downtime Through DR Testing
Improve the accuracy of your DRP and your team’s ability to efficiently execute recovery procedures through regular DR testing.

Create Visual SOP Documents that Drive Process Optimization, Not Just Peace of Mind
Go beyond satisfying auditors to drive process improvement, consistent IT operations, and effective knowledge transfer.

Prepare for a DRP Audit
Assess your current DRP maturity, identify required improvements, and complete an audit-ready DRP summary document.

Bibliography

A Structured Approach to Enterprise Risk Management (ERM) and the Requirements of ISO 31000. The Association of Insurance and Risk Managers, Alarm: The Public Risk Management Association, and The Institute of Risk Management, 2010.

“APO012: Manage Risk.” COBIT 5: Enabling Processes. ISACA, 2012.

Bird, Lyndon, Ian Charters, Mel Gosling, Tim Janes, James McAlister, and Charlie Maclean-Bristol. Good Practice Guidelines: A Guide to Global Good Practice in Business Continuity. Global ed. Business Continuity Institute, 2013.

COBIT 5: A Business Framework for the Governance and Management of Enterprise IT. ISACA, 2012.

“EDM03: Ensure Risk Optimisation.” COBIT 5: Enabling Processes. ISACA, 2012.

Risk Management. ISO 31000:2009.

Rothstein, Philip Jan. Disaster Recovery Testing: Exercising Your Contingency Plan. Rothstein Associates: 1 Oct. 2007.

Societal Security – Business continuity management systems – Guidance. ISO 22313:2012.

Societal Security – Business continuity management systems – Requirements. ISO 22301:2012.

Understanding and Articulating Risk Appetite. KPMG, 2008.