Do You Have Your Umbrella? (Disaster Recovery and Business Continuity Planning)

Posted
By

In the wake of some extreme weather during 2011 (earthquakes, tsunamis, tornadoes, hurricanes, and mudslides), what better time to review your disaster recovery and business continuity (DR/BC) solution and planning processes?

In some cases, DR/BC planning is a legal or regulatory requirement, but even where it is not, common sense argues for a sound DR/BC plan for any business. Why?

  • For most businesses, the dependency on computer systems, applications, databases, networks and electronic delivery systems increases daily – to the point where the efficiency and productivity of the business would drop precipitously if these tools are not available.
  • As more and more of a business’ computer systems become interconnected, the possibilities of failure in one system rippling across and adversely impacting other systems increases.
  • With the advent of cloud and distributed computing technologies, information systems frequently rely on computing infrastructure that is geographically dispersed, requiring a detailed analysis of the risks associated with service interruption in multiple locations.

What are the key activities and decision points for establishing a sound disaster recovery and business continuity plan? DR/BC planning can be boiled down to three key steps.

  • Develop a DR/BC plan based on business requirements. A DR/BC plan is a detailed description of actions to be taken in the event of a disaster to support the continuity of your business and operations. It addresses the recovery of all resources needed to run your business, including people, systems and data. To inform the development of a DR/BC plan, start with a business impact analysis of each discreet business function – how does this business function support the delivery of your business to customers, what other business functions is it dependent on, what other business functions depend on it, and what resources are required to ensure that that business function can operate. Each business function can then be ranked in order of importance and the appropriate recovery processes and timeframes can be established. Security measures should also be addressed to ensure the safeguarding of the company’s systems and sensitive data even in the event of a disaster.
  • Test the DR/BC plan on a regular basis. Periodic, comprehensive testing the DR/BC plan is critical to its success. Even very thorough business impact analyses often miss critical interdependencies or make assumptions that prove false in real life. Testing your plan – whether through tabletop exercises that “talk” through the steps of activating and executing the plan or full-fledged drills that include activating DR facilities and systems and reconstituting operations – often uncover unexpected circumstances that were not accounted for in the plan or where the procedures outlined in the plan were not adequate to meet the business needs. Document all such findings of what worked well and what did not via a structured post-exercise evaluation and debriefing and use the lessons learned to enhance the plan going forward. Regular testing also provides a means of training (and refreshing) the impacted personnel on DR/BC plan.
  • Update the DR/BC plan as necessary to account for changes in your business or technologies. In order to be effective, the DR/BC plan cannot be developed and then put on a shelf. Given the pace of technology and business process change in today’s competitive marketplace, a good DR/BC plan will remain sound and relevant only if it is viewed as a “living document” – continuously updated to reflect how your business operates today and will need to operate the day after the disaster happens.

How does outsourcing impact DR/BC planning? Outsourcing does not mean that a company needs to accept lower standards for disaster recovery and business continuity. In fact, outsourcing can offer the opportunity to improve a company’s DR/BC solution by using the vendor’s more established DR/BC services and because the outsourcing process itself may shine light on current weaknesses. There are, however, certain complicating factors to consider when defining DR/BC services to be provided by an outsourcing relationship.

  • Will the vendor’s standard DR/BC solution meet your requirements? The question isn’t if an outsourcing vendor offers a DR/BC solution, but whether their standard solution is acceptable to meet your company’s requirements. You should engage the vendor in a frank and open dialog about what your requirements are (yes, you will need to be able to articulate them!) and what their solution provides in order to identify and resolve any gaps (for example, recovery times requirements). Obviously, one key factor to consider — the more customized a solution the more likely that there will be additional costs associated with implementation (including the purchase of additional infrastructure, if needed, which you may decide to purchase yourself or have the vendor provide) and ongoing maintenance / operation. Also, interestingly, we find that vendors are sometimes hesitant to share details of their solution. This is a red flag – if they can’t or won’t clearly describe their solution and be willing to commit it to writing, you may need to consider alternatives to your DR/BC planning.
  • Will you use a third party DR/BC provider for any part of the solution? There are many third parties that specifically offer disaster recovery services, ranging from full service solutions to simply providing reserved facility space that a company can use in the event of a disaster. If you choose to use a third party disaster recovery vendor in addition to an outsourcer for your regular production systems, then consider (and align with both vendors on) where the responsibility hand-offs and demarcations are between the outsourcing vendor and the DR/BC vendor.
  • How will the vendor’s DR/BC Plan integrate with your overall DR/BC Plan? In today’s world of narrowly-scoped outsourcing arrangements and multi-vendor environments, it is often necessary for a company to maintain an overall DR/BC plan to ensure that all aspects of its operations can be recovered in the event of a disaster. Similar to when integrating with a third party DR/BC provider, when integrating the outsourced vendor’s DR/BC solution into your overall DR/BC plan it is absolutely crucial to understand the responsibility hand-offs and demarcations.

Regardless of what DR/BC solution you end up with as part of the outsourcing, make sure it is well described and documented in the contract. Key topics to address, in addition to a general solution description of where and how the services will be recovered, include:

  • Recovery Time Objectives (RTOs) and Recovery Point Objectives (RPOs) for each of the services. These measure, respectively, how quickly the vendor will have the service back up and running and when data is involved how current that data will be when recovered. You may have a single RTO and RPO expectation for all of the outsourced services, or they may range based on the criticality of the service. Regardless – get them documented (and consider whether you want service level commitments tied them).
  • Timeframe for implementing a custom DR/BC Plan. If you do require a customized DR/BC solution, make sure that there is alignment on how quickly the vendor will implement the solution. Clearly this should be prior to the completion of the transition to the services of vendor, but it is important to consider where in the transition timeline this should take place. For example, some companies may prefer to have the backbone of the solution in place prior to the start of the general transition of the services, so that the it can be tested in conjunction with the overall transition.
  • Testing frequency and your ability to participate. As previously mentioned, periodic testing of the DR/BC solution is crucial to both confirming that the solution continues to meet the DR/BC requirements and providing valuable practice to ensure vendor personnel know what to do if a disaster actually occurs. In fact, it is such an important activity that you, as the customer, may want to actual participate in the testing so you can verify for yourself that the solution works.
  • Vendor’s participation in your company’s overall DR/BC testing. If you do have an overall DR/BC plan that requires integration with the outsourced vendor’s solution, then you also might require the vendor to participate in your testing of your own DR/BC plan. This might be separate from the vendor’s testing of their plan – or you might require that they align their tests to occur along with your own tests so that the full, end-to-end recovery process can be reviewed.
  • Integration with Force Majeure provisions. Most contracts include standard force majeure provisions – if a “force majeure event” occurs, there is some level of excused performance for the party affected by the event and (often) termination rights for the other party if the event affects performance for a prolonged period of time. You should be careful, though, that the force majeure provisions do not excuse the vendor from performing the agreed DR/BC services. If this carve-out is not clear, then you run the risk that the force majeure provision may in fact nullify obligation the vendor might have to actually perform the DR/BC services in the event of a disaster that constitutes a force majeure event (I can’t think of any circumstance when this would ever be the intent!).
  • Additional fees, if any, for the DR/BC services. As mentioned above, DR/BC services may incur additional fees – most commonly when the vendor is providing a custom solution. Make sure that these are known and included as part of your business case analysis, and then documented appropriately as part of the charges under the contract.
  • Parameters governing the declaration of a disaster (and proper escalation notifications and communications). Who is responsible for declaring a disaster? What is the communication plan in the event of a disaster? Who (you or the vendor) is responsible for making each of the communications?
  • Service Level or other performance expectations during a disaster. Many vendors will want performance relief in the event of a disaster. If the agreed DR/BC solution does not permit compliance with the steady state service levels, then consider negotiating separate service levels that do reflect the expected level of service during a disaster.

Remember, the ultimate goal of proper disaster recovery and business continuity planning is to permit the recovery of your business by minimizing the loss and downtime sustained by your company’s operations in the event of a disaster. Having a plan that is sound, up-to-date, and ready to be implemented will allow you to start off the new year on the right foot!