How do you decide what to recover first after a major outage?
The short answer
Decide recovery order by working backwards from the minimum service the business needs, then sorting everything that service depends on so foundations come first: network, identity, core infrastructure and data, then applications and interfaces. Business priority decides what to recover. Dependencies decide the order. Agree both before the outage.
The second restart
Hour one of a major outage. The team restores the database first, because everyone agrees the database is critical. The application still won't start. It needed the identity service, which nobody had brought back yet. Second restart, another hour gone, and the business is still waiting.
This happens because recovery plans are written per application. Each plan works for its own system. In an infrastructure-wide event, somebody has to coordinate hundreds of them, many with changed owners, and nobody holds the order across systems. I stopped believing that recovery can be planned bottom up by stitching application plans together.
Two inputs: priority and dependencies
A recovery sequence needs two inputs, and they come from different people.
- Priority comes from the business. Which services must come back first, and at what minimum level?
- Dependencies come from technology. What does each of those services need running underneath it?
Priority without dependencies restores the right things in the wrong order. Dependencies without priority restore everything, slowly. You need both, agreed before the outage.
Foundations come first, and they are shared
Once priority is clear, sort the dependencies so that nothing comes back before what it needs. In most organizations the order looks like this:
- Network and connectivity
- Identity and access
- Core infrastructure: compute, storage, the data center or cloud region
- Shared platforms: databases, messaging, integration
- Business applications behind the priority services
- Interfaces to partners, suppliers and customers
The first three layers are shared by almost everything. In a large organization's model, the network, the identity provider and the core data centers often support hundreds of other assets. Get them wrong and every downstream plan fails, however good it is.
In a hospital hit by ransomware, that means downtime procedures run on the wards while IT restores clean network segments, then identity, then the patient record database and application, then the lab and imaging interfaces.
When you can't trust the environment
Cyber recovery changes the question. You know how long a full restore takes. How long until you have somewhere trusted to restore into, working access, and backups you are confident in? That time comes before the restore, and it is often the longer part.
If IT can't give a credible recovery time yet, the business still has to decide how to serve customers. Start the workarounds anyway. Use the recovery sequence to show what will come back first once the environment is trusted.
Put your vendors in the sequence
Your service providers sit in the sequence too. A provider that picks up the ticket after 12 hours is still within its agreed 24-hour response time. By then your own recovery target has passed.
When the provider restores its platform, work remains on your side before the business can use it. Agree in advance who coordinates that handover, and add it to the sequence as its own step.
Make the sequence dynamic
A single static priority list breaks when the scenario differs from the one it was written for. In an incident, essential is whatever got hit. Derive the sequence from the dependencies for each scenario: a lost data center needs a different order than a compromised identity provider, and an ERP outage that leaves the plants running blind needs another. The fixed priority list still matters for the day everything gets hit at once.
A practical way to begin
- Pick one scenario, such as ransomware in your main data center or the loss of a cloud region.
- Agree with the business which services must come back first, and at what minimum level.
- List everything those services depend on, including identity, network and vendors.
- Sort the list so every item comes after what it depends on. Mark circular dependencies and decide how to break them.
- Walk through the sequence with IT, the business and key vendors, and record how long each step takes in practice.
How we do this in resimate
resimate builds recovery sequences from your real dependencies, vendors included. Pick a scenario and the minimum services the business needs, and resimate proposes the order: what comes back first, what waits, and where a dependency cycle needs a decision. When the dependencies change, the sequence changes with them. Teams go live in 4 to 6 weeks, on the data they already have.