A disaster recovery plan that has never been tested is not a plan. It is a document that creates confidence without providing the ability to recover.
The most common DR failure mode is not a missing backup or a wrong architecture — it is a runbook that references a component that was renamed six months ago, or a failover procedure that nobody has tried since the original engineer left, or an RTO target that was written by engineering without agreement from the business on what downtime would actually cost.
This guide covers the framework for setting recovery targets that mean something, the four DR patterns and what each costs, and the testing discipline that separates a recovery capability from a recovery plan.
RTO and RPO: Setting Targets That Business Requirements Justify
Before any architecture discussion, RTO and RPO must be agreed with business stakeholders — not set by engineering alone.
Cost of downtime analysis
| System | Revenue impact/hour | Reputational cost | Regulatory cost | Total downtime cost/hour |
|---|---|---|---|---|
| Customer-facing SaaS product | $50,000 | High | SLA penalties | $80,000+ |
| Internal ERP/billing system | $20,000 (delayed revenue recognition) | Medium | Audit findings | $30,000+ |
| Data warehouse/analytics | $0 direct | Low | None | $5,000 (staff idle) |
| Development environment | $0 | None | None | $2,000 (lost productivity) |
Recovery targets must be proportionate to downtime cost. A system costing $80,000/hour of downtime justifies significant DR investment; one costing $2,000/hour does not.
Standard tier classification
| Tier | RTO | RPO | DR pattern | Typical cost (% of primary) |
|---|---|---|---|---|
| 1 — Business critical | < 1 hour | < 15 minutes | Warm standby or active/active | 40–100% |
| 2 — Important | 1–8 hours | < 1 hour | Pilot light | 15–25% |
| 3 — Standard | 8–24 hours | < 8 hours | Backup and restore | 5–10% |
| 4 — Low priority | 24–72 hours | < 24 hours | Cold backup only | < 5% |
The Four DR Patterns
Pattern 1: Backup and Restore
Primary region: Full production environment
↓
Automated backup to S3 / Blob Storage / GCS (daily + transaction logs)
↓
Disaster: Restore backups to new environment in DR region
↓
Environment recreation from IaC: 30–60 minutes
Data restore from backup: 1–4 hours depending on dataset size
DNS cutover: 5 minutes
Total RTO: 2–6 hours
Best for: Tier 3–4 systems. Lowest cost. Highest RTO.
Pattern 2: Pilot Light
Primary region: Full production environment
DR region: Core database running with replication (minutes of lag)
Infrastructure templates ready to deploy
↓ Disaster ↓
Scale up DR region compute: 10–20 minutes
Promote DR database to primary: 5 minutes
Redirect DNS: 2 minutes
Total RTO: 15–30 minutes
Best for: Tier 2 systems with RTO requirements of 1–4 hours.
Pattern 3: Warm Standby
Primary region: Full production environment at 100% capacity
DR region: Reduced-capacity environment running (e.g. 25% of primary)
Database replication active with < 30 second lag
↓ Disaster ↓
Scale up DR region to 100%: 5–10 minutes
DNS failover: 2 minutes
Total RTO: 10–15 minutes
Best for: Tier 1 systems with RTO < 1 hour requirements.
Pattern 4: Multi-Site Active/Active
Region A: 50% of traffic
Region B: 50% of traffic
Database: Multi-region writes with synchronous replication
↓ Region A failure ↓
DNS health check detects failure: < 30 seconds
Traffic automatically shifts to Region B: < 1 minute
No manual intervention required
RTO: < 1 minute | RPO: Near zero
Best for: Revenue-critical, customer-facing systems where any downtime is unacceptable.
Database-Specific DR Considerations
Databases are almost always the longest recovery step. Plan specifically for your database engine:
| Database | Cross-region replication | RPO achievable | Failover mechanism |
|---|---|---|---|
| AWS RDS (Multi-AZ) | Read replicas | Minutes (async) | Manual promote or Aurora Global failover |
| AWS Aurora Global | Global database | < 1 second | Managed failover in < 1 minute |
| PostgreSQL (self-managed) | Streaming replication | Seconds (async) | Manual promote replica |
| MongoDB Atlas | Global clusters | Seconds | Automatic with election |
| Azure SQL | Active geo-replication | Seconds (async) | Failover group automatic |
| Cosmos DB | Multi-region writes | Near-zero | Automatic |
Synchronous replication (zero RPO) imposes write latency equal to the inter-region round-trip time — typically 50–150ms between US and EU regions. For write-heavy workloads, this is a performance trade-off that must be evaluated explicitly.
What DR Tests Must Validate
A DR test that only verifies that backups exist is not a DR test. A real DR test validates:
□ Failure detection: Alert fires within the defined detection SLA
□ Escalation: On-call engineer reached within 5 minutes of alert
□ Runbook execution: Every step executable by someone who was not on the original team
□ Infrastructure recreation: IaC deploys a clean DR environment without errors
□ Data restore: Backup or replication produces a verifiable, complete dataset
□ Application health: All health checks pass after cutover to DR
□ Data integrity: Spot-check of restored data against known-good records
□ External integration: Third-party integrations (payment, email, SSO) function in DR
□ Actual RTO measured: Time from failure declaration to verified recovery vs target
□ Actual RPO measured: Age of oldest unrecoverable data vs target
□ Runbook gaps documented: Steps that were unclear, missing, or wrong
□ Runbook updated: Gaps fixed before test is declared complete
Common DR Failures and How to Prevent Them
| Failure | What actually happened | Prevention |
|---|---|---|
| Backup exists but is not restorable | Backup succeeded but the restored database had corruption | Monthly restoration test to a separate environment; verify row counts and spot-check data |
| IaC deploys but application fails | IaC had hardcoded primary region values (AMI IDs, subnet IDs) | DR-region IaC tested quarterly in isolation |
| RTO target missed by 4× | Nobody had done the restore before; 3 hours of troubleshooting | Annual timed DR test with fresh-participant runbook walkthrough |
| DNS failover worked but users couldn't log in | Auth service pointed to primary region, not included in failover | Complete dependency map before DR architecture; test end-to-end, not just infrastructure |
| DR database weeks behind | Replication lag alert was in the DR region; nobody monitored the DR region | Monitor DR replication lag continuously; alert on lag > RPO threshold |
| Runbook referenced decommissioned component | System was migrated; runbook never updated | Change control requires DR runbook review for any infrastructure change |
Disaster recovery for cloud workloads is achievable at manageable cost — the four patterns cover most requirements, and cloud-native tools reduce the implementation burden significantly compared to traditional data-centre DR. The gap between plans that work and plans that fail on the day is almost always testing cadence and runbook maintenance, not architecture.
For help designing a cloud DR architecture appropriate to your RTO/RPO requirements and budget, or assessing your current DR readiness, see our Cloud Engineering capabilities or get in touch. Our Cloud Migration guide covers the broader architectural decisions that set the foundation for effective DR.