Define recovery as a business operation
A backup job completed last night. Today an operator deletes the wrong customer records. The database process is healthy, replicas are current, and the application returns successful responses. Availability monitoring says little about whether the business can recover the missing state.
Define the recovery point objective, or RPO, as the tolerable loss of data measured against a chosen incident scenario. Define the recovery time objective, or RTO, as the allowed time to restore the agreed service capability. State when the clock starts and what “service restored” includes.
An application that can accept logins but cannot reconcile orders has not necessarily met its business RTO. The drill must test the capability customers rely on, not simply the moment PostgreSQL accepts connections.
Build an independently usable recovery set
PostgreSQL continuous recovery depends on a suitable base backup and the required uninterrupted WAL sequence. Database configuration and surrounding infrastructure also need separate recovery coverage. A replica and a backup solve different failure scenarios. PostgreSQL continuous archiving
Inventory encryption keys, credentials, extensions, database versions, application images, schema migrations, object storage, and network policy. A perfect backup encrypted with an unavailable key is unusable. A database restored successfully into an incompatible application release can still fail every business operation.
Keep recovery access independent enough to survive loss of the production control plane, while restricting it to authorized operators. Test that access during routine drills; an incident is a poor time to discover that the only recovery credential lived on the failed host.
Verify artifacts, then actually restore them
pg_verifybackup checks a base backup against its manifest and performs documented validation steps. Its documentation cautions that verification does not replace a test restore. Use a compatible tool version and understand the limits of the checks performed. PostgreSQL pg_verifybackup
Restore into an isolated environment with outbound side effects disabled. Do not let restored workers send old emails, retry historical payments, or publish stale events. Use explicit deny rules or disconnected test integrations rather than relying only on a developer remembering to disable a feature flag.
Record the artifact IDs, recovery target, selected timeline, tool versions, elapsed stages, and every manual intervention. A drill that succeeds only because an expert remembers an undocumented command is evidence of a runbook gap.
Select the target from evidence
An incident may be detected long after the harmful write. Choose the recovery point from audit evidence, transaction context, and operator review. Restoring to “five minutes ago” is not a strategy if the corruption began yesterday.
When practical, restore candidate points into separate isolated environments and compare domain invariants. Preserve the damaged production state for investigation before replacing it. For selective repair, extract verified records from an isolated recovery and apply a reviewed reconciliation process; whole-cluster recovery is not automatically a safe row-level repair tool.
| Recovery stage | Evidence | Stop condition |
|---|---|---|
| Artifact acquisition | Backup and WAL inventory | Missing required segment or inaccessible key |
| Database recovery | Target reached and consistency checks | Recovery target cannot be established |
| Application startup | Compatible release and schema | Migration or extension mismatch |
| Domain reconciliation | Orders, balances, permissions, object references | Unexplained business discrepancies |
| Traffic release | Side effects and routing reviewed | Duplicate execution or stale access risk |
Reconcile systems that did not roll back
Your payment provider, email service, object store, and downstream consumers may have advanced beyond the restored database. Replaying an outbox from before a payment can issue an already completed operation again unless stable business identities survive the recovery plan.
Compare restored orders with external payment records. Reapply required deletion and access-revocation decisions before exposing data. Verify that database references point to the intended object versions, not merely an object with the same filename.
Treat event consumers explicitly: rebuild disposable projections where possible, reconcile durable external effects, and fence old workers before enabling replacements. Restoring the database while an old worker still writes to another dependency creates a split operating state.
Measure the full drill and use the result
Capture time to detect, authorize recovery, retrieve artifacts, replay WAL, start the application, reconcile, and release traffic. Identify the longest stage and improve that stage first. Faster backup creation does little for RTO if reconciliation takes most of the recovery window.
Use synthetic business markers to estimate the latest recoverable committed state and compare it with the declared RPO. Test missing credentials, unavailable backup locations, and a malformed application release as separate scenarios. Keep the exercise controlled and isolated from real customer effects.
Finish with an evidence package: selected target, measured data loss, elapsed recovery time, failed checks, manual steps, and assigned remediation. Repeat after material changes to storage, encryption, database versions, or deployment topology. Recovery confidence should come from recent demonstrated capability, not the age of a green backup dashboard.