Deployment automation makes moving forward fast. Recovery requires equal attention. The previous artifact may still exist, but that does not mean the previous system can safely return after schemas, messages, caches, configuration, and client behavior have changed.
Define the decision threshold
Before release, choose the signals and time window that trigger a pause or rollback. Use user outcomes—errors, latency, failed transactions, saturation, or data integrity—not only deployment health. Name who can make the decision and avoid requiring a committee while impact grows.
Design compatibility into change
Prefer additive database changes, tolerant readers, and staged removal. Separate schema migration from application cutover when the risk justifies it. Version messages and APIs so old and new components can coexist during a gradual rollout.
For irreversible data transformation, rollback may mean rolling forward with a fix or restoring from a protected point. State that honestly before the change.
“Redeploy the old version” is not a rollback plan until compatibility and state have been tested.
Limit the first consequence
Canary releases, small deployment rings, and feature flags reduce the population affected by a wrong assumption. Keep the control path independent enough that a broken release cannot prevent the flag change or traffic shift needed to recover.
Record the exact artifact, configuration, infrastructure version, and feature state that made up the previous good release. A label such as “last stable” is useful only if it cannot drift.
Practice the reverse path
Exercise rollback in an environment that includes realistic data and dependencies. Measure duration, verify queued work, confirm telemetry, and check that operators have permission. After a production rollback, preserve evidence and fix the recovery path as carefully as the original defect.
Rollback is part of delivery, not an admission that delivery failed. Building it first turns recovery from improvisation into a known engineering operation.