The Complete Guide to Deployment Strategies: What You Can Still Undo, and What You Cannot

• Introduction
• 1. Separating deployment from release
  • 1-1. The cost of separating them
  • 1-2. The most common failure
• 2. Reversible and irreversible changes — a triage checklist
  • 2-1. Three tiers
  • 2-2. Things teams misclassify in practice
• 3. The strategy catalogue and what each one assumes
  • 3-1. This is genuinely contested
• 4. What a canary needs to mean anything
  • 4-1. The control must be concurrent
  • 4-2. Derive metrics from SLIs, and limit how many
  • 4-3. Size and duration must be representative
  • 4-4. What a canary actually sells you is error budget
  • 4-5. What a canary cannot catch
• 5. Schema changes — what breaks deployment strategies most often
  • 5-1. Expand, migrate, contract
  • 5-2. If you keep only one rule
  • 5-3. Particularly dangerous DDL
  • 5-4. Down migrations are usually a lie
• 6. Feature flags: a rollback button, or new technical debt?
  • 6-1. What a flag is really worth
  • 6-2. What a flag really costs
  • 6-3. For a flag to be a rollback button
  • 6-4. This is genuinely contested
• 7. Designing the rollback itself
  • 7-1. Rollback is a normal path, not an exception procedure
  • 7-2. Rollback versus roll forward
  • 7-3. Secondary failures caused by the rollback
  • 7-4. After you cross an irreversible boundary
• 8. What to watch and when to stop — deployment gates and SLOs
  • 8-1. Write the abort criteria as numbers before deploying
  • 8-2. Metric lag is the lower bound on step length
  • 8-3. What to gate on, and what not to
  • 8-4. When the gate is too sensitive
• 9. Measuring deployment
  • 9-1. Speed and stability are not a trade-off
  • 9-2. The distortion when metrics become KPIs
• Quiz: Check your understanding
• Wrapping up
• References
• Further reading

Introduction

Articles about deployment strategies usually end as a tool catalogue. This blog already has one of those. Feature Flags and Progressive Delivery, Fully Dissected organises dark launches, canary rollouts and flag management tools technique by technique, and the CI/CD and GitOps posts explain how to assemble a pipeline. I have no intention of rewriting them.

This post re-sorts the same material along exactly one axis: can this change still be undone, or has it already passed the point of no return? Once you can answer that question, the choice of strategy usually settles itself. When you cannot answer it, every tool fails in the same way.

That is why the centre of gravity here is not tooling but data. Teams that boast about zero-downtime deployment come apart, almost without exception, at schema changes and data migrations — precisely the part that rollout tooling does nothing to help with.

1. Separating deployment from release

Plenty of organisations still use the two words interchangeably. But if you cannot separate them, you are left with exactly one way to undo anything.

• Deployment: the event in which new code comes to exist in the production environment. The process is up and passing health checks, but nobody may be traversing that path yet.
• Release: the event in which real user traffic starts flowing through that code path. Traffic percentages, flag switches and routing rules decide it.

Bundle the two events together and there is only one unit of undo: the deployment artefact. When something goes wrong you have to run the pipeline again and deploy the previous image, and that usually takes minutes. Separate them and the unit of undo becomes a traffic percentage or a flag value, which takes seconds.

1-1. The cost of separating them

Separation is not free. Deferring the release means the new path and the old path coexist inside one binary. Code grows, the number of combinations you must test grows, and it becomes unclear which path is actually executing. Section 6 deals with that cost separately.

1-2. The most common failure

The code is separated but the schema is not. The application can be rolled back with a single flag, but an ALTER TABLE that has already run has no flag. No matter how sophisticated the pipeline is, reversibility ends here.

2. Reversible and irreversible changes — a triage checklist

Before you plan a deployment, answer these five questions. If an answer is ambiguous, it is safer to classify the change as irreversible.

Q1. Do the side effects of this change survive outside the process?
    (DB writes, outbound API calls, published messages, email/push, payment capture)
Q2. Can the previous version of the code read the data the new version wrote?
Q3. Is the time to undo shorter than the time to detect the problem?
Q4. Does the act of undoing itself create a new state change?
Q5. Have you published a contract externally? (public API, webhook payload, event schema)
    If you do not know how many consumers exist, the change is already irreversible.

2-1. Three tiers

2-2. Things teams misclassify in practice

• Index creation is usually conditionally reversible. You can drop it, but the locking and I/O surge that happened during creation cannot be undone. What you can reverse is the result, not the outage the process already caused.
• Publishing a message is almost always irreversible. If a consumer already received it, there is no such thing as un-publishing.
• Even a change that looks read-only becomes near-irreversible once it poisons a cache. A wrong value written with a long TTL means users keep seeing the old result long after the code is reverted.

Reversibility is not a property of code. It is a combination of code, data and time. The same code change is reversible while nobody has written anything, and irreversible after a million rows.

3. The strategy catalogue and what each one assumes

Strategies are not good or bad; each one assumes something different. Used without satisfying its assumption, a strategy is just a dangerous deployment with a respectable name.

Recreate    [v1 v1 v1] → (full stop) → [v2 v2 v2]     rollback = redeploy, downtime again
Rolling     [v1 v1 v1] → [v2 v1 v1] → [v2 v2 v1] …    rollback = roll backwards, same duration
Blue-green  [blue v1] and [green v2] side by side → router flip   rollback = flip back, seconds
Canary      [v1 95%] + [v2 5%] → raise the percentage rollback = set to 0%, seconds
Shadow      [v1 100%] + [v2 mirrored traffic, responses discarded] rollback = stop mirroring

• Recreate: assumes planned downtime is acceptable. When two versions genuinely cannot coexist — a bulk migration that needs an exclusive lock, for instance — this is the honest choice. The downside is that rolling back also incurs downtime.
• Rolling: assumes two versions can be alive at the same time safely. That includes bidirectional schema compatibility, session independence and message format compatibility. Using rolling updates without verifying that assumption is the single most common cause of incidents.
• Blue-green: assumes you have the resources to keep two production-sized environments, and that sharing state between them is fine. Compute is easy to duplicate; the database is usually shared. So blue-green makes application rollback a matter of seconds while solving nothing about schema.
• Canary: assumes observability. You must be able to split traffic and to view the metrics of the two populations separately. Section 4 covers this in detail.
• Shadow (traffic mirroring): assumes you can isolate the new version so it produces no side effects. If mirrored traffic actually charges a card or sends an email, that is not shadowing, it is double processing. Excellent for performance and error-rate verification, but since responses are discarded, correctness verification needs a separate comparison mechanism.

3-1. This is genuinely contested

Whether blue-green's resource cost or canary's operational complexity is more expensive is a question teams answer differently. The blue-green camp points out that the switch is atomic and the rollback procedure is simple. The canary camp counters that blue-green exposes 100% of traffic in a single instant and therefore does not actually reduce risk. There are three axes: infrastructure cost, observability maturity and traffic volume. In a service handling a few thousand requests a day a 5% canary is statistically meaningless, and blue-green is the reasonable answer.

Branch strategy carries the same disagreement. Trunk-based development shortens the integration interval and keeps deployment units small, but grows flag debt. Long-lived branches keep the codebase clean but release a large lump at merge time. Rather than one being better, it is a trade between deployment unit size and flag management cost.

4. What a canary needs to mean anything

The Google SRE Workbook defines canarying as a partial and time-limited deployment of a change in a service together with its evaluation, and calls the rest of the fleet the control. Four practical requirements follow directly from that definition.

4-1. The control must be concurrent

The Workbook explicitly warns against before-and-after comparison. The reason is that time is one of the biggest sources of change in observed metrics. Traffic mix and cache hit rate at 10am differ from 11am. So the comparison target is not "yesterday's me" but "the previous version running right now, next door".

4-2. Derive metrics from SLIs, and limit how many

The Workbook asks that metrics "be able to indicate problems in the service" and suggests keeping them to perhaps no more than a dozen. It also requires that metrics be clearly attributable to the change being canaried. Gate on every graph on the company dashboard and unrelated noise will keep halting deployments, until people learn to ignore the gate.

4-3. Size and duration must be representative

A canary must be sizeable and last long enough to be representative of the overall deployment. At the same time its duration has to fit the release cadence. As the Workbook puts it, if you release daily you cannot let a single canary run for a week.

There is arithmetic here that people routinely forget. If the problem you are hunting occurs with probability 0.1% and only 200 requests reached the canary, the expected number of occurrences is 0.2. Seeing nothing is the normal outcome, so this canary should be read not as "passed" but as "not observed". That arithmetic is not in the source above; it is my own added interpretation.

4-4. What a canary actually sells you is error budget

The Workbook's core argument is simple. If the canary population is 5% of the fleet and its error rate is 20%, the overall error rate is 1%. A canary is therefore not a device that removes bugs, but one that shrinks their exposure surface. Seen this way, canary percentage and duration are not arbitrary numbers but values you derive backwards from the error budget burn you are willing to accept. SLI/SLO/Error Budget-based Reliability Engineering and the SLO & Error Budget Calculator help with that calculation.

4-5. What a canary cannot catch

The Workbook is equally clear about the limits. Test environments are not 100% identical to production, and shared failure domains and some stateful interactions only appear at full scale. Connection pool exhaustion, cache stampedes and downstream saturation stay quiet at 5% and blow up at 100%. A passing canary is not proof of safety; it is the absence of obvious failure.

5. Schema changes — what breaks deployment strategies most often

This is the centre of the article. Rolling, blue-green, canary — every one of them assumed that two versions can be alive at once safely. Schema changes attack that assumption directly.

5-1. Expand, migrate, contract

The Parallel Change pattern documented by Martin Fowler is the standard answer. Fowler describes the expand phase as augmenting the interface to support both the old and the new versions, and the migrate phase as updating all clients using the old version to the new version, which can be done incrementally. Only once every usage has been migrated do you perform the contract phase and remove the old version. The pattern is attributed to Joshua Kerievsky.

-- Phase 1, expand: add it nullable. Supplying a default at the same time can, depending
-- on the engine, rewrite the whole table and hold a long lock, so keep them separate.
ALTER TABLE orders ADD COLUMN currency_code text;

-- Phase 2, migrate: the application writes to both columns while reads still use the old one.
-- Historical rows are filled in batches — not all at once, but in resumable key ranges.
UPDATE orders SET currency_code = 'KRW'
 WHERE currency_code IS NULL AND id BETWEEN 1 AND 10000;

-- Phase 3, contract: after confirming every read has moved to the new column,
-- remove the old one days or weeks later, in its own separate deployment.
ALTER TABLE orders DROP COLUMN currency;

5-2. If you keep only one rule

Never put two phases into one deployment. Combine expand and migrate and a rollback leaves old code facing new data; combine migrate and contract and the column you need to roll back to is already gone. There must be at least one stabilisation period between phases, and its length should equal the longest window in which you might still consider a rollback.

5-3. Particularly dangerous DDL

• Adding a NOT NULL constraint: it validates every existing row, so on a large table it holds a long lock. Fill the values from the application first, finish the backfill, and add the constraint last.
• Renaming a column: it looks atomic but is effectively a drop plus an add. Use expand-migrate-contract instead of a rename.
• Changing a type: often requires a rewrite, and the moment old code cannot parse the new type the change is irreversible.
• Creating an index: check whether an online creation option exists and, if so, whether a failed run leaves debris behind.
• Large backfills: they need a batch size, a pause between batches, a resume point and a kill switch. Backfills that create replication lag and take down read replicas are a common incident.

5-4. Down migrations are usually a lie

Many migration tools demand a down script, but once data has been deleted, down restores the schema and not the data. The reversal you can actually trust in production is therefore not a reverse migration but an expand phase designed so that no reversal is needed. If old code runs unchanged on the new schema, reverting the code is enough — and that is the real purpose of the expand phase.

6. Feature flags: a rollback button, or new technical debt?

6-1. What a flag is really worth

A flag's value is that it decouples time-to-undo from pipeline speed. If the pipeline takes 12 minutes, a rollback takes 12 minutes; a flag takes seconds. In an organisation whose detection time is three minutes, that difference cuts outage duration to a quarter.

6-2. What a flag really costs

One flag turns one code path into two. In theory n flags produce 2 to the power of n combinations, and only a tiny fraction of them are ever tested. This is where the bug that only fires under one particular combination of three supposedly independent flags comes from.

Release flag     create the removal ticket and expiry date with the flag   lifetime: days to 2 weeks
Experiment flag  the experiment's end date is the expiry date              lifetime: the experiment
Operational flag kill switch. Accept long life and document it             lifetime: indefinite
Entitlement flag not really a flag — it is a product feature               lifetime: permanent

6-3. For a flag to be a rollback button

• The off path must actually be executing. An old path nobody has run for a month has probably rotted. Exercise both paths periodically in canary or staging.
• A flag only reverses code paths. Data written in the new format while the flag was on stays there when you turn it off. In other words, the moment you turn a flag on, the feature becomes conditionally reversible by the section 2 criteria.
• Define the default for flag evaluation failure. If the system does not fall to the safe side when the flag service stops answering, the flag itself becomes a new single point of failure.

6-4. This is genuinely contested

Whether flags reduce or increase risk is an honest, open argument. The reduce camp cites rollback time and exposure surface. The increase camp argues that abandoned flags become permanent branches that make the code hard to reason about, and that the feeling of "we can always turn it off" makes verification sloppier. In practice the axis of this argument is whether a process exists that forces flag removal. Where it does, the first camp is right about your organisation; where it does not, the second is.

7. Designing the rollback itself

7-1. Rollback is a normal path, not an exception procedure

A rollback that has never been executed is not a plan, it is a hope. Organisations where it actually works execute a rollback at least once during release rehearsal and verify every time that the previous artefact is still in the registry, who holds the permission to run the rollback command, and whether configuration changes revert together with the code.

[Rollback decision checklist — fill this in before the deployment starts]
1. Detect: which metric shows this failure, and within how many minutes?
2. Threshold: at what value do we stop? (fix the number before deploying)
3. Execute: what is the rollback command, who holds the permission, how long does it take?
4. Data: what happens to the data this deployment has already written?
5. Irreversible: what is the compensation procedure for the parts we cannot undo?
6. Notify: who do we tell? (internal, customers, external API consumers)

7-2. Rollback versus roll forward

Rolling back is not always the answer. There are two criteria: whether the previous version was definitely healthy, and whether rolling back is safe with respect to data. If either is in doubt, pushing a fix forward quickly is safer. Roll-forward leans on the optimism that you can fix it fast, though, so during an incident you need a time limit and a rule that converts to rollback when it expires.

7-3. Secondary failures caused by the rollback

The act of undoing can itself cause an outage. The Google SRE Book defines a cascading failure as a failure that grows over time as a result of positive feedback. Immediately after a rollback, caches are empty, connections are re-established, and the requests that failed retry all at once — which is exactly that feedback loop.

• Always schedule retries with randomised exponential backoff. That is the SRE Book's explicit recommendation.
• Set a retry budget. The book's example is 60 retries per minute in a process.
• Retries multiply across layers. Three layers retrying four times each turn one user action into 64 attempts.
• Distinguish retriable from permanent errors in code, and never retry a permanent error.
• Rejecting early beats queueing indefinitely. Load shedding and graceful degradation are the mechanisms.

Making retries safe requires idempotency. RFC 9110 defines idempotent as meaning that the intended effect on the server of multiple identical requests is the same as for a single request, and classifies GET, HEAD, PUT, DELETE, OPTIONS and TRACE as idempotent. POST is not. For the design details see Idempotency and Retries: APIs You Can Trust and the Retry & Cumulative Probability Calculator.

7-4. After you cross an irreversible boundary

Once you have passed the point of no return, only compensation remains: a correction notice for wrongly sent alerts, a repair batch for miscomputed values, a change advisory to external consumers. What matters is not inventing this procedure after the incident. Anything classified as irreversible in section 2 should carry its compensation procedure in the deployment plan.

8. What to watch and when to stop — deployment gates and SLOs

8-1. Write the abort criteria as numbers before deploying

Watching a dashboard mid-deployment and deciding "this looks fine" is easily post-hoc rationalisation. Writing the numbers and the observation window down in advance turns judgement into verification.

# Example — deployment gates written down declaratively
canary:
  steps: [1, 5, 25, 50, 100] # traffic percentages
  interval: 15m # observation window per step, longer than metric lag
  analysis:
    - metric: request_error_rate # derived from an SLI
      compare_to: baseline # concurrent control, not a past point in time
      fail_if: canary > baseline * 1.2
    - metric: latency_p99
      fail_if: canary > baseline * 1.3
  on_failure: rollback # abort automatically, then notify a human

8-2. Metric lag is the lower bound on step length

If the metrics pipeline lags by five minutes, a three-minute canary step passes without seeing anything. Each step's observation window must exceed the metric lag plus at least one aggregation interval. An auto-promotion pipeline that skips this calculation races to 100% and only then starts alerting.

8-3. What to gate on, and what not to

• Gate on: user-facing SLIs. Request error rate, upper latency quantiles, success rate of the core conversion flow.
• Reference only: resource metrics like CPU and memory. The causal link to the change is weak and the noise is high.
• Never gate on: metrics with no attribution to the change. This is exactly the Workbook's attribution requirement quoted above.
• Error budget burn rate: for abort decisions, the rate of burn is more useful than the absolute value.

8-4. When the gate is too sensitive

Frequent false positives teach people to ignore or bypass the gate. After introducing gates, record what fraction of aborts turned out to be real defects. When that fraction is low, the problem is usually metric selection rather than the threshold.

9. Measuring deployment

DORA defines its four key metrics as follows.

• Deployment frequency: the number of deployments over a given period, or the time between deployments.
• Change lead time: the amount of time it takes for a change to go from committed to version control to deployed in production.
• Failed deployment recovery time: the time it takes to recover from a deployment that fails and requires immediate intervention.
• Change fail rate: the ratio of deployments that require immediate intervention following a deployment.
• Rework rate: the ratio of unplanned deployments caused by a production incident.

9-1. Speed and stability are not a trade-off

DORA states this plainly: its research has repeatedly demonstrated that speed and stability are not trade-offs, and that for most teams the metrics are in fact correlated. In DORA's own words, the real trade-off over long periods of time is between better software faster and worse software slower.

Where that result meets this article's subject is clear enough. Teams that deploy often are stable because their deployment units are small, and a small unit is by definition an easily reversible one. Reversibility is not the price of speed; it is its precondition.

9-2. The distortion when metrics become KPIs

Make deployment frequency a target and you can manufacture the number by splitting meaningless deployments. Make change fail rate a target and you create an incentive not to record failures as failures. The four metrics only mean anything together, and any one of them distorts the moment it becomes an individual performance goal. That warning is not a sentence from the source above; it is a practical caution I am adding.

Quiz: Check your understanding

Wrapping up

Choosing a deployment strategy is not choosing a tool. It is deciding whether this change is still reversible, deferring the irreversible parts as far as possible, and writing down the compensation procedure for whatever irreversible portion remains.

Compressed into one sentence: reversibility is determined by your data model, not your deployment pipeline. A team that honours expand-migrate-contract is safe with whatever deployment tool it uses, and a team that skips it trips over the same spot no matter what it uses.

References

• Canarying Releases — Google SRE Workbook — cited for the definition of canarying, the concurrent-control requirement, metric selection and the limit on their number, representativeness of size and duration, the calculation that a 5% canary with a 20% error rate yields 1% overall, and the limits of what a canary catches. Checked 2026-08-15.
• DORA metrics: the four keys — DORA — cited for the definitions of deployment frequency, change lead time, failed deployment recovery time, change fail rate and rework rate, and for the conclusion that speed and stability are not trade-offs. Checked 2026-08-15.
• Parallel Change — Martin Fowler — cited for the definitions of the expand, migrate and contract phases and the attribution to Joshua Kerievsky. Checked 2026-08-15.
• Addressing Cascading Failures — Google SRE Book — cited for the definition of cascading failure, randomised exponential backoff, the 60-retries-per-minute budget example, the multiplication of retries across layers, and load shedding. Checked 2026-08-15.
• RFC 9110 — HTTP Semantics — cited for the definition of idempotence in section 9.2.2, the list of idempotent methods, and the fact that POST is not among them. Checked 2026-08-15.
• The three-tier reversibility classification in section 2, the sample-size arithmetic in 4-3, the rollback decision checklist in 7-1 and the KPI distortion warning in 9-2 are not taken verbatim from the sources above; they are procedures and interpretations assembled in this article.

Further reading

• Related post on this blog: Feature Flags and Progressive Delivery, Fully Dissected
• Related post on this blog: SLI/SLO/Error Budget-based Reliability Engineering
• Related post on this blog: The Complete Guide to the Strangler Fig Pattern
• Related post on this blog: Idempotency and Retries: APIs You Can Trust
• Related tool: SLO & Error Budget Calculator
• Related tool: Retry & Cumulative Probability Calculator

The Complete Guide series

• Previous: The Complete Guide to Code Review: Designing Review as a Process
• Next: The Complete Guide to Refactoring: Changing Structure While Preserving Behaviour