Shipping without breaking the design

Three deployment shapes

Canary is the default for user-facing services because it limits exposure to a bad build. Rolling is fine for stateless workers. Blue-green is expensive but useful when you need an instant switch — for example a release with a risky dependency upgrade.

The migration problem

During a rolling deploy, old and new code both run. If the new code expects a column the old code does not write, the old instances produce rows the new code cannot read. This is the failure mode that turns a routine release into a data incident.

day 1  ALTER TABLE users ADD COLUMN email_norm text;   -- nullable, no default rewrite
       app writes both columns, reads email
day 2  backfill 10k rows per minute
day 3  app reads email_norm, writes both
day 5  app writes email_norm only
day 9  ALTER TABLE users DROP COLUMN email;

Flags and rollback

• Deploy dark, enable later. Ship the code path disabled, turn it on for internal users, then 1%, then everyone.
• Name flags with an owner and an expiry. A flag with no removal date is a permanent branch in production.
• Roll back on data, not vibes. Error rate above baseline, p99 above the SLO, or a business metric dropping for ten minutes triggers automatic rollback.
• Keep the previous artefact warm. Rollback that requires rebuilding the last image takes longer than the incident allows.

One more principle worth stating in an interview: deploys should be boring. Small, frequent releases with automatic rollback make failures small; large monthly releases make every failure a project. The design consequence is that the pipeline is part of the architecture, not an afterthought.