Rollbacks

When to roll back

Roll back when a deploy made it past the health gates but is now misbehaving in production; elevated error rate, latency regression, a missed dependency. ZopDay’s rollback is build-skipped, secrets-preserved, audit-trailed.

For deploys that fail their own health gates, ZopDay rolls back automatically: no human action required. Manual rollback covers the case where the regression appears after the deploy is marked live.

How rollback works

Every successful release is kept on the cluster for the configured grace window. Rolling back is a deterministic re-point:

1. Select the target release. The pipeline view shows the last N successful releases with their commit SHA, deployer, and timestamp. Pick the one you want.
2. Pre-flight check. ZopDay verifies the target release still exists on the cluster and that its supporting resources (ConfigMaps, Secrets references) are intact.
3. Apply. The selector flips back to the target release. For rolling deploys, replicas swap progressively. For blue/green, the swap is atomic.
4. Gate. Health gates run against the rolled-back version; readiness, probes, smoke tests.
5. Audit. A pipeline.rollback event captures actor, source release, target release, timestamp, and reason.

No rebuild happens. The image hash is the one that was already on the cluster; you are reverting, not redeploying.

Strategy-specific behaviour

Rolling

Rollback replays the rolling-update logic in reverse; the previous ReplicaSet is scaled up while the current one is scaled down, batched by the same maxSurge / maxUnavailable values. Typical completion time matches the original rolling deploy.

Canary

If rollback happens before the canary reaches 100%, it’s an immediate stop; the canary ReplicaSet is scaled to zero and traffic shifts fully to the stable version. Sub-second flip.

If rollback happens after the canary reached 100% (i.e. it became the new stable), the previous stable release is restored as a fresh rolling update.

Blue / Green

The selector flips back to the previous (blue) release, the last known-good version. This is atomic: every in-flight request after the flip lands on the rolled-back version. Old replicas are kept warm until the next deploy or the configured grace window elapses, whichever comes first.

What survives a rollback

• Secrets in your cloud vault are untouched. The rolled-back version reads the same vault path as before.
• ConfigMaps managed by the previous Helm release are re-applied.
• Persistent volumes are not touched. Database migrations are not reversed: you need to run a forward-compatible migration strategy if you want rollback to be fully safe (additive columns, expand-then-contract).
• In-flight HTTP requests complete on their original version (grace shutdown).

Safety gates

Before any rollback applies, the following gates run:

1. Authorization: the actor must have Editor or higher on the target environment (rollback is a write action).
2. Quiet window: if a release window has been declared as quiet (release freeze), rollback requires an explicit --override flag.
3. Target still exists: the rolled-back version’s image must still be pullable. Expired images fail the gate.
4. Dependency check: if the rolled-back version depends on a config / secret / CRD that has since changed, the gate surfaces the divergence and asks for confirmation.

If a gate fails, the rollback is blocked, not retried: a human decision is required.

Audit

Every rollback ships a structured audit entry:

{
  "event": "pipeline.rollback",
  "actor": { "type": "user", "email": "ops@example.com" },
  "service": "payments-api",
  "environment": "prod",
  "sourceRelease": "v2.41.0",
  "targetRelease": "v2.40.3",
  "reason": "elevated 5xx in monitor after v2.41",
  "appliedAt": "2026-06-01T05:01:23Z",
  "gates": {
    "authorization": "pass",
    "quietWindow": "pass",
    "targetExists": "pass",
    "dependencyCheck": "pass"
  }
}

The same record is forwardable to your SIEM on the Enterprise tier.

API and CLI

# CLI
zopday rollback payments-api --to v2.40.3 --reason "elevated 5xx in monitor"

# During a declared quiet window (release freeze), add --override to proceed
zopday rollback payments-api --to v2.40.3 --reason "elevated 5xx in monitor" --override

# API
curl -X POST https://api.zop.dev/v1/pipelines/payments-api/rollback \
  -H "Authorization: Bearer $ZOPDAY_PAT" \
  -d '{"target":"v2.40.3","reason":"elevated 5xx in monitor"}'

Generate $ZOPDAY_PAT as a Personal Access Token from your profile menu; see Authentication.

Both surface the same gates and emit the same audit record.