Workload Reliability: Checks, Retries and Schedule Health

When a scheduled action fails

If a start or stop fails, ZopNight retries it automatically, with a maximum of three retries, then moves it to a dead-letter queue instead of dropping it. The failure shows in the action status, so the UI can say what went wrong, and a notification goes to the channel you chose: Slack, Google Chat, Microsoft Teams or a webhook.

You can subscribe to scheduling events separately by trigger: scheduled starts and stops, manual ones, failures, retries exhausted, and override set, cleared or expired. A channel can be set per resource or per group, with a default for everything else.

Knowing whether schedules are working

• Schedule success rate. A dashboard widget reports successful, failed and missed firings. A firing on a resource that was already in the target state is recorded as skipped and left out of the rate, so a no-op does not count as a miss.
• State history. Every on-to-off and off-to-on transition is stored with its time and its cause: which schedule, which user, or which override. The timeline makes an unexpected stop easy to spot.
• Permission verdicts. After each discovery, ZopNight records whether it can discover, recommend and schedule each resource type, so “we can see it but cannot stop it” is stated up front. A permission that becomes newly denied triggers an alert.
• Billing and provider alerts. Degraded billing sync and provider-side incidents have their own notifications, so a gap in cost data is not mistaken for a drop in spend.

Workload reliability findings

The reliability category of recommendations looks at the workloads themselves, mostly on Kubernetes:

• Deployments or StatefulSets not fully ready, DaemonSets not fully scheduled
• missing liveness or readiness probes, missing CPU and memory requests or limits
• single-replica Deployments and HPAs pinned at their maximum
• PodDisruptionBudgets that block every voluntary disruption
• nodes under memory, disk or PID pressure, or not ready

For RDS, RDS Instance Not Using Multi-AZ sits next to RDS Multi-AZ on Non-Production Instance: the first adds resilience where it matters, the second removes cost where it does not.

Safe by construction

The same concern shows in how fixes run: one-shot stops on compute are advisory, destructive steps wait for an admin, and impact analysis shows connected resources first.

See reliability recommendations, Kubernetes cost management and audit trails.