zopdev writing tagged kubernetes. Engineering and FinOps notes, post-mortems, and benchmarks.
A deployed service's web address with no password on it is one click away from being locked down. ZopDay puts that click right next to the address itself: an admin clicks a lock icon, and the service…
A card that only tells you a service is running still leaves the actual question unanswered: running well, or running on borrowed time. ZopDay's project canvas used to show a single status per…
A node sitting at 13% processor utilization is common on a cluster that's never repacked its own workloads, and it still costs exactly as much as a node running at capacity. ZopDay's new Pod…
ROSA is invisible to the AWS APIs that discover EKS, so it drops out of inventory and every policy inventory feeds. One cluster list fixes that.
The HStack left-center-right alignment problem is not a niche edge case. It is a layout trap that 68,526 developers walked into and had to search their way out of (Stack Overflow, question 70776006).…
A service deployed on a VM used to run continuously, holding its full memory allocation whether one visitor showed up that hour or none did. That's the default assumption most deploy paths make: a…
The autoscaler decision you made at 50 nodes becomes a structural liability at 10,000. By the time the cluster grows to enterprise scale, the choice is load-bearing infrastructure. Replacing it…
Policy as Code works cleanly until it meets ten teams, and then it breaks in ways the pilot never predicted. The first three months feel like a governance win. Policies deploy, violations get caught,…
Point-in-time benchmarks produce misleading Kubernetes cost comparisons because cluster behavior, workload patterns, and pricing all shift across a 12-month horizon in ways a single snapshot cannot…
Off-hours incidents expose a structural flaw in how engineering teams are staffed: human cognition degrades sharply after midnight, and every minute of that degradation has a direct dollar cost…
The chart version a service is running is usually a guess. A team imports a Helm release or installs something from a catalog, and the chart version stays whatever it was on day one until someone…
Cloud pricing pages are built to sell compute and storage. Every other charge is buried.
Static runbooks fail at 3 AM not because engineers write them poorly, but because incidents refuse to follow the sequences those runbooks assume.
On-call engineers are routinely woken at 3am to execute the same five-step runbook they ran the night before, and the tooling to stop this pattern has existed in production environments for years.…
ZopNight's newest release notes open with a new feature: you can now migrate a database from one GCP Cloud SQL instance to another without leaving ZopDay, the Kubernetes platform ZopNight ships…
Alert-only incident response transfers the cost of every failure from the system to the engineer, and that transfer compounds at scale.
HashiCorp's August 2023 switch from the Mozilla Public License to the Business Source License forced every infrastructure team running Terraform to make a governance decision they had not budgeted…
At 500 managed resources, infrastructure drift stops being a maintenance nuisance and becomes a misdiagnosis engine that corrupts incident response at the root.
Most AIOps implementations treat the "Act" phase as the finish line, and that architectural choice turns automated remediation into a liability rather than a guarantee.
Autoscaling latency is not a performance problem. It is a billing problem. Every second a Kubernetes cluster waits to provision a node, existing nodes carry idle capacity that the cloud provider…
Picking the wrong Kubernetes autoscaling tool for a given workload type does not just leave performance on the table. It actively generates waste you pay for every billing cycle.
A search box that filters only the page you are on is not search, it is a per-page filter wearing search's clothes.
A green dashboard is not proof of a healthy system. It is proof that your automation closed a ticket. Those two outcomes are not the same thing, and conflating them is how engineering teams…
Visibility without workflow integration is a cost center, not a cost cure. Most engineering organizations have invested in dashboards, tagging policies, and cost explorer tools. The spend keeps…
HashiCorp's August 2023 relicensing of Terraform from MPL-2.0 to the Business Source License 1.1 was not a routine legal update. It was a governance event that forced every infrastructure team to…
Manual policy enforcement breaks at infrastructure scale because the number of enforcement decisions grows faster than the number of resources. A team managing 50 resources reviews policy by…
Alert routing is a notification system, not a resolution system, and treating it as the end state of incident response is why on-call engineers burn out.
Ad-hoc policy management breaks down precisely at the point where account count and resource sprawl outpace human review cycles. Below 50 resources across two or three accounts, a shared spreadsheet…
In Kubernetes, the namespace is where multi-tenancy actually lives. Resource quotas stop there. Network policy stops there. RBAC stops there. Then someone opens a live cluster view and sees every…
When you provision a service through ZopDay, you get its whole story: configuration, live status, and how to reach it. When you
AI Ops agents create a dangerous illusion: they close tickets fast, but they routinely fix the wrong thing first (ZopDev, "Why Your AI Ops Agent Fixes the Wrong Thing First").
The free-credit expiration is not a billing event. It is a systems failure that was designed into the architecture from day one.
A single bad deployment cost $180,000 not because the deployment was uniquely catastrophic, but because nothing in the system was configured to stop it from spreading (ZopDev, "Blast Radius by…
Static runbooks fail modern infrastructure because the gap between when a failure begins and when a human reads an alert is measured in minutes your SLO does not have.
At 200 resources, the architectural assumptions baked into every IaC tool become load-bearing walls, and some of those walls crack.
Alerting is not remediation. That gap between "alert fired" and "system restored" is where reliability erodes and costs accumulate.
GUI tools for developer workflows carry a hidden tax: every click, every modal, every context switch compounds into lost focus that terminal-native engineers refuse to pay.
The pipeline itself is not the risk. The risk is the gap between what the pipeline assumes is true and what is actually true in the environment it deploys into.
Cloud bills grow faster than the teams responsible for paying them, and the gap between what you spend and what you understand is where waste compounds silently.
Kubernetes cost overruns compound in silence because the billing signal arrives weeks after the spending decision. A developer sets a memory request too high on a Tuesday. The scheduler honors that…
Every cloud-native team building observability at scale hits the same three-way constraint: you cannot simultaneously maximize platform capability, minimize cost, and keep operational complexity low.…
HashiCorp's August 2023 relicense of Terraform from MPL-2.0 to the Business Source License forced every infrastructure team to make a governance decision, not a technical one.
Manual incident response at 2 AM is an organizational failure mode, not a staffing problem. When a bad deployment reaches production and an engineer's phone wakes them, the damage clock started…
Every internal developer platform carries a hidden 4-week tax: engineer time spent on platform setup before a single service ships (ZopDev, "The IDP Tax"). That number is not a rounding error. It is…
Most FinOps teams track one number when they need two, and that single blind spot turns cost governance into a cleanup exercise instead of a control system.
Most IDPs ship as friction-reducers and land as a new category of sprint tax. The promise is a self-service portal that abstracts infrastructure complexity. The reality, in production, is a platform…
HashiCorp's August 2023 license change from MPL-2.0 to the Business Source License forced every team running Terraform in production to make a governance decision they had not budgeted for. The BSL…
Most engineering organizations budget precisely for building an Internal Developer Platform and budget nothing for operating one. The build cost is visible: headcount, tooling licenses, sprint…
Every FinOps initiative follows the same arc: a burst of recoverable savings in the first weeks, then a structural decay that accelerates past month 3 (ZopDev, "Why FinOps Savings Decay Faster After…
Every runbook your team executes manually is an open automation ticket that nobody filed. That is the central problem. The runbook library is not documentation. It is a backlog in disguise, and most…
OOMKill is a reporting artifact, not a root cause. By the time the kernel logs the kill event and your alerting pipeline fires, the service already degraded for every user who hit it in the preceding…
P95 CPU became the default right-sizing signal because it reduces a complex system to a single number that executives can approve in a slide deck. We measured this pattern across 40 production…
VM services get debugged over SSH while pods get a live observability tab. A control-plane exec channel closes the gap across AWS, Azure, and GCP.
Most tools that promise to manage your existing workloads ask for one thing first: redeploy everything through us. That is the wrong price. A re-rollout against production carries a real downtime…
Your cloud bill shows a Kubernetes cluster as a single number. EKS, GKE, and AKS all roll node compute into one rolled-up figure. Finance sees the total. Nobody sees which namespace or deployment…
Most teams pick a VM because the runtime is simple. Then they spend a week paying for that choice in firewall rules, SSH key rotation, a certbot cron job, and a DNS ticket. The VM was never the…
AI assistants now connect to cloud cost infrastructure in 5 minutes, transforming how engineering teams query spending data. Claude Fable wires to cloud cost data via ZopNight in 5 minutes (ZopDev…
Cloud infrastructure pricing forces a binary choice: lock in capacity at a fixed discount or chase maximum savings with no guarantees. Commitment discounts deliver predictable costs by reserving…
Ticket-based infrastructure workflows inject a minimum three-day delay into every deployment cycle because each request moves through a queue where a centralized team must interpret, validate, and…
Kubernetes restarts failed pods faster than most alerting systems can fire, creating a class of incidents that resolve themselves before operations teams know they happened. This self-healing…
Most IDPs fail because they solve the wrong problem: they build self-service portals instead of standardizing the work developers already do. We measured this in production. Teams spend six months…
Building an Internal Developer Platform for 12 teams costs $400,000 (Platform Engineering for 12 Teams: The $400k IDP Bill), and understanding where that money goes determines whether you build or…
The Fargate Tax: Why Serverless Kubernetes Costs 38% More Past 200 vCPU-Hours Fargate is appealing because the pitch is clean: no AMI patching, no node group sizing, no cluster autoscaler tuning. You…
Kubernetes MTTR: From 43 Minutes to 9 With Structured Runbooks The median Kubernetes incident takes 43 minutes to resolve. Eight minutes of that is the actual fix. The other 35 minutes is engineers…
A product team announces a new feature on Tuesday at 10:00 AM Pacific. The marketing email goes out at 09:55. By 10:01 the load balancer is seeing 8 times its baseline request rate. The autoscaler is…
A new engineer joins on Monday. By Friday they need their first production-grade EKS cluster running so they can deploy the service they were hired to build. They open the company's Terraform module.…
A 500-pod cluster has one pod that restarted three times in the last 10 minutes. The operator on call does not know which pod. returns 500 lines of and a handful of interleaved through them. Finding…
A team writes the cron job that shuts non-prod down at 8 PM. The cron runs three commands in parallel: scale the EKS Deployments to zero, pause the Aurora cluster, stop the ElastiCache Redis nodes.…
A right-sized EKS cluster should not run at 40 percent node utilization. The pods declare requests that sum to 78 percent of node capacity. The cluster autoscaler provisions nodes to fit those…
The 3am page is rarely about something that needs a human. The on-call gets paged at 03:14 because a pod has crashlooped four times in five minutes. They open Slack, look at the logs, see "OOMKilled"…
The dashboard says CPU throttling is at 0.5%. The p99 latency on that container says 30% of requests just lost 80 milliseconds to scheduling delay. Both numbers are correct. They are measuring…
Istio sidecars cost 0.5 vCPU per pod at idle. At 100 pods, you're paying for 50 idle vCPUs. eBPF moves observability into the kernel — one hook point per node, not per pod. Here's the architecture, the tools, and when you still need Envoy.
Every service provisioned from a Backstage template starts with zero budget alerts, zero mandatory tags, and a dev environment that runs 24/7. The platform team didn't choose this — they just never added cost defaults to the template. Here's how to fix that.
OPA Gatekeeper rejects a pod before it ever runs. Here is how to write admission policies that block oversized resource requests, missing cost labels, and non-prod images at deploy time, not billing time.
Unrestricted pod egress runs every outbound call through NAT Gateway at $0.045 per GB. NetworkPolicy is both a security control and a cost control. Here is how to use it as both.
Cluster Autoscaler works on day one. Six months in, you have 12 node groups, 30% idle capacity, and scaling incidents during traffic ramps. Here is what changes when you switch to Karpenter.
Shared clusters without hard quotas become tragedy-of-the-commons cost problems. One team's memory leak becomes everyone's OOM. Here's how LimitRanges, ResourceQuotas, and namespace cost attribution fix that.
Istio adds 50-100m CPU and 50-100Mi memory per pod at idle. At 100 pods that's 10 extra CPU cores. Here's the overhead math at scale, what you actually get for it, and when lighter alternatives make more sense.
The average Kubernetes cluster runs at 13% CPU utilization. VPA, HPA, and KEDA each attack the 87% idle gap differently — here's which one cuts your bill and which one creates production incidents.
SCPs block cloud-level overprovisioning but can't see inside a Kubernetes cluster. OPA Gatekeeper fills the admission control gap — blocking wasteful pod specs before they ever schedule.
Most teams running shared Kubernetes clusters believe they have isolation. They have namespaces. It feels like separation. It is not. Here's how to configure actual multi-tenancy.
Running Apache Cassandra on Kubernetes is an architectural commitment. Explore token rings, stateful identity, and operational risks in this technical guide.
Explore the mechanics of running PostgreSQL on Kubernetes. Learn about WAL, storage, and replication to manage operational risks and ensure database durability.
DevOps is evolving fast. Discover the top 12 trends shaping DevOps in 2025—from SRE and automation to AIOps and culture—and how your team can stay ahead.
Ready to deploy Kubernetes in production? This comprehensive Kubernetes production checklist by Zopdev covers essential best practices for stability, observability, and cost efficiency.
Explore advanced strategies for implementing CI/CD pipelines. Dive into pipeline architectures, branching models, automated testing, and deployment techniques for modern software delivery.
Kubernetes is powerful—but let’s face it, it often feels like a black box wrapped in YAML. This blog breaks down why Kubernetes feels so overwhelming and shows you how to simplify it using real-world tools like Terraform, GitOps, and automation platforms like Zopdev.
One post a week. Sundays. No "10 ways to think about cloud" listicles, just the engineering and FinOps notes we'd want to read.
See. Find. Fix. Automatic.
Connect your first cloud account in under 5 minutes. See your first remediation in under 7. No credit card required.