MSK Kafka
What does zop.dev create for MSK Kafka?
zop.dev provisions an Amazon MSK cluster with its broker nodes, security group and storage, wired into the VPC private tier. Broker count and instance type are fixed at provisioning time, and MSK bills per broker-hour plus provisioned storage whether or not topics carry traffic.
| Field | Value |
|---|---|
| Cloud | aws |
Amazon MSK is managed Apache Kafka for event streaming between services. zop.dev provisions the cluster with SCRAM authentication, encryption, and logging wired in, and stores the generated credentials where applications can consume them.
An MSK cluster with SCRAM credentials and KMS
MSK cluster, SCRAM secret association, KMS key, CloudWatch log group, security group, Secrets Manager secret, and a randomly generated SCRAM password.
Broker count, instance type, and per-broker storage
kafka_broker_nodes (must be a multiple of client subnets); kafka_broker_instance default kafka.t3.small; kafka_size default 10 GiB EBS per broker; kafka_subnets[]; kafka_admin_user; kafka_topics[].Kafka is implemented on AWS only
Kafka is available on AWS only. Azure and GCP environments cannot provision it yet.
MSK bills for capacity, not throughput
Every broker bills per hour from the moment the cluster is created, plus the storage provisioned per broker. A cluster with no producers costs the same as one at full throughput.
That makes MSK one of the least forgiving resources to over-provision. A three-broker cluster on a large instance type running a development workload is a fixed monthly cost with no natural signal that it is oversized. Kafka does not report “idle” the way a database reports zero connections.
Removing brokers takes planning
You can add brokers to a running cluster, and since 2024 you can also remove them from clusters on M5 or M7g brokers, but only after moving every partition off the brokers being removed, and only in multiples of the cluster’s Availability Zone count. The module’s default kafka.t3.small brokers do not support removal at all.
Start smaller than feels comfortable. Adding a broker later is a single operation; removing one means reassigning partitions first.
Storage grows and does not shrink
Per-broker storage can be expanded and never reduced. With retention configured generously (the default instinct), storage climbs to the retention ceiling and stays there as a permanent floor on the bill.
Set retention deliberately per topic rather than accepting a cluster-wide default. A topic that needs seven days of retention and a topic that needs one hour should not share a setting.
Serverless is the alternative worth knowing
MSK Serverless bills per GB in and out rather than per broker-hour. For intermittent or spiky workloads that is dramatically cheaper than provisioned brokers sitting idle between bursts.
It is the right default for development and for anything whose throughput is unpredictable. The provisioned cluster is right when throughput is steady and high enough that per-GB pricing overtakes per-broker.