MSK Kafka
What does zop.dev create for MSK Kafka?
zop.dev provisions an Amazon MSK cluster with its broker nodes, security group and storage, wired into the VPC private tier. Broker count and instance type are fixed at provisioning time, and MSK bills per broker-hour plus provisioned storage whether or not topics carry traffic.
| Field | Value |
|---|---|
| Cloud | aws |
Amazon MSK is managed Apache Kafka for event streaming between services. zop.dev provisions the cluster with SCRAM authentication, encryption, and logging wired in, and stores the generated credentials where applications can consume them.
An MSK cluster with SCRAM credentials and KMS
MSK cluster, SCRAM secret association, KMS key, CloudWatch log group, security group, Secrets Manager secret, and a randomly generated SCRAM password.
Broker count, instance type, and per-broker storage
kafka_broker_nodes (must be a multiple of client subnets); kafka_broker_instance default kafka.t3.small; kafka_size default 10 GiB EBS per broker; kafka_subnets[]; kafka_admin_user; kafka_topics[].Kafka is implemented on AWS only
Kafka is available on AWS only. The kafka/azure module files are empty (not implemented).
MSK bills for capacity, not throughput
Every broker bills per hour from the moment the cluster is created, plus the storage provisioned per broker. A cluster with no producers costs the same as one at full throughput.
That makes MSK one of the least forgiving resources to over-provision. A three-broker cluster on a large instance type running a development workload is a fixed monthly cost with no natural signal that it is oversized. Kafka does not report “idle” the way a database reports zero connections.
Broker count is effectively permanent
You can add brokers to a running cluster. You cannot remove them. Sizing down means creating a new cluster and migrating topics, which for anything with retained data is a project.
Start smaller than feels comfortable. Adding a broker later is an operation; removing one is a migration.
Storage grows and does not shrink
Per-broker storage can be expanded and never reduced. With retention configured generously (the default instinct), storage climbs to the retention ceiling and stays there as a permanent floor on the bill.
Set retention deliberately per topic rather than accepting a cluster-wide default. A topic that needs seven days of retention and a topic that needs one hour should not share a setting.
Serverless is the alternative worth knowing
MSK Serverless bills per GB in and out rather than per broker-hour. For intermittent or spiky workloads that is dramatically cheaper than provisioned brokers sitting idle between bursts.
It is the right default for development and for anything whose throughput is unpredictable. The provisioned cluster is right when throughput is steady and high enough that per-GB pricing overtakes per-broker.