# MSK Kafka

> zop.dev provisions an Amazon MSK cluster with its broker nodes, security group and storage, wired into the VPC private tier. Broker count and instance type are fixed at provisioning time, and MSK bills per broker-hour plus provisioned storage whether or not topics carry traffic.

Source: https://zop.dev/integrations/provision/aws/msk-kafka
Updated: 2026-08-19

---

Amazon MSK is managed Apache Kafka for event streaming between services. zop.dev provisions the cluster with SCRAM authentication, encryption, and logging wired in, and stores the generated credentials where applications can consume them.

## An MSK cluster with SCRAM credentials and KMS

MSK cluster, SCRAM secret association, KMS key, CloudWatch log group, security group, Secrets Manager secret, and a randomly generated SCRAM password.

## Broker count, instance type, and per-broker storage

```text
kafka_broker_nodes (must be a multiple of client subnets); kafka_broker_instance default kafka.t3.small; kafka_size default 10 GiB EBS per broker; kafka_subnets[]; kafka_admin_user; kafka_topics[].
```

## Kafka is implemented on AWS only

Kafka is available on AWS only. The kafka/azure module files are empty (not implemented).

## MSK bills for capacity, not throughput

Every broker bills per hour from the moment the cluster is created, plus the storage provisioned
per broker. A cluster with no producers costs the same as one at full throughput.

That makes MSK one of the least forgiving resources to over-provision. A three-broker cluster on
a large instance type running a development workload is a fixed monthly cost with no natural
signal that it is oversized. Kafka does not report "idle" the way a database reports zero
connections.

## Broker count is effectively permanent

You can add brokers to a running cluster. You cannot remove them. Sizing down means creating a
new cluster and migrating topics, which for anything with retained data is a project.

Start smaller than feels comfortable. Adding a broker later is an operation; removing one is a
migration.

## Storage grows and does not shrink

Per-broker storage can be expanded and never reduced. With retention configured generously
(the default instinct), storage climbs to the retention ceiling and stays there as a
permanent floor on the bill.

Set retention deliberately per topic rather than accepting a cluster-wide default. A topic that
needs seven days of retention and a topic that needs one hour should not share a setting.

## Serverless is the alternative worth knowing

MSK Serverless bills per GB in and out rather than per broker-hour. For intermittent or spiky
workloads that is dramatically cheaper than provisioned brokers sitting idle between bursts.

It is the right default for development and for anything whose throughput is unpredictable. The
provisioned cluster is right when throughput is steady and high enough that per-GB pricing
overtakes per-broker.