# MSK Kafka

> zop.dev provisions an Amazon MSK cluster with its broker nodes, security group and storage, wired into the VPC private tier. Broker count and instance type are fixed at provisioning time, and MSK bills per broker-hour plus provisioned storage whether or not topics carry traffic.

Source: https://zop.dev/integrations/provision/aws/msk-kafka
Updated: 2026-08-19

---

Amazon MSK is managed Apache Kafka for event streaming between services. zop.dev provisions the cluster with SCRAM authentication, encryption, and logging wired in, and stores the generated credentials where applications can consume them.

## An MSK cluster with SCRAM credentials and KMS

MSK cluster, SCRAM secret association, KMS key, CloudWatch log group, security group, Secrets Manager secret, and a randomly generated SCRAM password.

## Broker count, instance type, and per-broker storage

```text
kafka_broker_nodes (must be a multiple of client subnets); kafka_broker_instance default kafka.t3.small; kafka_size default 10 GiB EBS per broker; kafka_subnets[]; kafka_admin_user; kafka_topics[].
```

## Kafka is implemented on AWS only

Kafka is available on AWS only. Azure and GCP environments cannot provision it yet.

## MSK bills for capacity, not throughput

Every broker bills per hour from the moment the cluster is created, plus the storage provisioned
per broker. A cluster with no producers costs the same as one at full throughput.

That makes MSK one of the least forgiving resources to over-provision. A three-broker cluster on
a large instance type running a development workload is a fixed monthly cost with no natural
signal that it is oversized. Kafka does not report "idle" the way a database reports zero
connections.

## Removing brokers takes planning

You can add brokers to a running cluster, and since 2024 you can also remove them from clusters on
M5 or M7g brokers, but only after moving every partition off the brokers being removed, and only
in multiples of the cluster's Availability Zone count. The module's default kafka.t3.small brokers
do not support removal at all.

Start smaller than feels comfortable. Adding a broker later is a single operation; removing one
means reassigning partitions first.

## Storage grows and does not shrink

Per-broker storage can be expanded and never reduced. With retention configured generously
(the default instinct), storage climbs to the retention ceiling and stays there as a
permanent floor on the bill.

Set retention deliberately per topic rather than accepting a cluster-wide default. A topic that
needs seven days of retention and a topic that needs one hour should not share a setting.

## Serverless is the alternative worth knowing

MSK Serverless bills per GB in and out rather than per broker-hour. For intermittent or spiky
workloads that is dramatically cheaper than provisioned brokers sitting idle between bursts.

It is the right default for development and for anything whose throughput is unpredictable. The
provisioned cluster is right when throughput is steady and high enough that per-GB pricing
overtakes per-broker.
