Right Sizing Your Apache Kafka Clusters on Amazon MSK

The Core Tradeoffs

When sizing a Kafka cluster on Amazon MSK, the key dimensions to reason about are network throughput, storage throughput, the number of brokers, the number of consumer groups, and the replication factor. Each of these interacts with the others in ways that make the decision space complex.

Network throughput determines how much data your brokers can ingest and serve at any given moment. Storage throughput governs how quickly data can be written to and read from disk, which matters especially for workloads with high retention requirements or bursty producers. The number of brokers sets your parallelism ceiling and affects how partitions are distributed. Consumer groups multiply the read traffic because each group independently reads the full topic. And the replication factor directly multiplies your write amplification since every message must be stored on multiple brokers for durability.

A cluster that looks right on paper for ingestion alone can fall apart once you account for three consumer groups and a replication factor of three, because the actual network and storage demand is far higher than the raw producer throughput suggests.

MSK Express Brokers

Amazon MSK Express brokers offer a fully managed storage layer that removes much of the operational complexity around disk provisioning and scaling. With Express brokers you get predictable throughput characteristics without needing to manage EBS volumes, monitor disk utilization, or manually rebalance partitions when storage fills up. This makes them a strong fit for teams that want to focus on their application logic rather than infrastructure tuning.

Express brokers simplify the sizing decision because the storage dimension is abstracted away. You primarily need to reason about network throughput and the number of brokers relative to your producer and consumer traffic.

When to Choose Standard Brokers

Standard brokers remain the right choice when you need fine grained control over storage configurations, want to use tiered storage for cost effective long term retention, or need to set replication quotas to manage cross broker traffic. If your workload benefits from decoupling hot and cold data across storage tiers, or if you need to enforce strict throughput limits between brokers, Standard brokers give you the knobs that Express intentionally hides.

Sizing Considerations

Start by estimating your peak producer throughput in megabytes per second. Multiply that by your replication factor to get the total write load across the cluster. Then account for consumer traffic by multiplying the producer throughput by the number of independent consumer groups. The sum of write and read traffic gives you the minimum aggregate network capacity your brokers must support.

From there, divide by the per broker network throughput limit to determine the minimum number of brokers. Add headroom for partition rebalancing, rolling upgrades, and traffic spikes. A common recommendation is to target no more than 60 to 70 percent utilization on any single broker under normal conditions.

For storage throughput, consider your message size, batch configuration, and retention period. Larger messages and longer retention windows increase disk I/O requirements. If you are using Standard brokers with EBS volumes, make sure the provisioned IOPS and throughput match your write amplification after replication.

Alternatives Worth Knowing

While Amazon MSK is the managed Kafka offering on AWS, the broader ecosystem includes other options. Confluent Cloud provides a fully managed Kafka service with its own sizing abstractions and enterprise features. WarpStream takes a different architectural approach by disaggregating storage and compute, which eliminates inter broker replication entirely and can reduce costs for certain workload profiles. Redpanda is a Kafka compatible streaming platform written in C++ that aims for lower tail latencies and simpler operations by eliminating the JVM and ZooKeeper dependencies.