> Markdown version of https://www.artie.com/blogs/we-left-msk-built-our-own-kafka. Site index for AI agents: https://www.artie.com/llms.txt

[Back to blog](https://www.artie.com/blogs)

Product spotlight November 3, 2026

# We Left MSK, Built Our Own Kafka, and Now Sell It at EC2 + 25%

[Robin Tang Co-founder & CTO](https://www.artie.com/authors/robin-tang)

[Share on LinkedIn (opens in a new tab)](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.artie.com%2Fblogs%2Fwe-left-msk-built-our-own-kafka) [Share on X (opens in a new tab)](https://x.com/intent/post?text=We%20Left%20MSK%2C%20Built%20Our%20Own%20Kafka%2C%20and%20Now%20Sell%20It%20at%20EC2%20%2B%2025%25\&url=https%3A%2F%2Fwww.artie.com%2Fblogs%2Fwe-left-msk-built-our-own-kafka)

All of Silicon Valley seems to be talking about AI solving Millennium Prize Problems and reaching AGI. Meanwhile, running infrastructure still mostly sucks. Take Kafka: AGI or not, keeping a cluster healthy takes engineers who understand networking.

The Confluent vs. Redpanda debate is from another era, and who cares? They're both good, and they both implement the Kafka APIs.

Kafka is still the industry-standard event bus. Most large teams prefer Kafka and its ecosystem, while others make do with tools like Amazon SQS or Google Pub/Sub.

We believe a Kafka launched in the AI era should do two things:

1. It should have a DevOps engineer baked into the product, so you don't have to be on call.
2. It should cost close to what you'd pay to run it yourself.

You used to have to pick one, because everything is a trade-off. We don't think you have to anymore.

## How We Ended Up Running Our Own Kafka

When I started Artie, I assumed I would use Confluent, because our data planes needed a cloud-agnostic Kafka. The deal-breaker came quickly: I couldn't change broker settings without buying an enterprise tier.

Redpanda required us to run its agent inside each of our data-plane clusters. That would have complicated every bring-your-own-cloud deployment we run for customers.

So we went with Amazon MSK, and MSK was good to us for a long time. But then the cracks started to show.

**Storage could only grow.** MSK lets you add disk to a broker but never take it back, and after every increase the cluster sits in a cool-down of at least 6 hours, sometimes more than 24, before you can scale it again ([AWS docs](https://docs.aws.amazon.com/msk/latest/developerguide/msk-update-storage.html)). So we overprovisioned every cluster, and when we still guessed wrong, we had to cut retention (delete data) or move topics to another cluster to get through the wait.

**Configuration changes were slow.** Changes that should take a few minutes took over an hour on MSK.

**Metrics cost extra.** Per-broker metrics meant paying for a higher monitoring tier or running our own JMX scraper.

**Adding brokers was manual.** MSK doesn't move existing partitions onto brokers you add. AWS's answer is `kafka-reassign-partitions.sh`, with no more than 10 partitions per call ([AWS docs](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices.html#bestpractices-balance-cluster)). So I wrote my own rebalancer job, with throttling and pauses between batches so production traffic wouldn't notice. Here's one of our clusters after we added 3 brokers, at three points in the hour that followed:

Partitions per broker after we added 3 brokers

Our Amazon MSK cluster at 13:05: a 617-partition gap between the busiest and quietest brokers

The dashed line is an even split, 437 per broker. Partition replicas per broker from our Datadog dashboard; the total grows because new topics were created during the hour.

| Time  | Broker 1 | Broker 2 | Broker 3 | Broker 4 (added) | Broker 5 (added) | Broker 6 (added) |
| ----- | -------- | -------- | -------- | ---------------- | ---------------- | ---------------- |
| 13:05 | 745      | 732      | 736      | 137              | 141              | 128              |
| 13:21 | 821      | 815      | 809      | 206              | 200              | 194              |
| 14:03 | 727      | 707      | 717      | 427              | 423              | 418              |

An hour in, the added brokers still held about 300 fewer partitions than the originals. We worked through every edge case like this one, but what settled the decision to move off MSK was a major outage caused by IAM authentication.

## The Night IAM Auth Took Down Our Brokers

On the afternoon of May 9, 2024, I added 3 brokers to one of our Kafka clusters, then moved a customer workload onto the newly scaled-up cluster.

At 9:24 pm, broker 2's CPU spiked, and a flurry of consumers got stuck in "Rebalance in progress" loops.

Kafka CPU by broker that night

Broker 2 went past our 60% alert and stayed above it for most of the next four hours.

9:36 pm Broker 2 87% Busiest other broker 13%

Pacific time, May 9 to 10, 2024. Traced from our Datadog panel, so readings are approximate. Move across the chart to read any minute.

It was hard to debug:

1. The error message didn't tell us much. It looked transient, but it wasn't.
2. We tried spreading the load by shuffling the bootstrap servers in our clients' Kafka connection strings, but that didn't work.
3. Turning off 40 Kubernetes consumer deployments helped, but as soon as I turned a few of them back on, the rebalance loop started again.

It turned out MSK's IAM authentication agent was eating the broker's CPU on every new client connection. [IAM auth](https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html) is CPU-intensive for the broker, so when enough clients reconnect at once, the broker spends its CPU authenticating them instead of serving data, which makes more clients time out and reconnect.

IAM auth's CPU on brokers 2 and 3

CPU used by MSK's IAM authentication agent alone, on the two brokers it overwhelmed.

10:17 pm Broker 2 135% Broker 3 61%

Pacific time, the same night as the chart above (CloudWatch showed UTC). Traced from CloudWatch's IAMAuthAgentCpuUsagePct taking the peak of each few-minute window, so readings and times are approximate.

[SASL/SCRAM](https://kafka.apache.org/documentation/#security_sasl_scram) pushes most of that work to the client, and we should have used it. The expensive part of SCRAM is hashing the password with its salt over a set number of iterations, and the client does all of it. The broker only does a small amount of hashing to check the client's proof. Pick "Everyone reconnects" below to see why that mattered:

Where the expensive part of auth runs

Each new client connection, before it can read or write

| Method                      | On the clients                                        | On the broker                       |
| --------------------------- | ----------------------------------------------------- | ----------------------------------- |
| IAM MSK, what we ran        | Signs the request                                     | Checks the signature and IAM policy |
| SASL/SCRAM What we moved to | Hashes the password with its salt, thousands of times | A few quick hashes and a compare    |

Large squares are the expensive work, small ones the cheap work. A picture of where the work runs, not a measurement.

That night we fixed it by adding jitter to reconnects. Later we moved to SASL/SCRAM. Over the next year, as we got good at running Kafka, we moved all our workloads onto our own Kafka on Kubernetes with [Strimzi](https://strimzi.io/), a [CNCF](https://www.cncf.io/projects/strimzi/) project we'd been following for a while.

## What We Built

**Brokers that heal in place.** If a broker dies, it comes back with the same disk, through a Kubernetes [PersistentVolumeClaim](https://kubernetes.io/docs/concepts/storage/persistent-volumes/), and catches up on its own. If the node is gone, a new one starts in the same availability zone and attaches the same disk. We use the recommended high-availability settings: every partition has 3 copies (replication factor 3), and a write needs 2 of them ([`min.insync.replicas`](https://kafka.apache.org/documentation/#topicconfigs_min.insync.replicas) = 2), so if one broker is down, the cluster keeps going.

When a broker fails

3 partitions, 3 copies each. Filled chips lead their partition.

1.
2.
3.

Every partition has 3 copies, one on each broker. A write counts once 2 of them have it.

**Automated rebalancing through Cruise Control.** When we first wrote our own rebalancer in 2024, it was naive. It flagged storage-heavy topics, moved partitions in chunks, and slept between iterations so production wouldn't be affected.

We kept adding logic, like hot-partition skipping, replication throttling, and leadership-aware placement. Eventually it mirrored LinkedIn's [Cruise Control](https://github.com/linkedin/cruise-control), so we switched to the real thing.

**Storage that scales with you.** We tackled storage in two ways:

1. **Storage autoscaling.** We took inspiration from [RDS storage autoscaling](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PIOPS.Autoscaling.html): disks grow before they fill, up to a limit you set, so a traffic spike doesn't force you to cut retention.
2. **Tiered storage.** At the cluster and topic levels, you choose how much recent data lives on broker disks and how much is offloaded to S3 ([KIP-405](https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage)). Consumers read both the same way. That way the disk doesn't keep growing with your retention, and history for replays sits in S3 instead of EBS.

**Your agent can run it too.** Everything you can do in the dashboard is also in our public API, which has an OpenAPI spec. Our MCP server is built on that spec, so your agent can do anything you can: create topics, change broker configs, start a rebalance and check on it. If letting an agent near production makes you nervous (it should), turn on read-only mode and let it look without touching anything. Terraform support for clusters and topics is next.

**Every broker setting is yours.** We ship sensible defaults, and you can change cluster configurations yourself. No enterprise flag blocks you, and changes apply in minutes instead of an hour.

**We carry the pager.** On MSK, you still plan storage, schedule upgrades, rebalance partitions, and answer the alarms. On Artie, our engineers carry the pager. The platform replaces failed brokers, rebalances partitions, grows disks, and rolls out upgrades one broker at a time. Every change we make shows up in your activity log, and broker metrics stream to your Datadog.

We did the same thing with Artie CDC, where we turned the on-call work our customers used to do for data pipelines into software. Now we're doing it for Kafka.

**Security.** You can deploy in Artie's cloud or in your own cloud environment. Clients connect over TLS with SASL/SCRAM, and only IP addresses on your allowlist can get in. AWS PrivateLink, GCP, and Azure are on our roadmap.

## What Kafka Should Cost

We believe managed infrastructure should be priced close to its actual cost, with a slim margin on top. Charging a big markup made sense when running Kafka well meant paying a big ops team. It doesn't anymore. We're an AI-native company that's already built all of this for ourselves, and we pass the savings on to our customers.

Our Kafka pricing has two parts:

* **AWS's cost, at on-demand prices with nothing added.** That covers the broker instances, their gp3 disks, S3 for tiered storage, the load balancer and its public IP address, and data transfer.
* **Artie's fee: 25% of the broker instance cost.** We don't mark up storage or data transfer.

You see that breakdown in the dashboard before you create anything:

![The dashboard's Review and create step for a Managed Kafka cluster in us-east-1 with 3 r8g.large brokers (2 vCPU, 16 GiB memory each) and 240 GiB of storage per broker. The cost lists brokers $258, broker disks (720 GiB gp3) $58, load balancer $16, public IPv4 address $4, and the Artie fee at 25% of compute, $65, for an estimated total of $401 a month, plus data transfer at AWS's per-GB rates: $0.01 in the same region, $0.05 to $0.09 to the internet or other clouds, and $0.006 through the load balancer.](https://www.artie.com/editorial/blogs/we-left-msk-built-our-own-kafka/dashboard-review-and-create.webp)

The last step before a cluster is created: AWS's cost line by line and Artie's fee, with data transfer rates listed separately from the total.

Data transfer is where managed Kafka bills balloon. On Artie, consumer reads in the same region cost AWS's $0.01 per GB. Confluent and Redpanda charge $0.035 to $0.05 for every GB in and out.

Here's the same workload on Artie and on Amazon MSK, at today's prices. Pick a size:

59% less than Amazon MSK

Monthly cost for 5 MB/s in, 2 TB retained, on 6 r8g.large brokers

* Artie $1,119 Infrastructure **$868** Data transfer **$122** Artie fee **$129**
* Amazon MSK $2,740

AWS prices for us-east-1 as of November 21, 2026; Amazon MSK prices from AWS's price list of October 14, 2026. Both assume every message is read once, in the same region. Artie runs in one availability zone; Amazon MSK runs the same number of brokers with the same memory across three zones and includes its cross-zone client traffic. Source: [Amazon MSK](https://aws.amazon.com/msk/pricing/).

One caveat: at launch, Artie clusters run in a single availability zone, while the MSK price assumes the same brokers across three zones and includes cross-zone client traffic. One zone is also why there's no cross-zone replication charge on your Artie bill. Each message is still stored on 3 brokers, so losing a broker doesn't stop producers, but a full zone outage takes the cluster down until the zone recovers. Multi-AZ is on our roadmap.

To price your own cluster, use the [calculator at artie.com/kafka](https://www.artie.com/kafka#price).

**Coming from another Kafka?** [Schedule a call](https://calendly.com/robin-artie/book-a-demo-with-artie-managed-kafka) and our engineering team will plan the migration with you. We use [MirrorMaker 2](https://kafka.apache.org/documentation/#georeplication) to copy your topics over, then help you move your producers and consumers. A self-serve migration tool in the dashboard is in the works.

Artie Managed Kafka is live today. [Create a cluster](https://app.artie.com/kafka-clusters?utm_source=web) (you'll see the price before anything spins up), or start with the [docs](https://www.artie.com/docs/managed-kafka/overview).

## Kafka in 2026

Most teams we talk to don't hate Kafka. They hate running it. It's complicated, and it pages you. Brokers and storage need scaling, data transfer needs watching, and if you misconfigure it, you can get hit with a massive bill.

We think running it should be boring and cheap. We're turning the work that used to need an ops team into software: replacing brokers, rebalancing partitions, growing disks, and catching problems before they page anyone. Every task we automate is one less reason for you to be on call, and one less line on your bill.

Cheap Kafka or good Kafka? We don't think you have to pick anymore. Come try to break it.

## Related articles

* ### [How We Built Hermes to Support Our Entire Team](https://www.artie.com/blogs/how-we-built-hermes)

  [Jacqueline Cheong Co-founder & CEO](https://www.artie.com/authors/jacqueline-cheong)

* ### [Introducing Artie Transfer](https://www.artie.com/blogs/introducing-artie-transfer)

  [Robin Tang Co-founder & CTO](https://www.artie.com/authors/robin-tang)

* ### [Announcing Artie’s New MotherDuck Destination](https://www.artie.com/blogs/announcing-arties-new-motherduck-destination)

  [Jacqueline Cheong Co-founder & CEO](https://www.artie.com/authors/jacqueline-cheong)

## We’re a data company

## Watch your data replicate live

[Start for free](https://app.artie.com/signup) [Book a demo](https://www.artie.com/contact)
