Substack is a subscription media platform where writers, podcasters, and video creators build direct relationships with their audiences. It began with paid newsletters and has expanded into an app and social network, but its core remains creator-owned subscriptions and the trust between creators and their subscribers.
Ask Mike Cohen how often he opens Artie's dashboard and the answer is "very seldomly." Ask what he does open and it's a Terraform file, which he edits when Substack needs a table added or dropped. That is the whole surface area of his relationship with the system that moves Substack's production Postgres into Snowflake.
That's it, he says. That's, like, the extent of my Artie interaction.
– Mike Cohen, Head of AI & ML Engineering at Substack
| Company Website | substack.com |
| Switched from | Heroku |
| Switched | June 2023 |
| Use case | Postgres + DynamoDB to Snowflake ingestion for analytics, experimentation, machine learning, and trust and safety |
Mike was Substack's only data person when he joined six and a half years ago. About two and a half years ago he shifted toward the company's ML work, though he describes that as still tethered to data in many ways.
Substack's production stack is mainly one Postgres database, plus some DynamoDB added over time. The rule at Substack is that nobody queries production. Analytics, business questions, experiments, models: all of it comes out of Snowflake. Which makes the thing that gets data from one to the other the kind of system nobody thinks about until it stops.
The setup nobody was complaining about
The system Mike had built did the same job Artie does now, on a rolling hourly basis. Every table refreshed once an hour as either a wholesale rebuild or a new slice. On the larger tables it was multiple hours.
It was hands-off and it mostly worked. When it missed an update, Mike had redundancies. Row-count checks compared prod against Snowflake. When a row didn't show up, he had a script that would set updated_at on that row in production to force it to sync.
We were just used to it. You kind of just get used to abuse in some way.
Then the tool they were using got acquired and was on its way to be sunset, and Mike had a six month timeline to find something else and move every table onto it.
The other thing that had changed is that CDC was now a possibility. It was something the team wanted for a while but had been blocked, because at the time Substack ran on Heroku and Heroku didn't allow it. The unblock for CDC occured when Substack moved to RDS on AWS.
The key thing I wanted was CDC because I was like, this will enable real time. But I don't even think I really knew at that point what that would be like, because I never had that in any previous job either.
The search was smaller than you'd assume. Substack isn't a big company, Mike points out, and not many people are working on this problem, the solution needed to work on its own. He set out with one thing he specifically did not want to do, which was end using a specific known vendor in the category. He had had heard horror stories about the product failures and pricing hikes and a suspicion that it wasn't real CDC anyway. He kept it as the doomsday option and never used it.
There was one other serious CDC contender, he was considering and had started building around it. Which is how he found the difference that decided things. That vendor sent the deltas into Snowflake without the reconciliation merge step, so getting Snowflake to reflect Postgres would have meant running that reconciliation himself, across every table, forever. Totally doable, he figured. Just an extra set of steps he wouldn't have with Artie.
What he wanted was narrower than "move the data." He wanted Snowflake to reflect Postgres and nothing else. When someone at Substack asked him that week what Artie is actually for, he answered in one line:
We intend for Snowflake to mirror prod completely.
The Substack data team intends for it to mirror prod completely so that anyone can analyze the business on any dimension at any point in time.
The video
The thing that interested him was not a benchmark or a security review. It was a short demo video on Artie's website showing a row inserted in Postgres and appearing in the destination, updated and reflected, deleted and gone.
It was exactly the way I sort of dreamed or imagined a product would work. That's exactly what I want, and exactly what I don't have in this batch solution.
Artie applies the merge inside the pipeline, so inserts, updates, and deletes land applied and the destination table holds source state rather than a log of changes. That is the whole product difference, and for Mike it was the whole decision.
Customer number one
Substack signed as Artie's first customer, which meant unproven at their data scale. There were a couple of hiccups in the early integration.
What made those hiccups survivable was who was on each side of them. Substack put its own database expert on it, a person Mike calls a sensei of databases. Robin, Artie's co-founder and CTO, took his input and turned fixes around fast.
We had two very skilled people kind of speaking the same language. They were able to resolve stuff as it came up and it's kind of mostly been incident-free since.
Asked whether he'd been worried it would go badly, Mike's answer was no, and the reasoning is less heroic than it sounds. He had a fallback he'd already half- built, and a doomsday vendor behind that. The catastrophic case, Artie somehow taking down the production database, never felt likely to him.
The benefits nobody scoped
The migration solved the problem it was scoped to solve. Anyone at Substack can now ask what the state of the business is and get an answer in seconds. Before, that meant waiting out the hourly clock, or connecting to a production follower where the larger aggregations couldn't run at all. Not knowing how many subscriptions you've added in the last hour is, Mike notes, kind of nuts.
Business intelligence was the anticipated win. The rest were not.
We were sort of just trying to solve a problem, and then there was a bunch of ancillary benefits that ended up being outcomes of having solved this other problem.
Experiment analysis had always been gated on batch data landing in Snowflake, so how fast Substack could read an experiment was a function of how fast the pipeline ran. That gate came off. Recommendation models train from the same warehouse, and a model can only recommend content it knows exists, so ingestion lag put a floor under how fresh any model could be. Removing the lag removed the floor. Trust and safety runs there too, and on a platform with user- generated content, lag in ingestion is lag before questionable content gets looked at.
There's also the system Substack never had to build. Without real-time ingestion, wanting sub-hour visibility eventually forces a hybrid: Snowflake for anything over an hour old, direct queries against the production follower for anything newer, and something merging the two at read time. Everything becomes more complicated. Substack skipped that architecture entirely and got one data source with everything in it, which is one less system to think about.
These days it's a Terraform file
For the last six months or so, Substack has managed table configuration through Artie's Terraform provider, so adding or dropping a table is a config change. The row-count checks and the forced-resync script are retired. Mike does not look at throughput numbers and is cheerfully uninterested in them. He only cares about it working and being fast.
Asked whether his data team is happier, he didn't hedge:
A thousand percent. A million percent.
See how Artie mirrors Postgres into Snowflake with managed CDC
Artie replicates your production database into your warehouse with change data capture and applies inserts, updates, and deletes in place, so the destination reflects source state rather than a log of changes.
About Artie: Artie is a fully managed platform for real-time replication from databases to warehouses and data lakes using change data capture (CDC). It provides sub-minute latency, exactly-once delivery, and automatic schema evolution without requiring teams to operate streaming infrastructure.
About Substack: Substack started as a newsletter email sending platform and has evolved into an app with a social layer. The core business is still subscriptions, and the trust between subscribers and the people they subscribe to.