Skip to contentManaged Kafka is live: Apache Kafka on AWS for half the cost of MSK
Back to customers

How Engage3 Rebuilt Its Data Replication Layer in Three Weeks

Engage3 delivers Commercial Decision Intelligence to retailers and brands, including tools to track and optimize their Price Image, the measure of how shoppers perceive a retailer's prices. Roughly 600 million row updates a week move from Engage3's operational Postgres databases into Snowflake, where they power the analytics and reports clients use every day.

As that volume grew, Engage3 wanted a replication layer that was faster, simpler to operate and more cost-efficient, with one tool covering every replication path instead of two.

Matthew Belo, who manages the databases, Snowflake, and now Artie, moved 88 pipelines to Artie in under three weeks, with no downtime.

Our clients rely on that pipeline working from us to them for their daily work. That's why our data has to flow consistently and reliably.

Matthew BeloPrincipal Data Architect at Engage3
Company Website
engage3.com
Started
August 2026
Sources
Postgres and Snowflake
Destinations
Snowflake and Postgres
Use case
Replication across US and EU environments for client analytics and reports

The stack

Engage3's two core products are Competitive Intelligence Management (CIM), which gives retailers item-level competitive pricing data, and Price Image Management (PIM), which recommends prices against a retailer's goals. Both run on data collected from public web sources, in-store audits and customer feeds. The web data alone lands in Snowflake at about half a trillion rows for the most recent year, with many more years of history behind it.

The platform runs in development and production environments across the US and EU, each with its own Postgres databases and a Snowflake instance. A products database is populated from in-store audits, customer input files and web data. Lighthouse, Engage3's product linking application, is where retailers link their own products, including private label and exclusive items, to the comparable competitor listings Engage3 has identified. Field data comes from the mobile app that Engage3 auditors use to capture shelf prices in physical stores, for comparisons that do not exist online.

Snowflake is where everything gets joined, modeled and turned into client reports, so the core replication job is Postgres to Snowflake, continuously, across every environment. Products, Lighthouse and field data are also kept in separate databases by design, which means a lot of Postgres-to-Postgres replication as well. The field app needs product reference data to attach store observations to. Lighthouse needs the catalog to link against. A handful of tables also flow the other direction, from Snowflake down into specific Postgres databases.

Everything Engage3 sells sits downstream of this. As Matthew puts it, "All of the information from Postgres going to Snowflake feeds all of the other products, CIM and PIM, that provide all of the insights to the customer." Replication is the path between what the company collects and what it delivers.

Outgrowing the replication layer

Engage3's data volume has grown faster than any single replication tool has kept up with. When Matthew joined five years ago, replication was a homegrown system built for a fraction of today's load, and one of his first projects was replacing it with a commercial ETL tool. That was the right call at the time: it was cost-effective and handled the volume Engage3 had then.

As Engage3's workload scaled, that tool required more hands-on maintenance, so the team moved to a managed replication service.

That service handled Postgres to Snowflake well. Postgres-to-Postgres replication with native data types preserved, which Engage3 needs between its products, Lighthouse and field databases, required a separate replication product with its own license and dedicated servers to run it.

Consolidating to one platform

Two replication tools, two licenses and dedicated servers added cost and operational overhead to a layer that every client report depends on. Engage3 wanted one managed platform that could handle every replication path, scale with its volume, and respond fast when something needed attention.

Running Artie's proof of concept

The criteria were functionality and how long it would take to build 120 pipelines. If each one took a week, the POC would have ended early.

The 26 development pipelines took an afternoon. By the next morning all of them had backfilled, including the large tables. EU production, another 25 or so, took about two hours. After the first one or two, a new pipeline was roughly ten minutes of work, and cloning turned the EU-to-US build into swapping source and destination.

What surprised me most during the rollout was how massively simple it was to build the pipelines. Once I got through the first one or two, it was ten minutes. And the ability to clone them was useful, because I had all of the EU ones and I had to create analogous US ones. It was great to just clone it, just switch the front and back ends. That was probably the most enlightening piece. It sealed the deal as far as functionality.

Matthew BeloPrincipal Data Architect at Engage3

The limits hit during the POC were account limits on pipeline count and total table count. In several cases Artie raised them before the Slack message asking for it was finished. One configuration issue surfaced while many backfills ran concurrently, and Artie fixed it in 15 minutes.

The capability that mattered most was native data types. Artie carries Engage3's Postgres types end to end, into Snowflake and into Postgres, rather than coercing them through an intermediate universal type model. Engage3's schemas are built on those types. That one capability collapsed two replication products into a single tool.

By the middle of the third week, 88 pipelines were built and backfilled across every environment.

Backfill without touching the source

The reference tables being replicated are well over a billion rows. During the POC, Artie pushed hundreds of millions to billions of rows per table into both Snowflake and Postgres destinations, and every pipeline finished seeding within two days.

The bigger concern was protecting the source databases. With the previous setup, a full re-seed took almost four days and held the Postgres write-ahead log for the entire window. Artie runs backfills in parallel with, and separately from, the CDC stream, so the WAL kept advancing normally and there was no replication lag.

Several of the largest tables have keys that are hard to split for parallel reads. Artie handled them at the same speed.

The speed at which Artie pushed hundreds of millions to billions of rows to the destination, whether it was Postgres or Snowflake, was extremely impressive. We were able to pretty much complete every single one of those pipes with all of that data within two days, and without any impact to the source system.

Matthew BeloPrincipal Data Architect at Engage3

Two gaps, closed in weeks

The gap analysis turned up a few things Artie could not do at the time.

Partitioned tables. Two or three tables have to land at the destination with the same partitioning as the source, for query speed. Artie shipped this a couple of weeks after Engage3 committed.

Snowflake as a source. A few tables flow from Snowflake down into specific Postgres databases. Artie supported Snowflake as a destination during the POC but not as a source. It shipped about a week after partitioned tables.

Making the switch

Artie runs on an Extra Small warehouse that is suspended most of the time. Pipelines sync about every 15 minutes and most operations finish in under 10 seconds. A 200-million-row bulk load that used to take about 20 minutes now takes about 25 seconds. Snowflake compute for replication dropped by roughly three quarters, and the dedicated replication servers are gone entirely.

Across the POC and the months since, Artie has had zero outages. The few issues that have come up were resolved in minutes, in several cases before Engage3 finished reporting them.

One engineer built 88 pipelines in under three weeks and cut over production in 30 days, with the migration running alongside the existing setup and no downtime. Two product gaps closed in weeks. Two products and two licenses became one. Artie has been fully managed in practice and in production.

Previous setupArtie
Products and licenses21
Snowflake warehouseMedium, running continuouslyXS, mostly suspended
Dedicated serversRequired for Postgres to PostgresNone
Backfill, billion-row tables~4 days, WAL held for the full window~2 days, no source impact
200M-row bulk load~20 minutes~25 seconds
Routine sync operations30 seconds to 6 minutesUnder 10 seconds
Native Postgres types to PostgresSeparate productBuilt in
Pipeline build timeUp to a week for some~10 minutes each, 26 in an afternoon

Running it

Engage3 now pushes around 600 million update records a week through Artie.

And Artie doesn't even breathe hard.

Matthew BeloPrincipal Data Architect at Engage3

There have been a few issues on the Artie side. None caused an outage. In four or five cases, Matthew was partway through typing a Slack message about a problem when Artie's team replied that they were already on it.

Whenever I have discussed or handed off anything to anybody in the organization at Artie, I don't have to worry about it sitting idly on the side. I know that it's going to be worked on. That level of persistence and caring is important.

Matthew BeloPrincipal Data Architect at Engage3

When something is delayed, the dashboard shows what and why, and the pipeline analytics are enough to reason about throughput without digging. For a team managing dozens of databases, 50 terabytes in Snowflake and web data measured in trillions of rows, that visibility is critical.

I've been in this business for over 46 years. I've written code in over 40 languages and worked with 16 different databases. I know how they should run, and I know how products should work with them. Artie has proven to be one of the most exceptional pieces I've added to the stacks I work with.

Matthew BeloPrincipal Data Architect at Engage3

What it adds up to

Engage3 switched replication tools to consolidate its stack, cut compute cost and work with a partner that responds in minutes and ships what it needs in weeks.

A few months in, the replication layer is one product instead of two, runs on a quarter of the Snowflake compute, and has not gone down. Data lands in Snowflake every 15 minutes, so the dynamic tables and pipelines that feed CIM and PIM stay current without anyone checking on them. The two capabilities Engage3 needed that Artie did not have shipped in weeks. And the migration itself, 88 pipelines and billions of backfilled rows, was done by one engineer in under a month with no downtime.

About Artie: Artie is a real-time data replication solution for databases and data warehouses. Artie leverages change data capture (CDC) and stream processing to perform data syncs in a more efficient way, which enables sub-minute latency and helps optimize compute costs. With Artie, any company can set up streaming pipelines in minutes.

About Engage3: Engage3 delivers Commercial Decision Intelligence to retailers and brands, including tools to track and optimize their Price Image, the measure of how shoppers perceive a retailer's prices. Its two core products are Competitive Intelligence Management (CIM), which gives retailers item-level competitive pricing data, and Price Image Management (PIM), which recommends prices against a retailer's goals.

Watch your data replicate live